Posted: Tue Oct 14, 2014 12:39 pm Post subject:
Digest in maps
I know this perhaps should not go here, but I'm asking it for both Tiberian Sun and Red Alert 2.
How is the [Digest] section calculated in map files?
I need to know this to create valid maps for multiplayer.
From what I've read, it is used to validate whether 2 players have the exact same map.
If they don't the game would throw an error "Scenario is corrupt" or something.
Removing [Digest] also prevents the validation from happening, but it is safer when it exists.
Please read;
I don't want to be pointed to FA2 because "it does it for you",
I need to know how I can 'manually' create a valid Digest value. _________________
I would assume it's some kind of hash value of the whole map.
The question is, if the game uses the same algorithm as FA to create this value and then checks if the Digest hash and the calculated maps hash are the same
or
if it only checks the hashs of the 2 maps and the Digest hashs (but not map hash against Digest hash), to see if they are different.
In the second case, you should be able to simple write there your own hash value.
In the first case, you would indeed need the algorithm the game uses to create the Digest value manually.
A simple test would be
-create a map
-write in the Digest only something like 1=2wingdangdudel4=
-give the map a partner
-change in your map something (like remove a tree)
then start the game
If it starts, you know that the manually set hash value isn't working. _________________ SHP Artist of Twisted Insurrection: Nod buildings
@CCHyper: This is not the case, it's not 1 for all.
@LKO:
I have tested it with someone on CnCNet.
If his [Digest] is different than mine; game says not all players have the map
If his [Digest] is the same as mine, but I modified the map (different map hash?); game says not all players have the map
I don't know how this works in XWIS or non-LAN games, it seems that if either of the hashes doesn't match,
the game will transfer the host's map to everyone.
So:
Mine His
[Digest] =/= [Digest]
[Map] == [Map]
Game detects difference, host sends over the map.
Mine His
[Digest] == [Digest]
[Map] =/= [Map]
Game detects difference, host sends over the map.
Mine His
[Digest] == [Digest]
[Map] == [Map]
But: [Digest] =/= [Map]
Game detects difference, host sends over the map.
I have not found any relation between [Digest] and the map hash other than above.
I think it's quite hard to properly test because this has no effect on singleplayer maps.
One could draw the preliminary conclusion that the map hash and [Digest] are actually compared and thus need to be valid.
One note:
If [Digest] is not present in the map file, it will not (at least for custom mp maps) be displayed in the map list. _________________
Is it the client or the actual game that says not all players have the map? In other words; do you get to see the ingame loading screen before you get the error or is an error displayed before the game even starts?
If the error comes from the client, then the issue obviously isn't with the game, but the client and you're best off contacting FunkyFr3sh about this (since he made and maintains the client). _________________ QUICK_EDIT
It's not CnCNet 5, forgot to mention that, sorry. The talk about hashes might be confusing, too, I just copied LKO's terminology.
It's the game that says it though, before loading any map. _________________
How did you test 3rd condition? Have you copied a Digest from another map into the current
one and transferred to the other player?
Finalsun/FA2 does not modify an existing digest value, when the map gets modified. If those
works then the idea may be is to keep it unique per file, not to decode and verify the checksum.
When the digest is removed and then saving the map in FS/FA2, it creates a new one different
from the old one everytime. Checksum are also not expected to be inside the same file itself. QUICK_EDIT
I changed the Digest of the file after I created / saved it to some horseshit line "lololderp".
We both used the same Digest value, so the map files themselves were identical.
The Digest value wasn't equal to what the game generates, so it detected changes.
Maybe when it finds that out it just assumes they're not the same (the maps) because the Digest has been tampered with.
I don't know how FA2 does it, it seems a tad bit inconsistent... _________________
Format looks like 1= and then 27 alphanumeric chars (upper + lower and also + and /) followed by
= at the end.
Should have tried from another map or by removing the entry and saving it in FA2 to regenerate.
And then transferring the map file to the other player (not from in-game). It could turn out to be a
randomly generated entry to a 28 char format to make it unique. QUICK_EDIT
Mix files appear to use the SHA1 digest for their checksums, at a guess I would suggest that this may be a base64 encoded SHA1 digest of the map before it has the digest section added. QUICK_EDIT
CCHyper told me it is a Base64'd SHA1 hash.
Based on that information I implemented methods to generate and write a digest.
Basically what it does is create a random hash, write 0x80 at the end and then Base64 it.
The last byte, 0x80, is always translated to '=' in Base64.
That's also why you quite often see '=' or '==' at the end of Overlay(Data)Pack:
It uses Format80 and the terminator/last byte is usually 0x80, used for the commands the compression uses.
I have tested the generated Digest on my PC and laptop, both use the same configuration (using Dropbox), so everything is the same.
The game (on PC, as host) did not tell me that Laptop did not have the map, it seems it saw both maps as a proper map. _________________
If the hash is random, what other information gets sent to decide if the maps are not the same then when the digest matches but the contents of one are altered? File length? Wouldn't it be possible to adjust for example tech levels on units whose tech level is declared in a map that way (turn an 11 or -1 into 10 for example) such that you could build but opponent couldn't? QUICK_EDIT
On a slightly tangental topic, does anyone know what the WW format80 encoder does more aggressively than the code from XCC? If you convert for example a CPS image from RA to pcx and back the image is always slightly larger. I assume that at least edwin has a WW format80 encoder built into it since it needs to encode to it for the pack sections in a map? QUICK_EDIT
On a slightly tangental topic, does anyone know what the WW format80 encoder does more aggressively than the code from XCC? If you convert for example a CPS image from RA to pcx and back the image is always slightly larger. I assume that at least edwin has a WW format80 encoder built into it since it needs to encode to it for the pack sections in a map?
What is format40/80 and what sections use this compression? I'll look into it. QUICK_EDIT
format80 is the compression format used for most images in the game, http://eob.wikispaces.com/Format+80 has an overview of the format on disk and how it is decoded. MapPack and OverlayPack are I believe first format80 compressed followed by base64 encoded to be recorded in the map ini file. http://cnc.wikia.com/wiki/Red_Alert_File_Formats_Guide briefly mentions how its used in the *Pack sections when it discusses base64 decoding.
Format40 is used in shp and wsa files, though I don't believe XCC even tries to use it in shp files, just format80 compressing all frames to save processing time testing which format is better. It is something like a compressed xor between the target image and some reference image.
As far as reference code for format40, I guess the only tool that would have WW code in it would be the recently leaked shp builder tool that leaked with the RA2 infantry models. QUICK_EDIT
Format40 and Format80 are the incorrect names, it looks very much like it is LCW as these sections are parsed through the LCWPipe/Straw. (Straw = Read/Get, Pipe = Write/Put).
EDIT: Yup, its definitely LCW compression/decompression. CPS file uses a version that seems to have been written in ASM (leftover from DOS perhaps).
EDIT2: I can not find anything related to LCW on the internet, but LZW appears a lot. Does anybody know of the format/s? QUICK_EDIT
Did you check whether it used all 5 command bytes for compression, Blade?
It could be like in my implementation, only using 0 (raw copy) and 4 (copy byte X n times).
It's the quickest way to get 'proper' compression.
The other command bytes take some time to figure out for writing because you have to look through the destination.
Writing the overlay packs is really easy with these 2 commands because about 80% is always 255. _________________
My code is largely cribbed from XCC at the moment though I wrote an equivalent for part of it that was still in assembly to make it a touch more portable. XCC looks like it uses all the commands (XCC actually has variants of format80 that do as you suggest) and it still generates format80 data larger than the original files. I could log the decisions the decoder and encoder make as they cycle a file, but there are thousands of bytes in a file and I may loose the will to live looking for where the encoder made a different choice and then working out why. QUICK_EDIT
I wouldn't worry about it that much.
The only difference is that the XCC variant takes up a bit more diskspace, in the end every SHP is equally large regardless of its compression.
Don't mind logging, I've been through it while working on the encoder, it's a bitch _________________
As far as reference code for format40, I guess the only tool that would have WW code in it would be the recently leaked shp builder tool that leaked with the RA2 infantry models.
Sorry to say this seems false, that shp tool will only encode the SHP (TS) format (C1 & C3 RLE) that does not use the format40/format80 methods that original SHP (TD-RA1) shps used thus can't be used to encode them.
More chance with maybe RA1 map editor... QUICK_EDIT
"LCW was a custom Westwood LZ sytle compression. LC = Louis Castle, W = Westwood (Louis Castle wrote this compression code the the VERY early garage days."
Now we know what the name is, who wrote it and what style it is based on!
EDIT:
Louis Castle wrote:
Wow. Yes, I wrote it.
Lempel - Castle - Welch. A bit tough to explain in a message but I'll give it a shot.
You can post new topics in this forum You can reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum