Unprotecting a protected MIX
Some background:
TS/RA2 Mix files consist of a header, an index, and a body. The header includes basic information about the MIX (does it have a checksum, is it encrypted, how many files are in it, how big is it). The index includes the size, location, and checksum of each file in the mix, and the body contains the actual files.
There are two main "MIX protector" or "Leechkiller" programs out there that will mangle the header (specifically, the internal file size) in such a way that RA2 reads the mix properly but XCC Mixer will refuse to open it. They both operate under essentially the same principle.
First, it is necessary to understand how the headers are laid out. The first four bytes are the encryption and checksum flags. If the MIX is has encrypted headders (not to be confused with "protected": "encrypted" refers to the Blowfish encryption that Westwood used on its own mixes; XCC breaks this just fine) the flags are 00 02 00 00. If the MIX has a checksum, the flags are 00 01 00 00. Neither of the current MIX Protector programs work on files with encrypted headers, (although they could be modified to do so with some effort), so I won't go into that right now.
On unencrypted MIXes (i.e. all the ones you will need to break), after the flags comes the "header". The header consists of a 16-bit integer that equals the number of files in the MIX, and a 32-bit integer that equals the size of the body of the MIX (the size of the MIX file minus the size of the header and index). These numbers are "big-endian", which means the bytes are arranged backwards (but the bits within each byte are arranged normally). (This is because the Intel x86 architecture is semi-retarded). So if bytes 04 and 05 (the number of files) are 0xB801, you need to rearrange this to 0x01B8 to get the actual number of files. (0x01B8 in decimal is 440, you can use Google Calculator to convert if necessary.)
sidenote: this means that MIX files larger than two gigabytes or containing more than 32,765 files are impossible under the RA2 engine.
So, here's the big picture so far: The first few bytes of the MIX are
byte 00: always 00 byte 01: is 01 if checksummed, 02 if encrypted, 00 otherwise. Should always be 00 on protected mixes. byte 02: always 00 (VK's Mix Protector will overwrite this with garbage) byte 03: always 00 (VK's Mix Protector will overwrite this with garbage) byte 04-05: big-endian number containing the number of files. byte 06-09: big-endian number containing MIX body size. (LeechKiller and VK's Mix Protector will both overwrite bytes 06 and 07 with garbage).
Then you have the index (12 bytes per index entry) and then the body.
All the "MIX protectors" do is add some number to the body size. XCC Mixer checks the body size first when reading a MIX, and rejects the MIX if the body size in the header is larger than the actual body size. (If I had the XCC source code, I could fix this easily, but I can't seem to find it on the XCC website, despite it being hosted on sourceforge. Strange.) It does this to avoid accessing unallocated memory and crashing with a segmentation fault.
Red Alert 2, on the other hand, apparently just ignores the body size in the header, and operates entirely on the information in the index. In theory, anyway. I suppose it's possible that "protected" mixes could cause RA2 to access memory out of bounds and crash, but I haven't seen it happen yet.
Anyway, VK's Mix Protector apparently just adds 2 to byte 06, causing XCCU to fail but making it easy to repair the MIX file. The XRML LeechKiller will add a random 16-bit value to bytes 06-07, instead, making it more difficult. To repair a "protected" MIX, you need to recalculate the body size and write it into bytes 06-09 with a hex editor.
To calculate the body size, take the actual file size by right-clicking on the file and clicking Properties. Windows should give you Size: n MB (x bytes). (Ignore "Size on Disk"). Take the number of bytes and apply the following formula:
BodySize = FileSize - ( (NumberOfFilesInMix * 12) + 10)
You see, the index has twelve bytes for each file, the header has four bytes for flags, two bytes for the number of files, and four bytes for the body size.
Don't forget, the body size and the number of files are all in big-endian form, and must be inserted as such.
Short form tutorial:
1) Grab your favorite hex editor and open the protected mix. 2) Ensure that the first four bytes (bytes 00 - 03) are 0x00000000. 3) Read the next two bytes (bytes 04-05) and reverse the order (e.g. b8 01 becomes 01 b8). This is the number of files. Convert it from hex to decimal. 4) Multiply the number of files by 12. Then add 10. We'll call this number H. 5) Take the file size. We'll call this number F. 6) F - H = B. 7) Convert B to hex. Pad the front with zeroes until it is eight hex digits long. Rearrange the bytes in big-endian order (e.g. 00 94 26 7F becomes 7F 26 94 00) 8) Overwrite the next four bytes (bytes 06-09) with this corrected value. 9) Save the MIX file and exit your hex editor.
Note: this will not recover the filenames of each file in the MIX. Those are gone forever.