ANM Files, SGS.DAT Compression, and MAP Editing

[黒い灯影]

ANM File Format - Just scroll down a lil

SGS.DAT Compression - Near the bottom

MAP Editing - Nearer the bottom

just to let you guys know
I've been working on the ANM file format since the last stone age (since 2 days ago)
also been trying to hack the DecSognaANM.GHP plugin, its just like a DLL file
to skip the "not supported" messge, and go ahead and uncompress a broken ANM file.

This is what i've found out about the ANM file so far
There is a color palette at the beginning , starting at offset 001E
thats shared across all the images stored within the ANM file
it goes Black,White, Blue(alpha), than darkest color to the brightest color


then right after that, is the Frame Count
if the image has an offset it will be listed right before the image pixel size
and right after that seems to be the image data, and is compressed
the header seems to be different for static images, and smaller offset images
The header of the data image is ALWAYS [OffsetX&Y, PixelSizeX&Y, ImageData]

UPDATED!!

you cant change the frame count, image pixel size, and any of the compressed stuff, it will give you an error and just crashes the SGS
you can only edit the the Color Palette and Offset


Here's a simple replacing of some of the colors


just need to figure out the Compression Algorithm,
to uncompress them, so we can edit the images(frames) and compress them back

[JG]

Have you tried swapping height and width to see what the effect is?  Doing so would not change the encoding bits (unless there's encoded line breaks of some form)  If it works the image might look slanted but it would give some clues as to which direction the pixels are filled (left vs right, top vs bottom)

Looking at the area you have shaded as "compressed"... well... it doesn't look compressed per se.  At best it looks like some from of run length encoding.  Real compression like Zip and Lzh would show sequences of slightly increasing values (for zip, the telltale sign is a sequence of bits that increase by 1 for each token)
Not familiar with any kind of RLE that looks like 0c0c 0c0c 0c0c 0c0c followed by 44 or 00ff though.  Off hand I'd suspect that 0c tranlsates into a stream of blue/alpha pixels though.

[黒い灯影]

well.. i did compare two different frames of the same pixel size
they were different length.. so i assumed it was compressed

i didn't try swapping the pixel height and width, only changing size of the image size, which cause the game to crash

... i'll add more on this later, as i have to get going now

[黒い灯影]

its very similar to one of the BMP format
ANM has Indexed Colors, I know that

I compared these three images, in ANM format

ANM Files, SGS.DAT Compression, and MAP Editing

all 3 are same pixel sized images
with 3 different data lengths

shortest being the first image
and the largest being the last image
this led me to believe that they were compressed
but im not that experienced with compressions
i would guess it is some weird form of RLE compression
instead of the length-byte starting first, its the data
and ending with the length

just tried swaping the pixel height and width, but it didnt work
----
just tried changing the length and height proportionally
took a frame size (640 * 400) doubled the height, and halved the width
this worked, im guessing it works in 4 pixel columns/ 4-pixel strips
if you zoom in on this image, and the image in the next post

ANM Files, SGS.DAT Compression, and MAP Editing

[黒い灯影]

Data gathered from BW.ANM
which has two frames,one completely Black,the other completely White

Code: [Select]

0A0A0A0A0A0A0A0A008F (1st Color = Black)
 0B0B0B0B0B0B0B0B008F (2nd Color = White)
 0C0C0C0C0C0C0C0C008F (3rd Color = Blue)inputs 1600 pixels per codeline (shown above)(4*400)
draws 4 colums of pixels(starting from the left)
(not sure if it starts from the top or bottom)

Example (black frame, 640x400 size)
0A0A0A0A0A0A0A0A008F is repeated 160 times

640 bytes total(+8 bytes, including offset and pixel size)per frame, if the frame is only a single color
repeated 160 times

TESTING:

ANM Files, SGS.DAT Compression, and MAP Editing

Replaced one line with

Code: [Select]

0C0C0C0C0C0C0C0C008F (3rd Color = Blue)then repeated the rest of the data with

Code: [Select]

0A0A0A0A0A0A0A0A008F (1st Color = Black)
Hex to decimal conversion
8F = 143  (might be useless info)

This might just be a while guess, but I think it may work in 4x4 blocks of data or in 4x2... i donno
i'll have to look into this more thou, before i can confirm it,  but now im having second thought about it thou
Well.. it works in 4-pixel strips, not blocks

[黒い灯影]

this proves it, its a weird form of RLE compression
with a weird set of rules

and the data is 4-pixel length/4 bytes


unless it RLE is in effect....
for the RLE compression to work, it has to follow these rules
if 0C0C0C0C is to be repeated, it needs
to be repeated twice

Code: [Select]

eg. [0C0C0C0C][0C0C0C0C]
This section & This section  wont work if the line isnt repeated twice

Code: [Select]

eg.
example 01: 0C0C0CFF0C0C0C0C - Wont Work
example 02: 0C0C515e0C0C0C0C - Wont Work
example 03: 1122334411223344 - Will Work
then must be followed by number of times repeated (can be 1 or 2 bytes) not sure about 3 bytes

Code: [Select]

example 01: 112233441122334401 - 1-byte RLE (01 creates an extra line, of 11223344, not 1122334411223344)
example 02: 1122334411223344008B - 2-byte RLE
example 03: 0C0C0C0C0C0C0C0C008F - 2-byte RLE (creates 400 lines of BluePixels (4 width x 400 height)
AND.. i think, It HAS to use the RLE compression, because I dont think it handles long repeating data that well
had a couple of crashes when i kept putting in 0C .. until i jammed another 0A in there somewhere, it was okay

AND.. the repeating lines CANT go past the bottom frame, it has to start a new RLE thing for the next 4-pixels over

NOTE: 0C is usually mapped to the Blue color, commonly used for alpha mapping

ANM Files, SGS.DAT Compression, and MAP Editing

with the data i gathered so far on the ANM format
i constructed my own ANM file, by grabbing one of the frame-data from 816.ANM, and editing the header
You can download it from Here
Its the same image of Cala from above

I think I'm almost cracked it, the only thing I need to figure out now is the 2-byte RLE compression

[JG]

I've solved it.

Its left to right, then top to bottom, in 4 pixel width columns as you've already specified.

If number >= 0Ah, subtract 0Ah and use that as your color index.  This means the maximum number of colors is restricted to 246 though - not sure if thats correct or not.  Its quite possible that 00 - 09 is used for color indices 246-255 but I haven't seen for sure yet.  I am sure that 00-09 does NOT indicate RLE.

A repeating sequence of 4 bytes indicates an RLE chain, followed by a one or two byte length code.  The RLE length code can be interpreted as follows:
If byte = 00h then, add 100h (256) and read a second byte.
If byte is not 00h, then add 1.

008B -> 100h + 8Ch -> 256 + 140 -> 396 (that's the height of your first 4 pixel wide column)
008F -> 100h + 90h -> 256 + 144 -> 400 (the height of the entire image)


Your image repeats 0A 0B 0C 0B (black, white, transparent, white) 396 times.  Then follows 16 bytes of non RLE pixels to complete the 4x4 grid at the bottom where her arm comes into picture.  Then you repeat your pattern again 391 times (0086 -> 256+135), then seven non-RLE lines (28 bytes) of her arm, followed by a 9 byte, 2 line RLE of skin color (0B 0B 0B 0B 0B 0B 0B 0B 01)


Thanks for laying the groundwork 黒い灯影.  With all the information in this thread I can create a converter between standard formats (bmp/gif/png) and ANM format.

[黒い灯影]

AWESOME!!!
I looked into the STF.ANM, which is the credits screen (with a size of 640x2800)
and changed one of the RLE lines, it looks like the maximum is repeating lines is 512 lines which is 00FF

Takes these codes to fill the whole 4-pixel strip (4x2800) with black

Code: [Select]

0A0A0A0A0A0A0A0A00FF
0A0A0A0A0A0A0A0A00FF
0A0A0A0A0A0A0A0A00FF
0A0A0A0A0A0A0A0A00FF
0A0A0A0A0A0A0A0A00FF
0A0A0A0A0A0A0A0AEF

[黒い灯影]

::SGS Archive Compression::

Right now I'm working on the Sogna's Compression algorithm, which seems easy enough
it's doesn't seem that complex, it seems it uses sorta of a dictionary compression, not RLE
it stores a pattern and repeats it, what ever part of it is needed(starting from the beginning of the pattern) it seems
it can also start from the middle of the stored pattern

If you look at this test SGS archive, It only contains 1 file, a renamed BW.ANM (which is a two frame animation, Black and White)

ANM Files, SGS.DAT Compression, and MAP Editing

been mapping out some of the patterns I've seen within this archive

Rundown of the color meanings:
Darkblue: SGS Archive Header
Green: Must be filled by a 00
Yellow: Acutal data
LightBlue: F0, F1, F2..etc pattern,  not sure
Red: FF Marker, not sure, but it seems to repeat every 11h bytes  or 17 bytes


when extracted these were the results (from the above SGS archive)

ANM Files, SGS.DAT Compression, and MAP Editing

ANM Files, SGS.DAT Compression, and MAP Editing

Disclaimer: These notes may be wrong,

UPDATE:
These notes were abit off, so look on page 2 of this thread, you'll see an updated version

[JG]

I'm making progress but I still have some bugs to work out. I can import about 20% of the ANM files I've tried now. Can't export anything as I end up with a slightly larger file size than I took in.  I suspect both issues involve RLE sequences that extend to the last line of the image and I'm probably not handling them right.  Give me another day or two to work this out.



What features my app will possess:
* Open ANM files to look at all the frames images [done, but buggy]
* Add new images to ANM files (enforcing the existing palette) [done]
* Create a new ANM file based upon a standard image file (creating the palette from that file) [done, but buggy]
* Add frames from one ANM file into another (enforcing the current palette) [done]
* Remove frames from ANM files [done]
* Change the order of frames in the ANM file [done]
* Modify the horizontal and vertical offsets of frames [done]
* Output to ANM file [done, but buggy]
* Output a single frame to standard format image or new ANM file [done]
* Output all frames at once to standard format images or new ANM files [done]
* Combine selected frames into a single composite image (no more manual composition of all those disparate bitmap files!) [composition done, but no option to save that combined image yet]

Reading ANM is very fast (under a second for most files), but saving to ANM format is terribly slow (several minutes for most files).  This is mainly due to .NET architecture, which gives you a not-too difficult method of creating a bitmap from an array of bytes (which would used in reading ANM), but no easy method to do the reverse (which would be helpful for writing ANM)


Also, the frame count field at byte 300h is probably only two bytes, not four.  I don't know what the bytes at 302h and 303h are used for yet, but I've come across an 80h in byte 303h a number of times and I've not been able to open any of them (the MAP*.ANM files seem to have alot of this.)  Its possibly some kind of flag to indicate a scaling factor or to signify extra bytes are to be used for the width and height of the image (as the maps are typically very large)  I haven't done any research to determine what yet.

[黒い灯影]

Nice, cant wait to get my grubby hands on that app, I'll do some more research on those extra bytes you talking about

I did some research on it.. i'm thinking it's a flag, if songa's RLE compression is used or not, 
00 = RLE
80 = No RLE

this is just a wild guess, as I didnt see any RLE compression used in the MAP files

ANM Files, SGS.DAT Compression, and MAP Editing

UPDATE!!
I've just confirmed it, it works on the binary level, the 80h, at 303h, activates a single bit, that tells the game, if it's compressed
also, the byte order seems to change, when you flag it (80h), from right to left, top to bottom, and the whole width of the image

It also works with images that have been exported to BMP, and converted back into an uncompressed ANM (editing header)
along with the 80h bit at 303h, i edited this image in Photoshop, added Alpha in the window

ANM Files, SGS.DAT Compression, and MAP Editing

[黒い灯影]

:: EDITING MAPS ::

Editing the MAP files... theres no compression at all, Simply change the header and copy over the color Palette
Change it into an OS/2 Bitmap... and open it into Photoshop, it should be upside down, Invert it vertically
Edit at will, Invert, save... then convert it into a ANM (editing the Header and all)

Check out an example I made Here

NightMap

ANM Files, SGS.DAT Compression, and MAP Editing

JG00 should easily be able to create a converter for these files, as they are just normal OS/2 BITMAPs in disguise

[Echelon]

Hey 黒い灯影, you made me come back! Heh.

So does this mean eventually you guys will be able to uncensor everything?

[黒い灯影]

awesome! hahah...
well.. this is the first step towards it.. so maybe sometime in the future we'll be able to see an unofficial uncensored version of Viper RSR, who knows maybe an unofficial extended version, or a completely new game even, using the SGS.exe as a base

Hyperguy as the main character... lol

I don't know if you knew this, but I'm also working on the game's WIN scripts, I'm just decoding them into my own human-readable language
If you check my site, under the forums, I've put down two game scripts, START.WIN, and MAIN.WIN
if you look at these forums, you should see a topic called "BlackShadow's Research" or something along that lines, you'll see I did some work with the other scripts as well

[JG]

I've solved both my import and export issues, and now that I've handled the uncompressed flag as well, I can read and write just about any ANM file I want.  I need to run it through a wider set than the 15 to 20 images I've tried though.  I'll make a test app that will try to decode every file in a directory one after the other, and also try it on stuff other than RSR files, to give the algorithm a solid work out.

Among other victories today... I've modified my code to internally work with 8 bits per pixel images instead of 32.  (You can still import high-fidelity frames, but they'll be reduced to 256 colors.)  Not only does it reduce memory consumption by 300%, but it creates smaller extract files, avoids dithering in export to GIF format, and preserves transparency information in export to both GIF and PNG formats.  (No more blue backgrounds on our images!)

I do need to figure out how to get direct access to the bitmap byte array so I can do the export to ANM quicker though.  Its intolerably slow right now.

Its close, you guys will have a version no later than the weekend.  黒い灯影, you may get a beta tomorrow.

[黒い灯影]

awesome! cant wait to try it out!

I was looking at the hex early this morning, and it just sorta hit me, i thought "offsets"
I believe I figured out how the compression is stored, its my own SGS.DAT archive (which means its a very simple file)
so now I'll have to move on to the more complex compressed files

Sogna's Acrhive compression uses some sort of callback method,
which I will now call SognaArchivedCallback Compression or SAC Compression for short, lol (sounds painful, i know)
thats what I would like to call it, what do you guys think it should be called?
since Sogna is not around anymore, we have to call it something

Like this for example, a line from the image below 11F0
You just need to separate the combined data, most likely works at the binary level, to retrieve the "0F"

11F0 --> F011 --> 0F + 0011
Add 1 to 0F (0F + 1 = 10h bytes)
Repeat 16 bytes of data from offset 0011h behind
(this would repeat 10h bytes of 00's)

ANM Files, SGS.DAT Compression, and MAP Editing

Code: ("SAC Compression Rules") [Select]

AllocateBytes must be inserted after every 8 bytes of Commands or Data (herein referred to as Blocks)
If CallBack is used within the next 8 Blocks, Must equal to
If its not used within the next 8 Blocks, must be 00

Values used in image
00h/FFh/1Fh/20h/F5h/FEh

Block = a command (eg. 51F0h = 1 Block,  15F125F1h = 2 Blocks), or Data (eg, C0C0C0C0 = 4 Blocks, 1122 = 2 Blocks)
--------------------------------------------
25F035F0FF45F055F065F075F085F095F0A5F0B5F0FF
Converts into

25F0 >F025 > 0F + 0025
35F0 >F035 > 0F + 0035
FF AllocateBytes
45F0 >F045 > 0F + 0045
55F0 >F055 > 0F + 0055
65F0 >F065 > 0F + 0065
75F0 >F075 > 0F + 0075
85F0 >F085 > 0F + 0085
95F0 >F095 > 0F + 0095
A5F0 >F0A5 > 0F + 00A5
B5F0 >F0B5 > 0F + 00B5
FF AllocateBytes
--------------------------------------------
(0F + 0025)
From Offset 0025 behind
Repeat sequence up to (0Fh + 01h = 10h bytes)
another way of writing it(0x0F + 0x01 = 0x10 bytes)

eg.
7+1 = 08h bytes
A+1 = 0Bh bytes
F+1 = 10h bytes
--------------------------------------------
but there's still one thing that doest quite add up to these rules
the first 43h Allocation... the only way I think it can make sense is adding the first and second 00's  between the DATA
which would equal 45h, then it would fit the first FF Allocation

the first part uses alot of callbacks from offset 0000, so the first 8 bytes of data, is repeated often, until it hits the first frame data

[wsw1231]

hi JG00, where can I download Sogna ANM Editor?

[JG]

hi JG00, where can I download Sogna ANM Editor?

If you'd read the posts instead of looking at the pictures, you'd see I said its not ready for public use yet, but its getting close.  Its only working on about half of the RSR files so I need to figure out where bugs are in the algorithm so I can get that rate closer to 100%.   I can only work on this a couple of hours a day so I probably won't get it finished until this weekend.  Please be patient.

On the bright side, I've figured out direct access to an existing bitmap's data so I can save to ANM format quickly now (thus improving encoding rate from about 5,000 pixels per second to about 1 million)

Oh, and I've got a built in method to view/edit the palette now, too.

[黒い灯影]

looking good so far
do you think you can implement sort of a saving to an animation, some time in the future?

[JG]

do you think you can implement sort of a saving to an animation, some time in the future?

There's two big problems in that regard:

1. Microsoft's .NET GIF codec provides no nice and easy method to save a multiple frame image.  It can read them but not write them without jumping through hurdles. As I understand it, you have to save each frame one a time, and it'll write the full palette and image data for each frame, instead of optimizing by using the master palette and transparent pixels on unchanged areas between frames.  I have no idea if it writes frame delay blocks between the image blocks either; its liable to create frame after frame with 0 centiseconds between them.  Don't ask me to write my own GIF encoder to get around these issues.

2. The ANM file contains no timing or self composition information, so the user would have to pick the two or three images that constitute a frame, and then decide that the frame delay is going to be (which may or may not be accurate) then click a button to add that image and delay to the file.  It could never be a magic button that magically creates a animated GIF that has the right frames and is timed right.


I'm content to be able to compose individual frames, save them to a transparent GIF or PNG, and then use a dedicated program like JASC's Animation Shop (oldie but goodie) to build an animated GIF while doing absolutely no individual frame editing.  Right now we have these 24bpp bitmaps with blue backgrounds that aren't aligned properly to where the image is supposed to be in the frame: meaning we have to reduce to 256 colors, convert blue to transparent, and then combine all the layers into a single image.  Rinse and repeat for every frame in the animation.  Then go into animation shop and combine all the frame images you just created into an animated GIF.  Takes several minutes for a short repeating sequence, maybe a hour for more complicated scenes.  I hope to cut that down to 1 or 2 minutes with my app.