PC98 - CTR files

[黒い灯影]

Okay, so i've been poking around in the PC98 files, now that I know more about reverse engineering.

Here are some of my notes:

SGSFILE.COM - is the main SGS.DAT file loader, stays in memory, loads chunks into memory, looks like 8kb blocks (actually SGS requests 8kb blocks)
SGS.COM - main game engine

SGS.COM hands off data load requests to SGSFILE.COM to read in data from the SGS.DAT file.

The opcodes are different in the PC98 version of the SGS engine, they don't seem to match up with the WIN version.
-I only followed the flow of data through the first couple of bytes so i didn't really see how it handled the end of the file.
-I don't know if this handles the other file types the same way, i'm guessing it might.  Only tested on the STARTUP.SZH file which is ID'd as 'SIL'

I found the section that processes the files after they've been read in.
Data flow decryption
I might have missed a section in the middle, when DX reached 0

Code: [Select]

STARTING POINT===================

RAW DATA: Start of STARTUP.SZH [data read in started after FileIdentifier'SIL']
1400 11C4 1110 9C18 4111 8E81 8000

REG:
EAX=00002000  EBX=00000005  ECX=00002000  EDX=0000e000
ESP=00008376  EBP=60002000  ESI=0000e000  EDI=0000a000
DS=263f  ES=263f  SS=263f  CS=263f  EIP=00001477 

ASM: @1477 
mov     dl, 7 ; DX = e007   
lodsb ; READ BYTE from DS:SI into AX (RB==14h) ;AX=2014
dec     bp              ; decrease counter? ;BP = 2000 > 1FFF
mov     dh, al ; DX = RB07  ; RB==ReadByte; ;DX = 1407
shl     dh, 1    ; ;DX = 2807
==================
File Data Manipulation:
VV
1400 11C4 1110 9C18 4111 8E81 8000

0x14 ==> 0x28
0x00 ==>
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x00]
mov     cl, 8 ; CX = 2000 > 2008 ; DX = 0007
sub     cl, dl ; CX = 2008 > 2001
xor     ah, ah ; AX = 2000 > 0000
shl     ax, cl      ; AX = 0000 > 0000
or      ah, dh    ; DX = 2807 ; AX = 2800
mov     dh, al ; DX = 2807 > 0007
mov     al, ah ; AX = 2800 > 2828
dec     bp ; BP = 1FFF > 1FFE
retn
==================
stosb ; Store byte into  ES:DI [AL = 0x28]
==================
File Data Manipulation:
  VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
28
===================================

lodsb                   ; Reads from DS:SI into AX [Read byte: 0x11]
mov     cl, 8 ; CX = 2001 > 2008 ; DX = 0006
sub     cl, dl ; CX = 2001 > 2002
xor     ah, ah ; AX = 2811 > 0011
shl     ax, cl      ; AX = 0011 > 0044 ; DX = 0006
or      ah, dh    ; AX = 0044 > 0044
mov     dh, al ; DX = 0006 > 4406
mov     al, ah ; AX = 0044 > 0000
dec     bp ; BP = 1FFE > 1FFD
retn
==================
stosb ; Store byte into ES:DI [AL = 0x00]
==================
File Data Manipulation:
     VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800
==================
dec     dl              ;DX = 4406 > 4405
shl     dh, 1 ; ;DX = 4405 > 8805
retn
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0xC4] ; AX = 0000 > 00C4
mov     cl, 8 ; CX = 2002 > 2008 ; DX = 8805
sub     cl, dl ; CX = 2002 > 2003
xor     ah, ah ; AX = 00C4 > 00C4
shl     ax, cl      ; AX = 00C4 > 0620 ; DX = 8805
or      ah, dh    ; AX = 0620 > 8E20
mov     dh, al ; DX = 8805 > 2005
mov     al, ah ; AX = 8E20 > 8E8E
dec     bp ; BP = 1FFD > 1FFC
retn
==================
stosb ; Store byte into ES:DI [AL = 0x8E]
==================
File Data Manipulation:
       VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E
==================
dec     dl              ;DX = 2005 > 2004
shl     dh, 1 ; ;DX = 2004 > 4004
retn
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x11] ; AX = 8E20 > 8E11
mov     cl, 8 ; CX = 2003 > 2008 ; DX = 4004
sub     cl, dl ; CX = 2008 > 2004
xor     ah, ah ; AX = 8E11 > 0011
shl     ax, cl      ; AX = 0011 > 0110 ; DX = 4004
or      ah, dh    ; AX = 0110 > 4110
mov     dh, al ; DX = 4004 > 1004
mov     al, ah ; AX = 4110 > 4141
dec     bp ; BP = 1FFC > 1FFB
retn
==================
stosb ; Store byte into ES:DI [AL = 0x41]
==================
File Data Manipulation:
          VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41
==================
dec     dl              ;DX = 1004 > 1003
shl     dh, 1 ; ;DX = 1003 > 2003
retn
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x10] ; AX = 4141 > 4110
mov     cl, 8 ; CX = 2004 > 2008 ; DX = 2003
sub     cl, dl ; CX = 2008 > 2005
xor     ah, ah ; AX = 4110 > 0010
shl     ax, cl      ; AX = 0010 > 0200 ; DX = 2003
or      ah, dh    ; AX = 0200 > 2200
mov     dh, al ; DX = 2003 > 0003
mov     al, ah ; AX = 2200 > 2222
dec     bp ; BP = 1FFB > 1FFA
retn
==================
stosb ; Store byte into ES:DI [AL = 0x22]
==================
File Data Manipulation:
            VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 22
==================
dec     dl              ;DX = 0003 > 0002
shl     dh, 1 ; ;DX = 0002 > 0002
retn
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x9C] ; AX = 2222 > 229C
mov     cl, 8 ; CX = 2005 > 2008 ; DX = 0002
sub     cl, dl ; CX = 2008 > 2006
xor     ah, ah ; AX = 229C > 009C
shl     ax, cl      ; AX = 009C > 2700 ; DX = 0002
or      ah, dh    ; AX = 2700 > 2700
mov     dh, al ; DX = 0002 > 0002
mov     al, ah ; AX = 2700 > 2727
dec     bp ; BP = 1FFA > 1FF9
retn
==================
stosb ; Store byte into ES:DI [AL = 0x27]
==================
File Data Manipulation:
               VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227
==================
dec     dl              ;DX = 0002 > 0001
shl     dh, 1 ; ;DX = 0001 > 0001
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x18] ; AX = 2727 > 2718
mov     cl, 8 ; CX = 2006 > 2008 ; DX = 0001
sub     cl, dl ; CX = 2008 > 2007
xor     ah, ah ; AX = 2718 > 0018
shl     ax, cl      ; AX = 0018 > 0C00 ; DX = 0001
or      ah, dh    ; AX = 0C00 > 0C00
mov     dh, al ; DX = 0001 > 0001
mov     al, ah ; AX = 0C00 > 0C0C
dec     bp ; BP = 1FF9 > 1FF8
retn
==================
stosb ; Store byte into ES:DI [AL = 0x0C]
==================
File Data Manipulation:
                 VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227 0C

==================
dec     dl              ;DX = 0001 > 0000
shl     dh, 1 ; ;DX = 0000 > 0000
retn
==================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x41] ; AX = 0C0C > 0C41
dec     bp ; BP = 1FF8 > 1FF7
retn

==================
stosb ; Store byte into ES:DI [AL = 0x41]
==================
File Data Manipulation:
                    VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227 0C41

================= (DL == 0)
loc_11477:             
mov     dl, 7 ; DX = 0000 > 0007
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x11] ; AX = 0C41>0C11
dec     bp          ; BP = 1FF8 > 1FF7
mov     dh, al ; DX = 0007 > 1107
shl     dh, 1 ; DX = 1107 > 2207
retn
==================

File Data Manipulation:
                      VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227 0C41


================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x8E] ; AX = 0C11 > 0C8E
mov     cl, 8 ; CX = 2006 > 2008 ; DX = 2207
sub     cl, dl ; CX = 2008 > 2001
xor     ah, ah ; AX = 0C8E > 008E
shl     ax, cl      ; AX = 008E > 011C ; DX = 2207
or      ah, dh    ; AX = 011C > 231C
mov     dh, al ; DX = 2207 > 1C07
mov     al, ah ; AX = 231C > 2323
dec     bp ; BP = 1FF8 > 1FF7
retn
==================
stosb ; Store byte into ES:DI [AL = 0x23]
==================
File Data Manipulation:
                         VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227 0C41 23
==================
dec     dl              ;DX = 1C07 > 1C06
shl     dh, 1 ; ;DX = 1C06 > 3806
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x81] ; AX = 2323 > 2381
mov     cl, 8 ; CX = 2001 > 2008 ; DX = 3806
sub     cl, dl ; CX = 2008 > 2002
xor     ah, ah ; AX = 2381 > 0081
shl     ax, cl      ; AX = 0081 > 0204 ; DX = 3806
or      ah, dh    ; AX = 0204 > 3A04
mov     dh, al ; DX = 3806 > 0406
mov     al, ah ; AX = 3A04 > 3A3A
dec     bp ; BP = 1FF7 > 1FF6
retn
==================
stosb ; Store byte into ES:DI [AL = 0x3A]
==================
File Data Manipulation:
                           VV
1400 11C4 1110 9C18 4111 8E81 8000
Stored:
2800 8E41 2227 0C41 233A
==================
dec     dl              ;DX = 0406 > 0405
shl     dh, 1 ; ;DX = 0405 > 0805
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x80] ; AX = 3A3A > 3A80
mov     cl, 8 ; CX = 2002 > 2008 ; DX = 0805
sub     cl, dl ; CX = 2008 > 2003
xor     ah, ah ; AX = 3A80 > 0080
shl     ax, cl      ; AX = 0080 > 0400 ; DX = 0805
or      ah, dh    ; AX = 0400 > 0C00
mov     dh, al ; DX = 0805 > 0005
mov     al, ah ; AX = 0C00 > 0C0C
dec     bp ; BP = 1FF6 > 1FF5
retn
==================
stosb ; Store byte into ES:DI [AL = 0x0C]
==================
File Data Manipulation:
                              VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C

==================
dec     dl              ;DX = 0005 > 0004
shl     dh, 1 ; ;DX = 0004 > 0004
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x00] ; AX = 0C0C > 0C00
mov     cl, 8 ; CX = 2003 > 2008 ; DX = 0004
sub     cl, dl ; CX = 2008 > 2004
xor     ah, ah ; AX = 0C00 > 0000
shl     ax, cl      ; AX = 0000 > 0000 ; DX = 0004
or      ah, dh    ; AX = 0000 > 0000
mov     dh, al ; DX = 0004 > 0004
mov     al, ah ; AX = 0000 > 0000
dec     bp ; BP = 1FF5 > 1FF4
retn
==================
stosb ; Store byte into ES:DI [AL = 0x00]
==================
File Data Manipulation:
                                VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00
==================
dec     dl              ;DX = 0004 > 0003
shl     dh, 1 ; ;DX = 0003 > 0003
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0xA0] ; AX = 0000 > 00A0
mov     cl, 8 ; CX = 2004 > 2008 ; DX = 0003
sub     cl, dl ; CX = 2008 > 2005
xor     ah, ah ; AX = 00A0 > 00A0
shl     ax, cl      ; AX = 00A0 > 0000 ; DX = 0003
or      ah, dh    ; AX = 00A0 > 1400
mov     dh, al ; DX = 0003 > 0003
mov     al, ah ; AX = 1400 > 1414
dec     bp ; BP = 1FF? > 1FF1
retn
==================
stosb ; Store byte into ES:DI [AL = 0x14]
==================
File Data Manipulation:
                                   VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00 14
==================
dec     dl              ;DX = 0003 > 0002
shl     dh, 1 ; ;DX = 0002 > 0002
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x04] ; AX = 1414 > 1404
mov     cl, 8 ; CX = 2005 > 2008 ; DX = 0002
sub     cl, dl ; CX = 2008 > 2006
xor     ah, ah ; AX = 1404 > 0004
shl     ax, cl      ; AX = 0004 > 0100 ; DX = 0002
or      ah, dh    ; AX = 0100 > 0100
mov     dh, al ; DX = 0002 > 0002
mov     al, ah ; AX = 0100 > 0101
dec     bp ; BP = 1FF1 > 1FF0
retn
==================
stosb ; Store byte into ES:DI [AL = 0x01]
==================
File Data Manipulation:
                                     VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00 1401
==================
dec     dl              ;DX = 0002 > 0001
shl     dh, 1 ; ;DX = 0001 > 0001
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x2A] ; AX = 0101 > 012A
mov     cl, 8 ; CX = 2006 > 2008 ; DX = 0001
sub     cl, dl ; CX = 2008 > 2007
xor     ah, ah ; AX = 012A > 002A
shl     ax, cl      ; AX = 002A > 1500 ; DX = 0001
or      ah, dh    ; AX = 1500 > 1500
mov     dh, al ; DX = 0001 > 0001
mov     al, ah ; AX = 1500 > 1515
dec     bp ; BP = 1FF0 > 1FEF
==================
stosb ; Store byte into ES:DI [AL = 0x15]
==================
File Data Manipulation:
                                        VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00 1401 15
==================
dec     dl              ;DX = 0001 > 0000
shl     dh, 1 ; ;DX = 0000 > 0000
retn
================
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x00] ; AX = 1515 > 1500
dec     bp ; BP = 1FEF > 1FEE
retn
==================
stosb ; Store byte into ES:DI [AL = 0x00]
==================
File Data Manipulation:
                                          VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00 1401 1500

================= (DL == 0)
loc_11477:             
mov     dl, 7 ; DX = 0000 > 0007
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x0C] ; AX = 1500>150C
dec     bp          ; BP = 1FEE > 1FED
mov     dh, al ; DX = 0007 > 0C07
shl     dh, 1 ; DX = 0C07 > 1807
retn
================== (DL > 0)
lodsb                   ; Reads from DS:SI into AX [Read byte: 0x00] ; AX = 150C > 1500
mov     cl, 8 ; CX = 2007 > 2008 ; DX = 1807
sub     cl, dl ; CX = 2008 > 2001
xor     ah, ah ; AX = 1500 > 0000
shl     ax, cl      ; AX = 0000 > 0000 ; DX = 1807
or      ah, dh    ; AX = 0000 > 1800
mov     dh, al ; DX = 1807 > 0007
mov     al, ah ; AX = 1800 > 1818
dec     bp ; BP = 1FED > 1FEC
==================
stosb ; Store byte into ES:DI [AL = 0x18]
==================
File Data Manipulation:
                                               VV
1400 11C4 1110 9C18 4111 8E81 8000 A004 2A00 0C00 00AF 1000 1560 000A 8044 4000 2AFC 0605 5800 02A0 2530 0046 B100
Stored:
2800 8E41 2227 0C41 233A 0C00 1401 1500


[JG]

Good luck with this. I took the knowledge learned about the Windows .WIN and .ANM file makeup and it didn't translate directly to the PC98 code. Its possible that SGSFILE program does a simple decryption of the data loaded, like an XOR mask. If it does have an XOR mask and we can figure it out, it might then line up to the Windows code.

[黒い灯影]

Okay.. so I started working on my decryption code...
And.. it only partially works, it goes awry after a few loops, so I'm still missing something.
but I am having a bit of success, as I can see some text.. but no JP dialogue yet from the SIL files.

Also tried my program on some ANM files, and saw some patterns..
but its too early to say anything other than a few things match up.

[黒い灯影]

Decryption Decompression Program complete
okay, so the files are actually compressed, not encrypted

The SFX files contain the name DJ TOMZAO or TOMAZO, so that's pretty neat.

I attached some of the files I ran through my program from CTR,
and also I threw in some Viper BTR files, and it also worked for those
If you wanna look through them and make any comparisons between WIN and PC98 versions of the files.

Also, the text seems to be stored in a strange format, i'll have to see how the SGS engine handles it.

[JG]

You aren't using the right code page to decode the text. PC98 uses JIS X 208, which I think is Windows code page 20932.
For example "0x23 0x48" converts to wide letter H, and appears as "#H" in the text view of your hex. In other words, the "#H#A#P#P#Y#!!#E#N#D" you see is merely wide-character "HAPPY END" in JIS X.

[黒い灯影]

hmmm, that's interesting, I seem to be having trouble displaying that in any hex editor I use 
must be a platform thing, I guess i'll skip that issue for now, and continue to work on
identifying the opcodes and the data they consume in the script for now.

[JG]

I think that codepage has to be manually added. Shift-JIS is the default codepage installed when you add the Japanese language in Windows.
Even if you didn't have it installed you could get a mapping table to convert it to Unicode. (I could even create one for you if needed badly enough.)

[黒い灯影]

Actually.. I was looking around in what character encoding Java supports.. and I saw that it actually had support for JIS X 0208, and was able to see the JP text from the SIL scripts.

[黒い灯影]

PC98 SGS has two image formats, one is MSV and the other ANM.

I'm getting closer to getting the MSV format.
Apparently it fills pixels up from the top, single column on the Y axis,
then advances on the X once maxed height is reached
not entirely sure what the first byte/short is... but the 2nd short is the max height, i believe.

Code: [Select]

5000 9001 0000 008F 0000 008F 0000 008F
0000 008F 0000 008F 0000 008F 0000 008F
0000 008F 0000 008F 0000 008F 0000 008F
a double "00", after the 9001, triggers a skip reading in bytes for the RGB values, leaving them at 0x00.
It copies over the RGB values,
after that, one of 2 things can happen,

• if it reads a 0x00, then it will read the next byte, 0x8F then 0x100&0x8F = 0x18F
  and repeat the last RGB values that many times, 0x18F or 399.  so that tells me the fill order of the pixels.
  repeats last operation that many times. 
• if it reads a value, say 7E, then it takes that as the repeat value.
  

Code: [Select]

0000 008F 0000 7E01 0000 0102 0000 0304
This is just speculation, though

This pic was my first attempt at just looking at the data and assuming how the data is laid out.

[JG]

Guess, but...

50 00 -> width of 0x0050, or 80 pixels
90 01 -> height of 0x0190, or 400 pixels
00 -> color index (black/transparent)
00 -> A
00 -> B
8F -> C

If A is non zero, counter = A
If A is zero and B is non-zero, counter = (A + 1) * B - not so sure as don't have any examples here.
If A is zero and B is zero then counter = 256 + C + 1

00 00 8f thus becomes 01 90 (400) pixels of color 0. In other words, the repeats 00 00 00 8F several times draw vertical black/transparent lines that fill image from top to bottom along one of the sides, because the image has black or transparent area on the left (or right, but probably left.) I'm fairly confident this is what we're seeing in all the 00 00 00 8F repeats.

After that though, I'm less sure.

00 -> color index 0
00 7E -> draw 0x17F (383) pixels?
01 -> color index 1
00 00 01 -> draw 0x102 (258) pixels?
02 -> color index 2
00 00 03 -> draw 0x103 (259) pixels?
04 -> color index 4


It's possible the fifth byte in the sequence identifies which color index is transparent. If so the repeating sequence isn't 00 00 00 8F but 00 00 8F 00 instead with the color index coming after the counter bytes instead of before. But it falls into the same problem pretty quickly.

00 7E -> counter of 17F (383)
01 -> draw 383 pixels of color 1
00 00 01 ...

[黒い灯影]

Just wanted to share the algorithm that processes the data in the MSV files,
unsure if its the same code that processes ANM files too, at this point.

This is what i'm analyzing right now

The first ReadByte block would be the one reading the first byte of 0000 008F

[黒い灯影]

Here's an example of the data flow of one column, this expands the data out to 400 bytes,
i'll have to find out where the palette is stored, since these are most likely indexed colors.
so it has XOR operations, 3 variables for bytes read(al, ah, dl), counter(bx), countdown(from 400)

Repeat:			R						R				R				R				R				R
ReadValue:	00	00	7e	01	00	02	00	00	02	04	00	00	05	04	00	00	02	02	00	00	01	01	00	00	FB
WriteValue:	00	-	00 * 126	01	01	03	03	-	03 03	07	07	-	07 07 07 07 07 07	03	03	-	03 03	01	01	-	01	00	00	-	00 * 251

[黒い灯影]

Almost getting it, just have to fix up my MSV decompression algorithm a bit, and find the indexed color palette.
Right now i'm just taking the index values of a pixel, and using those as the 3 values in RGB.

Unless thats the color palette in at the end of the MSV files.

[黒い灯影]

just some extracted data