Possible new version of SCI Companion

[Collector]

The only other SCI game I can think of that used CD Audio other than Jones was KQ6. And even that I think only the Windows version actually called it. The DOS version had a lo-fi audio equivalent in the resources. It certainly didn't sound like CD quality audio, anyway.

You are right in that the DOS version just used a lo-fi version. And not all releases of the CD included the CD-A track. This was probably just an oversight and the Windows game handles it by calling the lo-fi version when it can't find the CD-A track.

[troflip]

These are amazing. Awesome work. I'm guessing once you've worked out the flow control, you could generate the source in any format? e.g. original SCI vs. SCIStudio vs. Java-ish vs. <other language here>, etc.? - There's a lot of switch statements in there, isn't there? It seems like a lot of the scripts were based around that and state changes.

Yeah, it's fairly straightforward to generate in any format. I currently have code that outputs the underlying code structure in the 3 formats you mentioned, but I'm just focusing on the SCIStudio thing for now. And of course, recompiling it only works in SCIStudio-format now too. I'd do have a syntax parser for the java/c# style syntax - and it was mostly working - but I haven't picked up that code in a long time. In the end, I don't think java/c# is a great fit for SCI code, given the message-passing stuff that exists even down at the byte-code level.

As for "original SCI" syntax, that's on my radar, but isn't an immediate goal. It's similar enough to current SCIStudio syntax that writing the syntax parser might only be a few days work though. Although I'm not sure if having another "esoteric" language to code these games in is a good idea? I'd also like to have a firmer grasp on the exact syntax before bothering to get that working. And I'm not sure we will ever know...

Then again, the decompiled code might have to break a bit from SCIStudio syntax, and "original SCI" syntax could be a good way to formalize/clarify that break. One pretty frequent thing I see in the decompiled scripts is switch statements embedded in other expressions (for instance, passed as a parameter value to a function). SCI Studio doesn't handle these properly (to be fair, nor did SCI Companion prior to my latest updates), and the results crash the interpreter. It would be a bunch of extra work to restructure the code so that it is in a form that plays well with SCI Studio.

Another problem is with compound conditions (nested ands/ors). SCI Studio ignores any attempt to enforce a certain order of operations with parentheses. It basically just ignores parentheses (and so SCI Companion does this to right now, to ensure compatibility). But my decompiler generates compound conditions that need proper nesting. The only way to get the identical logic with SCI Studio syntax is to break things into multiple if statements, and I'm not sure whether I really want to bother with that extra work.

It is possible, and maybe even probable, that it wasn't updated as often and got out of sync sometimes, particularly I'm guessing when the same game is released on a newer interpreter version.

Yeah, vocab.999 isn't reliable (it's also missing from many games). For support for writing SCI1.1 games, I figure I'll finalize on a specific interpreter version (probably SQ5 or LSL6), and hardcode the set of kernels that work with that interpreter.

[troflip]

I'm curious to see some examples of assembly code that you haven't yet been able to identify. Maybe we could start a separate thread related to decompilation to discuss such examples. I'll have to grab my SQ5, if that is where you've been focussing.

I might do that eventually. In a couple of the recent examples I've come across, the actual correct code isn't too hard to figure out when looking at it by hand. It's just gaps in my decompilation algorithms. But I have encountered some truly perplexing byte code - either heavily optimized, or perhaps Sierra's parser also supported asm, and these functions were written in asm. At some point, I have to say "I've done enough work on this", and just make sure I can at least detect I wasn't able to decompile it properly, and fall back to asm. At this point, I'd say I have issues with maybe one out of every 20 or 30 functions or so.

Well, ok, here's one example. The method "advance" from SQ5 script 26.

Code: [Select]

    (method (advance) // method_0076
  0076:3f 04             link 4 // (var $4)
  0078:87 00              lap paramTotal
  007a:31 04              bnt code_0080
  007c:87 01              lap param1
  007e:33 02              jmp code_0082

        code_0080
  0080:35 01              ldi 1

        code_0082
  0082:a5 01              sat temp1
  0084:38 0084          pushi 84 // $84 indexOf
  0087:78               push1
  0088:67 20             pTos highlightedIcon
  008a:54 06             self 6

  008c:a5 02              sat temp2
  008e:8d 01              lst temp1
  0090:02                 add
  0091:36                push
  0092:63 14             pToa size
  0094:0a                 mod
  0095:a5 03              sat temp3

        code_0097
  0097:39 40            pushi 40 // $40 at
  0099:78               push1
  009a:8d 03              lst temp3
  009c:54 06             self 6

  009e:a5 00              sat temp0
  00a0:78               push1
  00a1:36                push
  00a2:43 06 02         callk IsObject 2

  00a5:31 2c              bnt code_00d3
  00a7:39 0e            pushi e // $e signal
  00a9:76               push0
  00aa:85 00              lat temp0
  00ac:4a 04             send 4

  00ae:36                push
  00af:35 04              ldi 4
  00b1:12                 and
  00b2:18                 not
  00b3:31 1e              bnt code_00d3
  00b5:39 07            pushi 7 // $7 nsLeft
  00b7:76               push0
  00b8:85 00              lat temp0
  00ba:4a 04             send 4

  00bc:36                push
  00bd:35 ff              ldi ff
  00bf:1e                 gt?
  00c0:2f 0b               bt code_00cd
  00c2:39 72            pushi 72 // $72 isKindOf
  00c4:78               push1
  00c5:51 40            class InvI
  00c7:36                push
  00c8:85 00              lat temp0
  00ca:4a 06             send 6

  00cc:18                 not

        code_00cd
  00cd:31 04              bnt code_00d3
  00cf:33 0f              jmp code_00e0
  00d1:33 c4              jmp code_0097

        code_00d3
  00d3:8d 03              lst temp3
  00d5:35 01              ldi 1
  00d7:02                 add
  00d8:36                push
  00d9:63 14             pToa size
  00db:0a                 mod
  00dc:a5 03              sat temp3
  00de:33 b7              jmp code_0097

        code_00e0
  00e0:38 00e1          pushi e1 // $e1 highlight
  00e3:7a               push2
  00e4:8d 00              lst temp0
  00e6:78               push1
  00e7:54 08             self 8

  00e9:48                 ret
    )

If you go to https://stamm-wilbrandt.de/GraphvizFiddle/ and paste the following in the box and hit "Draw", you'll see a graph of the flow structure of the function (before any control flow analysis has been done), which makes it easier to visualize.

Code: [Select]

digraph code {
n_0078_0 [label ="0078:\nlap\nbnt\n"] [color=red];
n_0076_0 -> n_0078_0 [color=red];
n_0076_0 [label ="0076:\nlink\n"] [color=red];
n_ffff_0 -> n_0076_0 [color=red];
n_0082_0 [label ="0082:\nsat\npushi\npush1\npTos\nself\nsat\nlst\nadd\npush\npToa\nmod\nsat\n"] [color=red];
n_0080_0 -> n_0082_0 [color=red];
n_007c_0 -> n_0082_0 [color=red];
n_00c2_0 [label ="00c2:\npushi\npush1\nclass\npush\nlat\nsend\nnot\nbnt\n"] [color=red];
n_00b5_0 -> n_00c2_0 [color=red];
n_0080_0 [label ="0080:\nldi\n"] [color=red];
n_0078_0 -> n_0080_0 [color=red];
n_00e0_0 [label ="00e0:\npushi\npush2\nlst\npush1\nself\nret\n"] [color=red];
n_00cf_0 -> n_00e0_0 [color=red];
n_007c_0 [label ="007c:\nlap\njmp\n"] [color=red];
n_0078_0 -> n_007c_0 [color=red];
n_00d3_0 [label ="00d3:\nlst\nldi\nadd\npush\npToa\nmod\nsat\njmp\n"] [color=red];
n_00c2_0 -> n_00d3_0 [color=red];
n_0097_0 -> n_00d3_0 [color=red];
n_00a7_0 -> n_00d3_0 [color=red];
n_0097_0 [label ="0097:\npushi\npush1\nlst\nself\nsat\npush1\npush\ncallk\nbnt\n"] [color=red];
n_0082_0 -> n_0097_0 [color=red];
n_00d3_0 -> n_0097_0 [color=red];
n_ffff_0 [label ="ffff:\nINVALID\n"] [color=red];
ENTRY_0 -> n_ffff_0;
ENTRY_0 [label ="Main 31"];
n_00cf_0 [label ="00cf:\njmp\n"] [color=red];
n_00c2_0 -> n_00cf_0 [color=red];
n_00b5_0 -> n_00cf_0 [color=red];
n_00a7_0 [label ="00a7:\npushi\npush0\nlat\nsend\npush\nldi\nand\nnot\nbnt\n"] [color=red];
n_0097_0 -> n_00a7_0 [color=red];
n_00b5_0 [label ="00b5:\npushi\npush0\nlat\nsend\npush\nldi\ngt?\nbt\n"] [color=red];
n_00a7_0 -> n_00b5_0 [color=red];
}



[MusicallyInspired]

The only other SCI game I can think of that used CD Audio other than Jones was KQ6. And even that I think only the Windows version actually called it. The DOS version had a lo-fi audio equivalent in the resources. It certainly didn't sound like CD quality audio, anyway.

You are right in that the DOS version just used a lo-fi version. And not all releases of the CD included the CD-A track. This was probably just an oversight and the Windows game handles it by calling the lo-fi version when it can't find the CD-A track.

Really? My copy of the King's Quest Collection Series KQ6 doesn't do that. In fact it errors out and crashes.

[Collector]

It has been years since I tried, but as I said, there were some releases that did not include the track. A couple of the standalone releases and the 2nd collection did not. I won't even mention the 2006 VU release. I don't remember any crashes with these. It would be a bit of a pain to test it. I would have to set up a 32-bit system or install Win3x in DOSBox to check the Win version.

[MusicallyInspired]

The VU release just uses DOSBox for all the games doesn't it? So it would just use the lo-fi version?

[Collector]

It includes the Win interpreter. Out of the box it will use the ancient version of DOSBox, but can be setup to use the Win. The point is that it does not include the CD-A track. My patch for the release originally used the Win interpreter until ScummVM supported the game.

[lskovlun]

Code updated

[Collector]

Code updated

Any chance for an exe version that can work from the command line?

[troflip]

Right now I'm working on the workflow for decompilation (since it will not be just quite as simple as click a button and go)

Attached is a dialog showing roughly what I've got so far. The scripts are listed on the left. You can decompile them individually, or as a group. Decompiling them will - in addition to generating the source file - generate an .sco file. The .sco files (as some of you know) contain the script variable names and public procedure names for scripts (so when compiling other scripts you can reference global variables, or public procedures in other scripts).

These names don't exist in the compiled code anywhere, so it will be up to the user to give them meaningful names (I can provide guesses for variable names depending on how they are used, but procedures are more difficult). So there is a little box where you can edit the variable and procedure names for an .sco (that's the treeview in the bottom center of the dialog), and those names will be picked up if you re-decompile the script.

On the top right is the generated source code for the selected script, and on the bottom right will be status/error messages.


Of note, this will also allow you to regenerate .sco files from compiled scripts (although you'd have to put in the correct var/proc names manually) - I seem to recall some threads in here where .sco files got corrupted, so this might help.


I also found that SCI1.1 games have a lot of scripts with invalid exports (they point to nothing). Currently in SCI0 with SCISTudio/SCICompanion, if you remove public procedures or instances from a script (or move them around in the file), other scripts will reference the incorrect procedure/instance until they are also recompiled. This probably causes confusion/headaches for anyone that hits this.

This is just a guess, but I suspect that Sierra (at least for SCI 1.1) may have "fixed" the index of a public proc/instance once it was created. If the programmer later removed some, they would be replaced by '0' entries in the export table, but the other exports would retain the same index. This should actually be pretty easy to do, by simply trying to match the names of exports to those existing in the .sco file.

[lskovlun]

Code updated

Any chance for an exe version that can work from the command line?

That was not in my plans, no.

[Collector]

Oh well. It would have been nice for batch processing.

[MusicallyInspired]

Anyone have any idea how we might re-enable the debug mode in the later SCI interpreters? Or is that something that we'd need a whole rebuilt interpreter itself for? Would the SQ4 beta come in handy here? (It has the debug mode enabled)

[troflip]

I was going to focus on SCI1.1, since I figured the more advanced the interpreter the better. But that's a good point, there is no internal debugger for any SCI1.1 games, right? That sucks.

That's actually going to make it really hard for me to validate my decompiler... I suspect I'll run into random crashes with no way to know what happened. I guess I'll have to test the decompiler mainly on SCI0 games, and just hope there aren't too many SCI1.1-specific issues.

Does SCUMMVM have a debugger? I'm pretty sure FreeSCI did, not sure if it made it through the move to SCUMMVM.

[Collector]

Debugging might be a factor in the choice of what game to base a new template on. Several SCI1.1 games still have debug enabled with the use of "magic files". SCI1 is another thing altogether. You might base it on one of the games that you can substitute the Xmas 90 ecard's SCIV.EXE. For later SCI1 removing the VERSION file from PQ3 is supposed to activate debug. As to SQ4, is there anyway to turn off debug?