Possible new version of SCI Companion

[Collector]

I wanted this to be not just for game development, but also for creating patches and other fixes for existing games as well as to help archivist like for the Sierra Museum and the Art of Sierra. NewRisingSun created an number of patches for the many timer related bugs on the SCI1.1 era, but there are still other bugs he missed. Enrico Rolfi made some tools for translating SCI games into other languages. He did the GK2 Italian translation. I have made a few patches for Freddy Pharkas, where I have corrected defects in a few view resources. There were also a number of the audio resources that needed to be equalized. These are included with my FPFP installer. I want to include some basic tools for such efforts. The IDE does include a hex editor to this end. A script disassembler would be a great addition for such purposes.

Brandon Klassen of the Sierra Museum and the Art of Sierra has been needing an easier way to access resources in the SCI archives. SV only lets you extract or export resources one at a time.

But it is not just for extraction. I want to be able to add resources to a game by simply dropping them into the project explorer. Being able to export a picture resource to work with it in an external image editor and then reimport into the game would be great for SCI1 and later games. For a new SCI+ game a background artist could use the old Sierra technique of scanning hand painted backgrounds and dropping the lot of them into the project explorer


NSIS is one of the standard installer packages that is used for quite a few companies. And yes, it is what I used for the installer for Betrayed Alliance. It is what I have used for all of my installers. I also used it for my installers for the Sunlight Games rerelease of Gold Rush!. The scripting language has been described as a cross between PHP and assembly. It is extremely powerful and flexible. Most of my installers are for old DOS games and automate setting them up to run in DOSBox.

There are a number of features that I did not mention. The debug feature will automatically run the game directly from the IDE using a debug build of DOSBox. I have incorporated a few wizards, like a new game wizard that prompts the user for things like a game name, which template he wants to use (the SCI Studio, Cloudee's PnC template or one that uses Gumby's sciAudio of hopefully SCI1, If we can add support for later versions of SCI). The wizard will then copy the right template files to new project's folder and edit the scripts to add the game's name. The IDE is fairly customizable. It also includes a launch bar to the right of the main form's toolbar that you can drag and drop programs or shortcuts to external tools, websites etc.

The game's settings file is in XML and the import function will parse the game.ini from a Studio or Companion project to XML. I started the GUI with saving program settings to the Registry, but have been migrating them to XML.

I have been updating the abandoned Visual AGI, an old IDE in C# that was started a number of years ago for the authors to learn dotNET. I was able to fairly easily fold this into Developer so it can now load AGI games and open & edit AGI view and picture resources.

We are not ready to publicly release it, but send me PM if you want to take a look at it.

[troflip]

Well it sounds like you're targeting slightly different scenarios (more modding/patching than building games from scratch?), so I wouldn't want you to abandon this project.

It's possible I'll be able to structure the SCI Companion code so that lower level resource-manipulation stuff is in a separate library that could potentially be used by your project sometime in the future. That might be unrealistic though.

As an aside, does anyone know why later Sierra games had so many resources in patch files instead of in the resource.map? Like for instance, Space Quest 5 has about 40 views (v56 files) in the game folder. Are these really just patches that shipped afterwards? Or were they there from the start? Or do I have some "hacked" version of SQ5?

[Collector]

It is definitely intended for game creation, too, just that the scope goes beyond.

Are you using manged or unmanged code? It is a little more involved using unmanged in C#. How hard would it be to port your script disassembler to C#?

That is normal for SQ5.

I suspect that the separate patch files had more to do with the increasingly complexity of the games made for more bugs. Many bugs would have not been revealed until the last moment (some not until after the initial release). In the case of SQ5 many of those were for localization purposes. The patches were just dropped in at the last moment or repackaged for a 2nd release. Otherwise the installation package would have had to be completely redone. By repacking the patches into the archive they would have had to resplit the resource archive and possibly alter the the concatenation part of the install script. A patch file requires no alteration of the install.scr file, just drop the patch in the master copy. the install scripts used a wild card (copy %2:*.*) for the file copy part to copy all of the files from the diskette. The only alteration that might have been required would have been if the patch files added and extra disk to the distribution package. Even that would be a smaller, quicker edit than having to alter the concatenation part of the script. Remember, there was a lot of pressure to get product out of the door.

[troflip]

Ooohhhh I am so psyched for this! I can finally properly continue working on SQ1Retro. I was stuck working with two 256 View editors, one would not preserve or set the origin point and the other would screw up the palette by making all black colours white. I assume your editors don't have these problems?

Is it also possible to match an imported background's palette to a pre-existing SCI palette instead of creating a new one? How does it currently work? Do the imported backgrounds automatically set aside the first 64 colours in the palette for sprites and simply use the remaining 192 only? That's how Sierra's backgrounds worked. Otherwise sprites will look wrong in-game.

Do you have any information how people generally edit palettized images? What common file formats are used to save these images? (For instance, I don't think jpeg support palettes, right?). Are there separate palette files that are commonly used?

[Collector]

You would never want to use JPG or any lossy format. BMP or GIF would be the obvious choice. TIF and PNG would also be OK. BMP can support 32bit color and TIF & PNG 48bit. GIF only 256 colors, which is fine for SCI1. The palette for GIF is specified in the file. TIF & PNG can have the palette embedded in the file. There is the Microsoft PAL format for an external file, which I believe is simply an XML. This is a bit different from the SCI PAL resource. Another member here (Aftertaste) wrote a utility in C# that extracts the palette from a view or picture resource and could convert between SCI and Microsoft. He has been more active on my private developers forum.

It does not support anything under SCI.1, but you might also want to take a look at Enrico Rolfi's FotoSCIhop (C++) code.

[MusicallyInspired]

Generally the best way to work with 8-bit color games and palettes would be to use a file format with an 8-bit palette itself. Most old game programs/editors use 8-bit BMPs for this. But I suppose any 8-bit picture format would do. But the logistics of how all that would work is beyond me. They probably each have their own way of storing the palette.

I once tried converting a BMP to a SCI1 picture manually via hex. Obviously I was doomed to failure, but I did notice that the palette header portion was mostly the same; R, G, B values for each bit separated by a null bit or some other random character (can't remember which used what between SCI1 PIC and BMP). I exported a background as a BMP with SCI Viewer and loaded it into PSP5 and fooled around with the colour sliders to make the Deltaur ship in SQ1VGA blue instead of green. But when I went to copy the palette data from my altered BMP to the original SCI PIC in a hex editor the picture came out red ingame instead of blue. So the RGB values must be in an inverted order for every colour which is weird (like BGR instead or something).

As far as images using an entire palette instead of the last 192 (minus the last one), I say leave it to import as is and let the developer plan ahead for that. For instance, creating a BMP that already has the first 64 and the 256th colours set ahead of time. What would be welcome I suppose, though, would be to create some kind of ability to convert an image's palette to condense it to 192 of the most used colours and convert the rest to the nearest matches of what's left over. Again, maybe with other options like dithering, nearest, etc like SCI Comp currently has for importing bitmaps.

Again, I have no idea of the logistics of any of this...

[Collector]

Yeah, I was forgetting about part of the palette being reserved.

[troflip]

Generally the best way to work with 8-bit color games and palettes would be to use a file format with an 8-bit palette itself. Most old game programs/editors use 8-bit BMPs for this. But I suppose any 8-bit picture format would do. But the logistics of how all that would work is beyond me. They probably each have their own way of storing the palette.

I once tried converting a BMP to a SCI1 picture manually via hex. Obviously I was doomed to failure, but I did notice that the palette header portion was mostly the same; R, G, B values for each bit separated by a null bit or some other random character (can't remember which used what between SCI1 PIC and BMP). I exported a background as a BMP with SCI Viewer and loaded it into PSP5 and fooled around with the colour sliders to make the Deltaur ship in SQ1VGA blue instead of green. But when I went to copy the palette data from my altered BMP to the original SCI PIC in a hex editor the picture came out red ingame instead of blue. So the RGB values must be in an inverted order for every colour which is weird (like BGR instead or something).

Yeah, I've just tested loading some 8 bit images, and the palettes are there and intact, so it should be trivial to extract (I'm just using gdiplus to load them, so I don't need to worry about any specific formats - I've tested it with 8 bit .gif and .bmp so far). The main work will be in giving the user options when they import it. I figure:
    - import with palette exactly as is (ignoring which slots are "reserved")
    - import with palette indices to the slots NOT used by global palette 999 (generally index 64 to 254) (if there is room). This would be the most common option.
    - import with palette shrunk to fit to the free slots 64-254 (if no room), either by recalculating from scratch, or rejecting least commonly used palette colors and replacing them with the closest remaining ones
    - remap to the current palette in the pic (basically throwing away the imported palette and generating anew)
    - for images without a palette (e.g. 24/32 bit images), either
        - generate custom palette based on free slots (64-254)
        - generate custom palette ignoring global palette
        - map colors to current pic palette

Whew!

Are you using manged or unmanged code? It is a little more involved using unmanged in C#. How hard would it be to port your script disassembler to C#?

It's in native code. It would be a pretty large chunk of work to port it to C#. But if it's in its own library, it wouldn't be hard to make calls to it from managed code if the API layer is "thin" (i.e. just give it the game folder, a script #, and a place to put the resulting file). But I think there are already simpler tools out there that do that?

With regards to the decompiler (not disassembler), it's coming along, though I haven't done too much work on it lately. However, I think I'm going to introduce a new piece of syntax for the "SCI Studio" syntax to support the ternary operator. This is used extensively in the scripts I've decompiled so far. You can sort of emulate it with if/else statements, but it's less than ideal. So, in C++, this is:

Code: [Select]

gEgo.view = (someVar == someValue) ? 1 : 2;

And in the SCI studio syntax, I'm proposing:

Code: [Select]

(send gEgo:view(? (== someVar someValue) 1 2))

Currently in SCI Studio syntax, you'd need to use:

Code: [Select]

(if (== someVar someValue)
    (send gEgo:view(1))
)
(else
    (send gEgo:view(2))
)

I tried cramming if/else statements directly into the assignment, but I couldn't get something that compiled with SCI Studio (plus it looks ugly). Expanding it out into the above form isn't really feasible either, as they often part of statements that use multiple selectors on the same receiver.

edit.....  ok, it does look like SCI Studio permits this (but SCI Companion barfs on it!). Not sure why I came to a different conclusion before... maybe there are some scenarios where it doesn't work? Anyway, it looks like this:

Code: [Select]

    (send gEgo:view( (if (== someVar someValue) 1) (else 2) ))


[Collector]

I understand that you are not doing it in C#. I was asking if managed C++ because it is easier to make calls from C# to managed C++ than unmanaged. I didn't think you were, but just thought I would ask.

With regards to the decompiler (not disassembler), it's coming along, though I haven't done too much work on it lately.

Fantastic. And I was under the impression that it was just a disassembler. This would be a huge boon for creating game patches.

[troflip]

Managed c++ would be like the worst of both worlds, haha.

I found an interesting website that talks about the original SCI syntax:
http://www.scriptinterpreter.com/syntax

It of course has the notorious code snippet from PQ Swat, but also a discussion of the syntax's similarity to Smalltalk and LISP, and how Brian's SCI Studio syntax differs.

I'm not sure who owns that website, there's no contact information.

In working on the decompiler though, I've come across patterns that indicate certain language features. There was the ternary operator that I mentioned previously, but also another pattern I came across today that tripped up my decompiler. It evaluates an expression, and if it's true it performs an action on the result of that expression. In this case it was just checking if an object was null, and if not it disposed it:

Code: [Select]

(if (global_0019)
    (send <result-of-above-expression>:dispose())
)

Now, in SCI Studio syntax you could express this as :

Code: [Select]

(if (global_0019)
    (send global_0019:dispose())
)

However, if the conditional expression in the if statement was something that caused side effects - such as calling another function - then the above two things result in different behavior:

Code: [Select]

(if (SomeFunction())
    (send <result-of-above-expression>:dispose())
)

vs

Code: [Select]

(if (SomeFunction())
    (send SomeFunction():dispose())
)

The latter ends up calling SomeFunction twice, whereas the first only calls it once. There's no way to do this directly in SCI Syntax, you'd have to add a temp var to avoid the function being called a second time:

Code: [Select]

(= temp0 SomeFunction())
(if (temp0)
    (send temp0:dispose())
)

But the way the disassembly is, it suggests the original syntax had support for this. Without support for this, when decompiling I need to use an additional temp var (which complicates things), or verify that the conditional expression has no side effects, and repeat it (which is also complicated)..

[Collector]

Yeah, that is Lance Ewing's SCI site. He is the author of the original AGI PicEdit. His other site is agifans. Unfortunately he has not posted here is some time. A couple of relevant threads of his:

Original SCI syntax

SCI Decompiler?

BTW, when you are looking a scripts to disassemble or decompile, are us using ones from games compiles with Studio/Companion or official games compiled with Sierra's tools? Not sure if that would make much of a difference in teasing this apart.

[troflip]

BTW, when you are looking a scripts to disassemble or decompile, are us using ones from games compiles with Studio/Companion or official games compiled with Sierra's tools? Not sure if that would make much of a difference in teasing this apart.

I'm looking at official games. There isn't much to learn from decompiling games made with Studio/Companion.

[Collector]

It could be useful when you at the point of testing the decompiler to compare it to the original source, but I guess you are not at that point, yet.

[MusicallyInspired]

Yeah, I've just tested loading some 8 bit images, and the palettes are there and intact, so it should be trivial to extract (I'm just using gdiplus to load them, so I don't need to worry about any specific formats - I've tested it with 8 bit .gif and .bmp so far).

Oh good! I guess things are much less complicated with modern programming. I keep trying to do things manually. Maybe I don't have to? I feel like if I want to learn everything I have to learn to do all these complicated things...

The main work will be in giving the user options when they import it. I figure:
    - import with palette exactly as is (ignoring which slots are "reserved")
    - import with palette indices to the slots NOT used by global palette 999 (generally index 64 to 254) (if there is room). This would be the most common option.
    - import with palette shrunk to fit to the free slots 64-254 (if no room), either by recalculating from scratch, or rejecting least commonly used palette colors and replacing them with the closest remaining ones
    - remap to the current palette in the pic (basically throwing away the imported palette and generating anew)
    - for images without a palette (e.g. 24/32 bit images), either
        - generate custom palette based on free slots (64-254)
        - generate custom palette ignoring global palette
        - map colors to current pic palette

Whew!

That looks great. I can think of a couple other conditions but they probably aren't necessary at all. After experimenting a bit we can probably tweak some things or add some features I imagine? I was going to suggest some advanced features like what SCIComp currently has for importing backgrounds for 16-color vector format, but I figure if anyone needs anything more robust than what you have here they could edit an image beforehand in PSP or PhotoShop, which will probably be happening anyway. Nice to know these things are possible! I'm so excited for this...

[troflip]

Just a little update. I have import and export of palettized bitmap images working for pics.

Importing an image:



An exported pic background in photoshop, with accompanying palette:


There's definitely some weirdness with the palettes on import and export. I support 4 file formats right now: png, tif, bmp and gif. Only bmp works perfectly end-to-end. tif and gif end up always being 256-color palettes, even if less entries are used (i.e. I export a gif of the pic background with a palette of 160 colors, but when loading it into photoshop, all 256 colors are marked as being used, the remaining just filled with black). Not a huge issue. Png does not work properly yet, I'm not sure where the problem is.