Community

SCI Programming => SCI Development Tools => Topic started by: OmerMor on January 11, 2019, 06:17:32 AM

Title: SCI32 Source Code
Post by: OmerMor on January 11, 2019, 06:17:32 AM
https://github.com/OmerMor/SCI32 (https://github.com/OmerMor/SCI32)

This repository contains snapshots of the SCI interpreter and system scripts between 11/1994 - 10/1995, and you can learn about the evolution of the SCI in this era.
You can see the these snapshots in git's commit history: https://github.com/OmerMor/SCI32/commits/master (https://github.com/OmerMor/SCI32/commits/master)
Title: Re: SCI32 Source Code
Post by: Daventry on January 11, 2019, 07:18:44 AM
Thanks so much. Have you sci0, approximately 88-90 years?
Title: Re: SCI32 Source Code
Post by: MusicallyInspired on January 11, 2019, 09:10:57 AM
Awesome!! God bless you and your connections!
Title: Re: SCI32 Source Code
Post by: EricOakford on January 11, 2019, 12:17:45 PM
Maybe now we can get to work on improving SCI32 support in Companion. I could maybe work on putting together a decompile of the SQ6 demo, and using that as the basis for the SCI32 template game. We now have original procedure and global names!
Title: Re: SCI32 Source Code
Post by: Kawa on January 11, 2019, 01:22:45 PM
WOOHOO!
Title: Re: SCI32 Source Code
Post by: Collector on January 11, 2019, 02:55:22 PM
Thank you Omer!
Title: Re: SCI32 Source Code
Post by: Kawa on January 11, 2019, 03:59:15 PM
I tried to compile it. After a bunch of work regarding missing implied int and some scope shenanigans, I got up to the point where it can complain about sound drivers in 640x480, then crashes.
Title: Re: SCI32 Source Code
Post by: Collector on January 11, 2019, 04:39:15 PM
I have not yet looked through this. What version is this interpreter?
Title: Re: SCI32 Source Code
Post by: Kawa on January 11, 2019, 05:30:16 PM
INFO.CPP says 2.100.002.
Title: Re: SCI32 Source Code
Post by: EricOakford on January 11, 2019, 08:44:23 PM
Okay, here is my decompile of the SQ6 demo. The system scripts have been identified (namely, those in the 64xxx range), as have the system globals (namely, the first 100).

The scripts compile without error, but the game will refuse to run, giving "Error 42: This game has not been version-stamped".
Title: Re: SCI32 Source Code
Post by: Collector on January 11, 2019, 09:29:23 PM
INFO.CPP says 2.100.002.

So we are talking about KQ7/SQ6 era.
Title: Re: SCI32 Source Code
Post by: lskovlun on January 11, 2019, 10:26:10 PM
Meanwhile, I've been working on a savegame dumper for SCI32 games (where that is actually possible, unlike earlier SCI generations). It uses ScummVM to get data on class layouts and such (and for the compression code). I've already used it to investigate one bug in KQ7.

EDIT: Sierra SCI savegames, of course.
Title: Re: SCI32 Source Code
Post by: Collector on January 12, 2019, 12:22:27 AM
What bug?
Title: Re: SCI32 Source Code
Post by: lskovlun on January 12, 2019, 02:19:28 AM
Some KQ7 rooms set up global state and may not reset that state properly if you teleport out (using the debugger). This causes a crash down the road (so far down the road, in fact, that you wouldn't make the connection between the two events). It only happens if you edit variable 13 directly, not when a script calls the newRoom: method. This, of course, means that the KQ7 debug script works, since it does it the proper way. But that is not so easy to do by hand.
Title: Re: SCI32 Source Code
Post by: Kawa on January 12, 2019, 06:16:04 AM
The latest version of Watcom I could find that would compile the interpreter without needing anything like adding missing int type specifiers and had the required compiler and linker, 11b, gave me the same result as yesterday's attempt with OpenWatcom: it switches to high-res, throws up a message about the sound driver if you set that up wrong, then crashes. I had to remove the univbe and spmode library inclusions to finish the make, and I guess one of those might be related to the crash?

https://winworldpc.com/product/watcom-c-c/110b

(7 didn't have the right programs, 10 choked on memory.cpp, and OpenWatcom is too strict.)
Title: Re: SCI32 Source Code
Post by: Kawa on January 12, 2019, 09:44:54 AM
I'm not sure what I did but it no longer crashes. On the flipside, it doesn't seem to take resource volumes?
Title: Re: SCI32 Source Code
Post by: OmerMor on January 12, 2019, 04:17:23 PM
Thanks so much. Have you sci0, approximately 88-90 years?

Unfortunately - not.  :(
Title: Re: SCI32 Source Code
Post by: EricOakford on January 15, 2019, 11:08:45 PM
Here are the SCI32 header files adapted for SCICompanion. Yes, we should add support for non-integer and complex defines, since Sierra used them. For that matter, we have information on the SCI32 kernel functions.
Title: Re: SCI32 Source Code
Post by: Fox on April 17, 2020, 04:55:42 PM
Hello there, I am new here on this forum :). I tried out to compile this SCI32 source code with varying results :o. The code is from an early C++-Era. One needs to correct a lot of grammar failures first wenn you want to compile this on the latest (1.9) OpenWatcom-Version. I managed to fix all the error messages but still didn't get it to work with OW v1.9. It seems to work however on early versions of Watcom C++ (10.6) and MASM 6.11. Testing with the Watcom Debugger I found out that the code crashes directly after running the opcode pseudo-machine. I managed to get both the debug version with the opcode debugger and the normal version compiled.

In the c++ version of the pmachine there was a lot of inline assembly code inserted which resulted stack balance errors wenn trying to run the code. They optimized the pmachine by making inline assembly jumping statements to other parts of the program out of control of the compiler. The problem with hidden jumping to other c++code positions without the compiler knowing is that you can't be sure it clears the stack propery and jumps back to the correct position.

One can see that the code was written in a transition period moving from C to C++ language. Some stuff like const values without type specifiers are not allowed anymore in modern C++ language. The used #define statements to implement template-like functionality. It must have been very hard for the original team to develop this code. in the win3.1 and win95 it was probably not easy to debug 32-bit dos protected mode applications. People had to do with a lot of system crashes. The code had to be fast and efficient also. They of course had no other choice then to implement some stuff with assembly code. There were probably also difficulties with implementing audio drivers and stuff under protected mode. I tend to prefer however the readable C-style-code like used in the game Doom of the time era and which is more easy to port to other platforms and compilers.

I got the pmachine to work again by removing this code and translating the assembly-code-optimized version in plain traditional C-code. The only game it seems to work with is Space Quest 6. The Adlib music is flawlessly. The only problem I still have is with the audio samples. These play way to slow wit a low of clicking. There seems to be a problem with updating the sound buffers which are sent to the soundcard. The Watcom debugger is not that good compared to modern day options and it is often losing track and displaying the false source code while running, perhaps I should try with Dosbox-Debugger an IDA Pro 5.0 like the scummvm dudes do.

There perhaps is too much of the legacy stuff in this code to use it as a base for further development.  It may be more efficient to rewrite a SCI32 interpreter from ground up with modern stuff. There is however room for cool hacks like:
- changing the Pmachine to a Lua-interpreter
- Writing the game logic directly in C/C++ which can be loaded dynamically and interfaces with the kernel api
- Improve sound support, like supporting AC97 in dos mode
- Adding a proper SB16 driver with high samplerate support and auto-initializing DMA
- Adding support for modern day picture format like .PNG . GIF etc.
- improved midi support and synth emulation
- Adding a classic sierra text parser back


Greetz from Mr. Fox
Title: Re: SCI32 Source Code
Post by: Kawa on April 17, 2020, 10:03:28 PM
I too have tried to compile and run SCI32, though I hadn't tried SQ6 specifically.



As for writing an all-new SCI terp, I actually have one in the works right now, including the thing where it runs Lua game code and uses PNG. I'm just absolutely stumped on pathfinding right now.
Title: Re: SCI32 Source Code
Post by: Fox on April 18, 2020, 02:39:48 AM
I like the stuff you did with SCI16 and it inspired me to try the same with SCI32. I was hoping to get it to work without any problems so people could use it to develop SCI32 games. The original SCI32 uses DOS4GW Professional which I don't have, I don't think that makes the difference. Turning off or on some optimizing options causes that the code doesn't run. Perhaps I should try to build the windows version of SCI32.

I included here the modified source code. To compile MASM 6.11 and Watcom C++ 10.6 is needed. It worked on PCEM and Dosbox.
to compile go the extracted directory in dos and type:

wpp386.exe /3s /j /s /ox /ot math.cpp


wmake.exe


to build without debug functionality use wmake.exe SIERRA=1

The math file needs to be compiled separately because Watcom crashes on that file with all the optimizing enabled.

I am interested about your SCI-engine. Is your SCI-engine a retro-coding project (MS-DOS and son on) or do you make it for modern machines? There seems to be a dude wo made a Sierra AGI-like engine for Unity.
Title: Re: SCI32 Source Code
Post by: Fox on April 18, 2020, 02:44:57 AM
One needs to remove the DACBLAST.DRV file to remove the non-properly working digital sample playback.
Enter tilde to enter the debug menu, use / to show the debugging options. It works with SQ6 from GOG.COM. You can make a ISO from the SQ6 directory. When SQ6 is loaded as a cdrom in the D: drive then you can go to D: and type c:sciub.exe for easy testing the compiled file. 8)
Title: Re: SCI32 Source Code
Post by: Kawa on April 18, 2020, 03:22:28 AM
I am interested about your SCI-engine. Is your SCI-engine a retro-coding project (MS-DOS and son on) or do you make it for modern machines?
My project (https://github.com/Kawa-oneechan/newsci) is surprisingly not a retrocoding one. It runs in SDL2.
Title: Re: SCI32 Source Code
Post by: Fox on April 18, 2020, 03:35:52 AM
I think it would be great to have a good SCI-engine with SDL2 8)
Even the original sierra coders commented somewhere in their code that it is garbage..
And SDL would be great for portability also (Linux, IOS, Android etc)
Good luck with your project!
Title: Re: SCI32 Source Code
Post by: Kawa on April 18, 2020, 08:39:07 AM
To be fair I'd have a better time and probably more success implementing 256-color mode (which would involve hacking the crap out of picopng among other things) or scaled sprites, than to implement pathfinding.

I copied SCI11's, it compiled fine but acted up. SCI32's likewise. ScummVM's works fine but makes me feel ethically and morally wrong. It's all on my twitter (https://twitter.com/search?q=from%3Anoxicodev%20pathfinder&src=typed_query&f=live).
Title: Re: SCI32 Source Code
Post by: MusicallyInspired on April 18, 2020, 10:50:11 AM
I love this stuff it's fascinating!
Title: Re: SCI32 Source Code
Post by: Fox on April 19, 2020, 06:59:02 AM
I am looking to get this SCI32 running on Windows. It might be useful to have a working sci32.exe under Windows which can be used to run test scripts to test the pathfindung and so on. In such a way one can implement its own pathfindung algorithm which mimics the behavior. I didn't check out wether the SCI16 algorithm is the same as SCI32 algorithm. The original algorithm seems to be patented by sierra and may not be used for another few years until the patent runs out.
Title: Re: SCI32 Source Code
Post by: lskovlun on April 19, 2020, 07:46:44 AM
The original algorithm seems to be patented by sierra and may not be used for another few years until the patent runs out.
That was a concern fifteen years ago when pathfinding was being added to FreeSCI. The original patent was filed in 1990; it has long since expired. I wouldn't worry about it now. On the other hand, I wonder why Kawa would be reluctant to borrow code from ScummVM and not from the original interpreter.

EDIT: The Message kernel call was patented too. I don't think anyone ever took that seriously.
Title: Re: SCI32 Source Code
Post by: Kawa on April 20, 2020, 03:03:30 AM
I didn't check out wether the SCI16 algorithm is the same as SCI32 algorithm.
It is. Rewritten in C++, so it has things like vector structs with operator overloading, some call styles changed, a few identifiers have been renamed, and the kernel layer (KAvoidPath and such) is in the same file, and it runs on 32-bit systems even though the SCIWord typedef is still 16-bit, but it's effectively the same algo.

An acquaintance on Discord had a go at trying to understand it, so they might explain it to me and I might reimplement it in NewSCI, but then my brain shut off on the topic. I attached their GIFs, which are basically pixel art animated versions of the ASCII art in the SCI source.
Title: Re: SCI32 Source Code
Post by: Fox on April 22, 2020, 12:43:39 AM
The current AGS version uses the A*-algorithm. It is very good for 2D maps and so on, well documented and it has lots of C++ examples on the internet. The AvoidPath algorithm was optimized to run on really slow machines. It is old style C++ code which marks the transition from C where they were comfortable with. One can write it much more tight and readable with the modern C++ standard libraries. Things like sorting the nodes of a polygon can be written in one line nowadays. The C++ Boost library provides a A* library and I believe there is also a Graph library. In either way you need to make polygons around your nonwalkable areas with each algorithm so the pathfinder knows where ego can walk. The advantage of A* is that you can program it in such a way that ego can walk more naturally around a tree or something. a normal person does not walk until he hits the tree and then walks around it.
Title: Re: SCI32 Source Code
Post by: Kawa on April 22, 2020, 07:08:14 AM
I'm okay with using A*, ScummVM uses it to great effect. My problem then is creating the graph for A* to use.

Allow me to steal an old picture from Phil's blog.
(http://blog.cascadiaquest.com/wp-content/uploads/2017/04/Paths.png)

The yellow lines, as I understand it, are key to the A* phase.

Great. Now I have to merge polygons together for use and somehow build a web of edges for A* to use.
Title: Re: SCI32 Source Code
Post by: Fox on April 23, 2020, 12:28:14 PM

AAAA 00XA X000 XAAA 000X AX00 AAAA AAAX
XXAA  00XA AX00 0XAA 00XA AX00 AAXX AAX0
00XA  0XAA AAXX 00XA XXAA AAX0 AX00 AX00
000X  XAAA AAAA 00XA AAAA AAAX X000 AX00


A = walkable area
X = white border line

My suggestion:
1) detect 'blunt' edges in the walkable border lines (the white lines) using possible byte masks above
2) when a blunt edge is detected generate a node in the A* Graph
3) Try to connect all the found blunt edges via yellow edges with other blunt edges these form the transitions to other nodes in the A* Graph. A yellow line can not cross a white line. Note that the sharp edges don't have yellow lines.
4) Make a function to determine which graph points surround ego, make the node lying most in the direction of the mouse click the start node
5) Run the A* algorithm with heuristic that ego likes to walk straight in direction of the target
6) The A* algorithm finds the edge point lying most close to destination point.
7) A simple moveTo can be made from last edge point to destination point.

Title: Re: SCI32 Source Code
Post by: Kawa on April 23, 2020, 03:47:28 PM
I think you may have misunderstood the picture.

Though it looks like SCI0, which had no polygons, Cascadia Quest is not in fact SCI0 and does have polygons. Phil added SCI11-style polygon support himself. The white lines in the picture are not like the white lines on the SCI0 control screen. They are just a visualization of the SCI11-style polygons.

And SCI11-style polygons are what I'm after for my engine.
Title: Re: SCI32 Source Code
Post by: troflip on April 23, 2020, 05:00:32 PM
There are two tricky parts:
- coming up with the graph of edges
- dealing with all the floating point issues

On my blog, for the first one I talk about:

Quote
Another possibility is to use the concave vertices of a polygon. David Gouveia has a good article on this. Basically, by determining the interior line-of-sight connections between the concave vertices, you have all you need for pathfinding. Your start and end points (as long as they are within bounds) are guaranteed to connect to one of the concave vertices.

But it looks like that link is now broken lol...
Title: Re: SCI32 Source Code
Post by: troflip on April 23, 2020, 05:01:08 PM
Hmm, looks like it's here now:
https://www.david-gouveia.com/pathfinding-on-a-2d-polygonal-map
Title: Re: SCI32 Source Code
Post by: Kawa on April 23, 2020, 07:53:50 PM
I've gone over that page four times now. It still won't click for me how to actually do those things.

Edit: David Gouveia's page mentioned two parts of Game Programming Gems 1 and 2. I acquired copies and checked out the parts about pathfinding and... I'm still lost.
Title: Re: SCI32 Source Code
Post by: EricOakford on August 15, 2020, 06:44:56 PM
I've put together a barebones template in SCI32. It was built from the Torin's Passage demo. Download it here (https://drive.google.com/file/d/1L7fC5a0AMokvD7MYLLpCy_QWyFm-4psc/view?usp=sharing).

All system scripts are based on the 10/12/1995 source.

I was pleased to find that the interpreter for the Torin demo appears to have no version stamp check. So that's one obstacle cleared.

There's still one other issue: if the resource file is modified, SCICompanion mistakenly auto-detects the resource.map format as SCI1.0 and no LOFSA absolute. This results in errors relating to offsets if the resource file is altered any further!
Title: Re: SCI32 Source Code
Post by: MusicallyInspired on August 15, 2020, 09:17:52 PM
Does the Torin's Passage demo interpreter still support MIDI-based sound resources?
Title: Re: SCI32 Source Code
Post by: EricOakford on August 15, 2020, 09:27:03 PM
Yes, the interpreter still has MIDI support; I tested it myself. But since the music drivers weren't included, I grabbed the ones from the SQ6 demo.
Title: Re: SCI32 Source Code
Post by: Collector on August 16, 2020, 02:21:02 PM
Nice, but wouldn't Companion need to be updated to handle things like V56 and P56 resources? Would the compiler need to be updated? It would be nice to be able to build SCI32 games.
Title: Re: SCI32 Source Code
Post by: Kawa on August 16, 2020, 04:56:30 PM
Nice, but wouldn't Companion need to be updated to handle things like V56 and P56 resources? Would the compiler need to be updated? It would be nice to be able to build SCI32 games.
Both seem to work fine... except that you can't edit P56 resources. The compiler seems to work too... it's just that map format thing that's the worst issue here.
Title: Re: SCI32 Source Code
Post by: EricOakford on August 17, 2020, 11:53:51 AM
One thing I did notice in regards to the view editor - if you modify a view, there's a risk of it not displaying properly in the game. I discovered this when removing the embedded palette from the stock arrow cursor - the lower half of the cursor is gone in the game, but it is shown properly in the view editor.

On the plus side, SQ6 Roger looks simple enough (mostly solid colors instead of being really pixellated like GK1 Gabriel) to recolor into a generic ego. Sometimes the cartoony look pays off...