Original AGI Interpreter

[lance.ewing]

After more reading, it looks like 'Let's C' will use their internal assembler if the source file ends in '.s' and it will use MASM if the source file ends in '.asm'. Since all the assembly source files end in '.asm', it looks like Sierra wasn't using the MWC built in assembler. So I guess that theory is debunked.

The fact that MWC by default uses MASM when the file extension is .ASM adds further support to MASM being what Sierra used for those files. So we have the comment in the SCROUT.ASM file that mentions MASM, and now also the fact that MWC uses MASM by default for .ASM files. We don't have a copy of their make file for AGI, so its hard to say if they used MWC for everything (with the .ASM files being delegated to MASM), or whether they built the .ASM files directly with MASM. I guess there wouldn't be much difference.

Further support that MASM is what they were using (can't remember if I mentioned this already) is that the floppy disk slack space for some games has directory entries for MASM. I checked the timestamp and file size mentioned in that dir entry and it matches MASM 4 exactly, which is the main reason I started out with using MASM 4.

I think that this IRPC directive is most likely going to work. I've drafted an initial implementation of the pcall macro using IRPC and logically it appears that it should work, from reading through the macro code I've come up with. I just need to work through the various syntax errors I've got in there at the moment, but I have a good feeling I should be able to resolve those and have something that works. Hopefully there will be good news later on.

[lance.ewing]

Unfortunately its bad news. The syntax shown in that MASM 5.0 MIXED.INC file doesn't' work in MASM 4. The IRPC bit to loop over the chars does work but not much after that. I'm going to try MASM 5.0 next using this approach. If it works, then I guess I can switch to MASM 5.0, since that was released at the end of July 1987. It is possible that they switched to MASM 5.0 at that time and started to make use of new features. For that to be true, it would mean that they weren't previously using a pcall macro.

[lance.ewing]

I'm certainly making more progress with MASM 5.0.

The MASM 5.0 manual does mention a couple of new features not in MASM 4 that would explain why I was finding it hard to get it working under MASM 4:

The use of angle brackets to force string evaluation is a new feature of Version 5.0 of the Macro Assembler. Previous versions tried to evaluate equates as expressions. If the string did not evaluate to a valid expression, MASM evaluated it as a string. This behavior sometimes caused unexpected consequences.

I was aware that MASM 4 didn't support the <> syntax for string equates, but this quote stresses the difficulty that people (like me I guess) had with trying to make MASM 4 treat something as a string.  It is possible to define a string equate, by relying on that fallback mechanism described above, but I'm now thinking that there isn't a way in MASM 4 to perform string concatenation, which is certainly possible in MASM 5.0, as I have that bit working now, i.e. building up a string character by character (which seems to be what is required to perform a substring in MASM 5.0 in the absence of the SUBSTR directive, which doesn't become available until 5.1).

The other big difference that make it difficult/impossible under MASM 4 is this one:

%text

MASM computes the expression's value and replaces text with the result. The expression can be either a numeric expression or a text equate. Handling text equates with this operator is a new feature in Version 5.0. Previous versions handled numeric expressions only.

It's that last point, i.e. that MASM 4 handled numeric expressions only, that causes issues with what I'm trying to do in the pcall macro.

I still don't have it working in MASM 5.0 yet. The biggest blocker is with the & character. It is certainly possible to loop over the characters in a string, and it is possible to test for the # character and other normal alphanumeric characters, but trying to test for & isn't working. I think it must be trying to do a substitution, which is what & does within a macro, by default. It is apparently possible to create a & literal using <>, e.g. <&>, but I haven't yet worked out how to make use of that. There is also something going on with types. I am wondering whether internally it treats a single char and a string as different types when doing an identical test, since my logging within the macro shows that I am comparing a character of & (from the irpc loop) with a & string constant (since I can't seem to use & directly in the comparison, due to the substitution issue I mentioned), but the comparison evaluates to false, even though my logging shows that both sides contain &.

Anyway, this is where I am at the moment. I have a couple more ideas to try, but they're starting to become work arounds and hacks. If one of them works though, I'll go with it.

[lance.ewing]

Have you looked into the FILEIO source from the SCI interpreter? Perhaps it's similar.

Omer, I probably should have asked this before now, but I assume that you don't have any original AGI interpreter source, other than what was extracted from the game disks?

Would save some time 

[lance.ewing]

The biggest blocker is with the & character. It is certainly possible to loop over the characters in a string, and it is possible to test for the # character and other normal alphanumeric characters, but trying to test for & isn't working. I think it must be trying to do a substitution, which is what & does within a macro, by default. It is apparently possible to create a & literal using <>, e.g. <&>, but I haven't yet worked out how to make use of that. There is also something going on with types. I am wondering whether internally it treats a single char and a string as different types when doing an identical test, since my logging within the macro shows that I am comparing a character of & (from the irpc loop) with a & string constant (since I can't seem to use & directly in the comparison, due to the substitution issue I mentioned), but the comparison evaluates to false, even though my logging shows that both sides contain &.

I think I've come up with a convoluted way of working around the above (assuming my hunch on the above is correct). I've got an equate constant for the & char, defined as <&>, and then for each character in the IRPC loop, I used a macro to convert it to a string, by defining another string equate that uses the char's value. I then use another macro to compare the two, and this seems to match for the & char. Now I need to wire that up into the main logic I have. I think I might have it working later today, but I've been there before. I feel a little more confident this time though. Let's see.

[mnicolella]

This would be pretty cool if you got it to work, although I would say it seems like an awful lot of effort in order to enable that syntax, I wonder if Sierra really would have put in that much effort in order to use that syntax. Certainly possible, just seems a lot easier to use something else especially considering there's only a handful of pcall uses in the code?

[lance.ewing]

This would be pretty cool if you got it to work, although I would say it seems like an awful lot of effort in order to enable that syntax, I wonder if Sierra really would have put in that much effort in order to use that syntax.

I have got my pcall macro fully working now in MASM 5.0, for all three types of argument. I am convinced that it wouldn't have been possible in MASM 4.

Certainly possible, just seems a lot easier to use something else especially considering there's only a handful of pcall uses in the code?

The fact that there is only a handful of them adds a little weight to it being something new that they were trying out. I can see 15 uses of the pcall macro in the code on the SQ2 disk but no uses in the code on the KQ3 disk. As mentioned recently, the KQ3 code appears to be older by at least half a year. Its not absolute proof obviously. Finding a pcall use in the code on the KQ3 disk would probably have eliminated MASM 5.0 being available to them, but since the code on the SQ2 disk is from Oct 1987, and MASM 5.0 came out at the end of July 1987, it does allow for them to be using it and potentially trying out a few new features and more powerful custom macros.

The most recent change history comment from Jeff Stephenson that I can see is from mid July 1987, and since he appears to be the person who most often added change history comments, I'm wondering whether maybe Jeff moved on to the SCI development at that point and others took over the AGI development, as Jeff is likely to have added further change history comments if he was still working on it. The initials pmk appear for Aug 1987 and Sept 1987. I think this must be Paul Krasno who is credited with working on the game development system for the AGI KQ4 release. I wonder if he was the one introducing the pcall usage. I guess we're unlikely to ever know that.

[OmerMor]

Omer, I probably should have asked this before now, but I assume that you don't have any original AGI interpreter source, other than what was extracted from the game disks?

Would save some time 

I'm afraid not... ?\_(ツ)_/?

[Kawa]

I love how utterly broken Unicode support is on this damn board X3

[lance.ewing]

As mentioned recently, the KQ3 code appears to be older by at least half a year.

I have been able to work out the date of the AGI interpreter source code files on the KQ3 2.14 720K Disk 1 floppy disk. Not only are the files on there, but there is also a section that has details about the files, including their name, size, start offset, and datetime. It isn't a standard DOS DIR table format, but it is similar, most likely part of the data written by whatever backup tool they used to make backups of the code.

So, from that, I've been able to determine that most of the files are dated the 5th Feb 1987, except for the MACRO.AH file, which is dated the 24th Mar 1987, and the GAME.AH file, which is dated the 31st March 1987. This range of dates is therefore about 6-8 months before the date of the code on the SQ2 2.0D disk.

The SQ2 disk doesn't have the MACRO.AH file, but given the gap in time, it is certainly possible that many more macros were added in between. I'm just guessing though as to how they were implemented, but I figure that if from a usage perspective they behave the same, then it should be fine.

The dates would associate most of the code extracted from the KQ3 disk to roughly the 2.272 AGI interpreter version, whereas the GAME.AH and MACRO.AH files are getting close to the date of the 2.411 version. They would obviously have had lots of internal builds between Feb 1987 to April 1987 to account for the jump in version number over that time. All we see are the released versions.

I can't remember if we have previously discussed the big jump in version number from the 2.440 version to the 2.903 version. It suggests many internal builds, and therefore a fair bit of development. There is a roughly 4 month gap between those two versions.

[lance.ewing]

I can see 15 uses of the pcall macro in the code on the SQ2 disk but no uses in the code on the KQ3 disk. As mentioned recently, the KQ3 code appears to be older by at least half a year. Its not absolute proof obviously. Finding a pcall use in the code on the KQ3 disk would probably have eliminated MASM 5.0 being available to them, but since the code on the SQ2 disk is from Oct 1987, and MASM 5.0 came out at the end of July 1987, it does allow for them to be using it and potentially trying out a few new features and more powerful custom macros.

This may be true of the pcall macro, but I've been looking over the code on the KQ3 disk more closely and can see that it uses the following macros that are not in its copy of the MACRO.AH file: enter, exit, do, until, dloop, .if, .else, .end. I already had all of those working under MASM 4, and since there are no uses of pcall in the code on the KQ3 disk, it doesn't rule out a switch to MASM 5.0 by Oct 1987. It is interesting that these macros (enter, exit, until, dloop, do, .if, .else, .end) are not defined in the MACRO.AH file. That suggests that there was another macro file somewhere, but the problem is that the source files that use these macros only include the MACRO.AH file, so there would have to have been another way that MASM pulled in the definitions of these macros.

I have been able to work out the date of the AGI interpreter source code files on the KQ3 2.14 720K Disk 1 floppy disk. Not only are the files on there, but there is also a section that has details about the files, including their name, size, start offset, and datetime. It isn't a standard DOS DIR table format, but it is similar, most likely part of the data written by whatever backup tool they used to make backups of the code.

I have noticed that the SQ2 disk has a similar section with details about some of the files, which will be interesting to look at, as this would give timestamps to a few of the files. Up to now, I've assumed that the code is all from the date of the memory map file, but if any have a timestamp newer than that, then that obviously wouldn't be true. We'd expect them all to be on or before the date of the memory map file. I'll take a look at that later today.

[lance.ewing]

I have decoded the details about the files on the SQ2 disk. It turns out that there are details for 85 distinct files, which covers most of the ones for which there is source on the disk. In the process of decoding this, I've discovered that the format is the DOS BACKUP format, i.e. the format created by the BACKUP DOS command. The fields match exactly. So that answers the question as to what tool they used. The physical floppy disk must have been used with the BACKUP command to create an archive of the AGI source, plus a few extras.

The following is the full list of files mentioned, their dates and sizes. Notice that there are two sizes: One is the total size of the original file. The other is the amount of the file in bytes actually stored on the disk. In most cases, the two sizes are the same, but for one of them, JRGRAPHX.ASM, the second size is smaller. This is because the disk ends before the whole file can be completed, so it is only a part of the file. The rest of it would presumably continue on the next floppy.

Code: [Select]

1987-02-05 ANIOBJ.H      2892 bytes  2892 bytes
1987-02-05 BEEP.ASM       758 bytes   758 bytes
1987-02-05 DIRFLAGS.H      87 bytes    87 bytes
1987-02-05 DOSFLAGS.H      89 bytes    89 bytes
1987-02-05 ERRMSG.H      1046 bytes  1046 bytes
1987-02-05 EVENT.H        642 bytes   642 bytes
1987-02-05 FLAG.ASM      2624 bytes  2624 bytes
1987-02-05 FLAGS.H         89 bytes    89 bytes
1987-02-05 HDIALOG.C     1593 bytes  1593 bytes
1987-02-05 INITDIR.ASM    696 bytes   696 bytes
1987-02-05 JRJMPTBL.ASM   415 bytes   415 bytes
1987-02-05 KBDDRV.C      1331 bytes  1331 bytes
1987-02-05 KEYDEFS.H      777 bytes   777 bytes
1987-02-05 KEYMAP.C       481 bytes   481 bytes
1987-02-05 LOG.C         1352 bytes  1352 bytes
1987-02-05 LOGIC.H        528 bytes   528 bytes
1987-02-05 MISC.ASM      3091 bytes  3091 bytes
1987-02-05 MOVEOBJS.C    2608 bytes  2608 bytes
1987-02-05 NEWROOM.C     2170 bytes  2170 bytes
1987-02-05 OBJACT.C      1421 bytes  1421 bytes
1987-02-05 OBJECT.H       494 bytes   494 bytes
1987-02-05 OBJLIST.H      627 bytes   627 bytes
1987-02-05 PICOBJ.C      1981 bytes  1981 bytes
1987-02-05 PICOBJ.H       317 bytes   317 bytes
1987-02-05 PICTURE.H      207 bytes   207 bytes
1987-02-05 POSITION.C    2721 bytes  2721 bytes
1987-02-05 PRIORITY.C     904 bytes   904 bytes
1987-02-05 PRODIO.H       450 bytes   450 bytes
1987-02-05 SAVEAREA.C    1218 bytes  1218 bytes
1987-02-05 SCRIPT.H       400 bytes   400 bytes
1987-02-05 SEGERR.H       371 bytes   371 bytes
1987-02-05 SEGIO.H        482 bytes   482 bytes
1987-02-05 SOUND.C       2199 bytes  2199 bytes
1987-02-05 SOUND.H        209 bytes   209 bytes
1987-02-05 TEXTWIN.H      148 bytes   148 bytes
1987-02-05 TRACE.H         92 bytes    92 bytes
1987-02-05 TYPES.H        774 bytes   774 bytes
1987-02-05 VIEW.C        6793 bytes  6793 bytes
1987-02-05 VIEW.H         990 bytes   990 bytes
1987-02-18 SEGPTR.C      3558 bytes  3558 bytes
1987-02-19 PICTURE.C     3123 bytes  3123 bytes
1987-02-19 TRACE.C       5163 bytes  5163 bytes
1987-02-20 PRINT.C      10989 bytes 10989 bytes
1987-03-24 JOYREAD.ASM   2478 bytes  2478 bytes
1987-03-25 MOVETO.C      2132 bytes  2132 bytes
1987-03-25 OBJLIST.C     4585 bytes  4585 bytes
1987-03-25 STATUS.C      4537 bytes  4537 bytes
1987-03-25 WANDER.C       846 bytes   846 bytes
1987-04-07 INTRPT.ASM    6760 bytes  6760 bytes
1987-04-07 MOTION.C      3990 bytes  3990 bytes
1987-04-30 CMDATA.ASM     937 bytes   937 bytes
1987-04-30 GAME.H        3867 bytes  3867 bytes
1987-04-30 INIT.C        3805 bytes  3805 bytes
1987-04-30 RESTART.C      862 bytes   862 bytes
1987-05-11 TIMERINT.C    1412 bytes  1412 bytes
1987-05-12 JOYDRV.C      3732 bytes  3732 bytes
1987-05-12 MSGSTR.C      1019 bytes  1019 bytes
1987-05-12 SEGMENT.C     6643 bytes  6643 bytes
1987-07-14 STRING.C      3946 bytes  3946 bytes
1987-07-23 CMOBJSBR.ASM 18448 bytes 18448 bytes
1987-07-24 MEMMGR.C      2299 bytes  2299 bytes
1987-07-27 PARSE.C       5531 bytes  5531 bytes
1987-08-16 VGJMPTBL.ASM   578 bytes   578 bytes
1987-08-21 DOTEST.ASM    8701 bytes  8701 bytes
1987-08-24 INITMACH.VGA  1866 bytes  1866 bytes
1987-08-24 IOBJSBRS.ASM  6853 bytes  6853 bytes
1987-08-24 RANDOM.ASM     758 bytes   758 bytes
1987-08-24 SAVENAME.C    1951 bytes  1951 bytes
1987-08-25 INITDIR.OBJ    118 bytes   118 bytes
1987-08-31 MAIN.C        2800 bytes  2800 bytes
1987-08-31 SQSG.3        2774 bytes  2774 bytes
1987-09-03 GETGAME.C     9437 bytes  9437 bytes
1987-09-03 HGRAPHX.OBJ   1852 bytes  1852 bytes
1987-09-03 HOBJSBRS.OBJ   963 bytes   963 bytes
1987-09-03 INTRPT.OBJ    1442 bytes  1442 bytes
1987-09-03 RESTGAME.C    2611 bytes  2611 bytes
1987-09-03 SENDIT.BAT      39 bytes    39 bytes
1987-09-04 CMGRAPHX.ASM 16569 bytes 16569 bytes
1987-09-09 MENU.C        8700 bytes  8700 bytes
1987-09-29 SCRIPT.C      2195 bytes  2195 bytes
1987-09-30 EQUIPCHK.ASM  8204 bytes  8204 bytes
1987-09-30 JRGRAPHX.ASM  7875 bytes  6144 bytes
1987-09-30 SCRACT.C      4869 bytes  4869 bytes
1987-10-01 SCROUT.ASM    7185 bytes  7185 bytes
1987-10-07 PRODFLAG.H      27 bytes    27 bytes

There are a number of things to draw attention to with regards to the dates:

• The earliest and most common date mentioned is the 5th Feb 1987, which is the same date as most of the files on the KQ3 disk.
• The most recent date is the 7th October 1987, i.e. the same date as included in the memory map file. There isn't any later date. That's a strong indication that the source files are a match for the memory map file, which we were already fairly confident in.
• Many of the files were updated during July, August, and September of 1987. This must represent the internal builds that progressed the versioning from the 2.4XX versions up to the 2.9XX versions

I have checked the file sizes mentioned and they match the actual sizes of the files that were extracted from the SQ2 disk, which supports the content being a single DOS BACKUP file, and that these dates are highly likely to represent the actual dates of each extracted file. It all ties together.

There is one thing about the dates that worries me, and that is the timestamp of the INTRPT.ASM file. There are four files that use the pcall macro. Three of those have a date after the release of MASM 5.0, but INTRPT.ASM is dated the 7th April 1987, nearly 4 months before MASM 5.0 was released. That would seem to eliminate the idea that they wrote the pcall macro after switching to MASM 5.0, unless they got an early release. It could mean that they somehow had it working in MASM 4.

[lance.ewing]

I have been able to successfully build all "complete" files of those extracted from the SQ2 & KQ3 disks. As mentioned, one of them is truncated due to the end of the disk.

This is a link to the macro file containing my implementations of the missing macros:

https://github.com/lanceewing/agi/blob/main/missing/MACRO.AH2

This could be quite different to what they had but it seems to work. I haven't been able to execute the code yet, as I can't really do that until all the missing modules are reconstructed, and everything then linked together, but a quick visual check seems to confirm they're doing what they should, the macros I mean.

I am currently in the middle of reconstructing the ACTION module. This is the one that contains the table of details about the AGI actions (i.e. commands) and also one subroutine to use that table to execute an action. Reconstruction is going well so far. I'll most likely look at the VAR module after that.

[lskovlun]

Another interesting idea might be to compare the macro use between AGI and SCI - if they were using the same, or nearly the same, versions of MASM. The MACRO.I file in SCI contains none of the things you've been talking about. I have only dipped my toes in that, though. I also looked briefly at the version of MASM in Kawa's SCI build environment - and it contains some interesting strings, like .if-.repeat-.while and carry, zero, overFlow, sign, parity, but is missing some others.

[lance.ewing]

I think that the SCI source would be using a newer version of MASM (6.xx) that has those built into MASM itself. The interesting thing is that the .if control flow directive is quite different from what the AGI source is using. One difference is that it ends with a .endif, whereas the one used in AGI ends with .end. Another is that the condition that appears after .if in the SCI source (and supported directly in MASM 6.xx) is more complex. It supports complex conditional expressions, e.g. .if ah && (al == eofCode). The one used in AGI only supports things like: .if not_equal, all variations of which translate to a conditional jump. It relies on the state of the processor already being set, so usually has a cmp immediately before. The SCI code does show examples of that simpler form as well, e.g. the zero, sign, etc. that you mentioned. The MASM 6 documentation mentions the two cases, i.e. "C-style" expressions, and simply checking the state of processor flags. I don't think the one used with AGI supports the C-style expression format.

Some other differences: The AGI code uses "repeat", "while", "until", "break", etc. without the dot at the start.

Given that the MACRO.AH file extracted from the KQ3 disk is dated the 24th March 1987:

1987-03-24 14:24:02 MACRO.AH       728 bytes   728 bytes  From KQ3 2.14 (720K disk 1)

and the INTRPT.ASM file that uses pcall (and several of the other missing macros) is dated only a couple of weeks later:

1987-04-07 09:00:32 INTRPT.ASM    6760 bytes  6760 bytes  From SQ2 2.0D (720K disk 1)

then it seems quite likely that the missing macros were never in the MACRO.AH file and instead came from somewhere else. Although having said that, the fact that the MACRO.AH file on the KQ3 disk has a datetime that isn't the 5th Feb 1987 means that they added something to it, the most recent update being on the 24th March 1987, so they were working on it to some degree. Is it possible that they coded all those missing macros in a couple of weeks? It seems unlikely, especially given that an implementation for pcall does not seem possible until MASM 5.0.

I have noticed that MASM 4 had a beta release that presumably came out before the official release date of MASM 4. That sets some precedent for MASM 5.0 being available in an early release of some kind, but I can find no mention of that, so currently we might have to assume that MASM 5.0 wasn't available until the 31st July 1987.

So if they weren't defined in the MACRO.AH, then what does that leave us? It seems unlikely to be honest that they were defined in a different macro file. If they were custom macros that Sierra wrote, then I'd expect them to be in that file. I have seen that the IBM Macro Assembler supported a pre-processor called SALUT but there is nothing in there that looks like these. However, it did give me the thought that something similar may have been used by Sierra, i.e. a pre-processor of some kind. If it was a standard pre-processor product though, that provided direct support for these directives, then I would expect to find some other examples of code online that uses them, but I haven't turned up anything so far.

Oooohhh, interesting. I just did a search for "masm" across the SCI interpreter code base. Some very interesting comments turned up in the SCI.DOC file that I believe has finally answered the question!!

Code: [Select]


The assembly language preprocessor for Intel code (as.exe and ap86.exe)
are now obsolete.  All assembly source code for the interpreter has been
converted to use the MASM 6.0 structured assembly constructs, which are
far superior.

***** PLEASE use these constructs rather than labels and jumps! *****

If you don't have a copy of MASM 6.0, get one from Larry.

Jackpot!!

So that does indeed solve the mystery. It confirms that they switched over to using the MASM 6.0 structured assembly constructs, but it also confirms the name of the pre-processor they were using prior to that, i.e. AS.EXE and AP86.EXE. I have seen both of these mentioned in the slack space of original AGI game disks, but wasn't able to turn up much on them. The comment above appears to suggest that SCI was using the same as AGI for these "structured assembly constructs" in early versions, i.e. before changing to MASM 6.

The comment in SCI.DOC is from Jeff, dated 3/3/92. So this is roughly when they switched from MASM 5.1 to MASM 6, or shortly before that.

I assume we don't have access to an earlier version of the SCI source? Or better yet, AS.EXE and AP86.EXE hiding somewhere?