Community
AGI Programming => AGI Development Tools => Topic started by: lance.ewing on May 30, 2024, 11:07:23 AM
-
*Split from the "C# AGILE" topic*
I reassessed the amount of original AGI source code that we have, stating now in the github repo README that it is 80%. My original estimate was based on the number of files, which is still calculated to be about 75%, with the inclusion of the extra source files from the KQ3 disk, but if we look instead at the actual amount of code across all modules, then it works out to be 82%. I rounded that down to 80% in the readme, as I didn't want to be too exact, as it probably depends a little on what exactly you count.
It looks like I should be able to increase the stated percentage even further, although I'll need to work out by how much. The reason for being able to increase it is due to some of the modules being from the library files of the Mark Williams C compiler, or from the PLINK86 OVERLAY.LIB library, which means that they're not technically part of the original AGI interpreter source. So far I've identified the OVLYM module as being the Overlay Manager module from PLINK86, and the SETJMP and PRINTF modules being from the Mark Williams libraries. I did wonder about the FILEIO module, but I've seen evidence in the slack space of some original Sierra disks that FILEIO.OBJ was a file that they compiled from source, i.e. not from a LIB file. I can't spot any other modules in the list that may have come from a standard library, so I think OVLYM, SETJMP and PRINTF are the only ones I can ignore for the purposes of determining the percentage of missing AGI source code.
Edit: The percentage didn't end up increasing that much. Ignoring those three modules, the percentage of original AGI interpreter source code that we have is approximately 83.5%.
-
I did wonder about the FILEIO module, but I've seen evidence in the slack space of some original Sierra disks that FILEIO.OBJ was a file that they compiled from source, i.e. not from a LIB file.
Have you looked into the FILEIO source from the SCI interpreter (https://github.com/OmerMor/SCI16/tree/master/INTERP)? Perhaps it's similar.
-
Thanks for the tip Omer. I hadn't looked in the SCI interpreter source yet. Looking over it, there are many similarly named functions in there, so I think the file/module serves a similar purpose. Its not an exact match though. As the original AGI FILEIO module appears to have been an ASM module, then it should be possible to reconstruct it by hand, assuming the actual source doesn't turn up. This is what the PLINK86 memory map has for the FILEIO module:
Module A from file FILEIO
Segment CODE.CODE, Addr = 58EC, Size = 1EE
CLOSE 0:5991
CREAT 0:58EC
DUPLICATE 0:59D3
FILEDT 0:5AB2
FINDFIRST 0:5A40
FINDNEXT 0:5A65
GETCURDIR 0:59F1
GETCURDRIVE 0:5A29
LSEEK 0:59AA
OPEN 0:590D
READ 0:592E
UNLINK 0:5974
VALIDDRIVE 0:5A7D
WRITE 0:5951
Segment DATA.DATA, Addr = AFA1, Size = 0
-
SCI's FILEIO is also done in assembly though.
-
Yeah, I know. Sorry, for the confusion. I think I worded my post poorly. Rather than implying that the SCI FILEIO module was not ASM, what I meant was that since the AGI FILEIO module appears to have been an ASM module (deduced from the AGI.MAP file, which specifies the module name as "A", which appears to be used as the module name for all ASM modules in the AGI.MAP File), then it should be possible to use a disassembly of a similar AGI version (e.g. 2.903) to come up with an ASM module that looks roughly like what the original ASM source file looked like. Even the word "possible" is a bit poorly used here though, since obviously we could use a disassembly to deduce what the C code looked like, if FILEIO had been a C module. So what I really meant to convey was that it would be easier (rather than possible) to put together what the original FILEIO.ASM module looked like in AGI, given that it was an ASM module.
-
Thing is, SCI's FILEIO module is more or less a minimal implementation of the old C library's file I/O functions. Before fopen(char* filename, char* mode), we had just open(char* filename, int mode). They're so standard, the actual open and read functions in SCI aren't even in FILEIO.S but in STDIOASM.S! With a name like that...
Combine the two and you should be able to reconstruct a single assembly module with all the stuff you listed.
-
That does make me feel like they got the code from a standard C library rather than writing it themselves. The Mark Williams Compiler versions 3 and 4 that I looked at didn't appear to be an exact match, although they do have a lot of those functions, just not all of them in a single module, and some I couldn't find at all. I can't say what the earlier versions of the MWC compiler had though, and there is evidence in floppy disk slack space that they were compiling with a v2.3.8 version of MWC at some point. Maybe that had a closer match.
Also in the floppy disk slack space of some original game disks are fragments of a DOS directory table that includes FILEIO.OBJ (alongside other .OBJ files known to be part of AGI), which initially suggested to me that they compiled it themselves, since I thought that the compiler would have something like that in a .LIB library file. Now that I've said that though, I have noticed that the v3 and v4 versions of MWC that I have both have .OBJ files in their LIB folder, in addition to the .LIB files. Perhaps the older versions of MWC had the FILEIO.OBJ file separate but it was then split up, refactored a bit, and included into a LIB file in the v3/v4 versions.
Interestingly, the FILEIO.OBJ file is immediately followed by VGGRAPHX.OBJ in the directory table fragment. That VGGRAPHX module wasn't added into AGI until the 2.9XX versions of AGI. The decoded timestamp from the directory entry for VGGRAPHX.OBJ is the 03-Sept-1987, and the decoded timestamp for FILEIO.OBJ is 25-Aug-1987. The origin of this directory table fragment must have been a hard disk, which seems to be the case for a lot of the slack space of floppy disks, since the cluster number is way beyond the size of a floppy. As I understand it from reading up on this years ago, the slack space data came from data left over in the buffers that DOS (or the copy tool; can't remember which) used when it copies the files to the floppy. So its an interesting view on what Sierra were using on their development machines. In the example I'm looking at, the original game disk is a 2.0C version of SQ2, and the hard disk appears to have had WHERE.SQ2 on it, immediately alongside these .OBJ files. The WHERE file is used with the test/dev non-production version of the AGI EXE. This would have been a WHERE file for SQ2 then.
This thread has gone way off the topic of the C# version of AGILE, but the main culprit is me. I wonder if we can split the recent unrelated posts off into a thread related to the original AGI interpreter source.
-
To be honest, if you grabbed a copy of say Turbo C or whatever, wrote a program that uses open and such from the standard library, and looked at the resulting binary, you'd probably see the same thing as in FILEIO.S: that they're fairly thin interrupt calls to DOS. fopen is much more involved.
-
Wow, you guys are bit-twiddle super sleuthing here. Impressive!
-
Does anyone know what assembler in 1987 would have been used by Sierra with the .ASM files in this folder:
https://github.com/lanceewing/agi/tree/main/src
e.g.
https://github.com/lanceewing/agi/blob/main/src/RANDOM.ASM
That RANDOM one is a very simple example. I assumed that they used MASM, and indeed some of the simpler files (e.g. JRJMPTBL.ASM) do work with MASM, but most of them fail with syntax errors. I've tried MASM 4 and 5. They complain about the .if, .else, .end, enter, exit, return, do, dloop, and a few others. I search online and can't seem to find anything that supports all of these. I'm starting to wonder if they had some custom macros defined. They did have a macros file, and it is included in files such as the RANDOM.ASM one, but the version of the macro.ah file that I extracted from the KQ3 2.14 disk doesn't have anything to support those.
-
This page suggests that those directives were added in MASM 6.0, but it looks like that shipped around 1991?
https://bytepointer.com/masm/index.htm
Also some of the source files directly mention MASM, search this file:
https://github.com/lanceewing/agi/blob/main/src/SCROUT.ASM
-
This page suggests that those directives were added in MASM 6.0, but it looks like that shipped around 1991?
Yeah, I noticed the same thing, e.g. that .if is in the there, but it ends with .endif rather than .end. I haven't tried MASM 6 yet, mainly for the reason that you mentioned, being that it wasn't around in 1987. Even MASM 5 was very new at the time that these .ASM files are from. I think I saw that it came out in Aug 1987. My assumption was that MASM 4 would have been used for most of the AGI source development.
Also some of the source files directly mention MASM, search this file:
https://github.com/lanceewing/agi/blob/main/src/SCROUT.ASM
Nice spotting! I hadn't noticed that yet. So that answers the question, i.e. it is definitely MASM, but for whatever reason, it doesn't like the code. I'm running it without any command line options though, other than the name of the source file. Not sure if I need to enable a feature somehow. Or maybe it gets pre-processed by something first. If I try to use the MWC (Mark Williams compiler) with the .ASM files, they delegate to MASM, but then it gets the same syntax errors. Something is missing.
-
I think it's also possible that Sierra got preview builds of MASM from Microsoft, perhaps that included the .if and other directive features ahead of them shipping in an official release. I wonder if you hunt through the MASM 4.x or 5.x executables if you would find any reference to these things, maybe behind an undocumented flag?
Here's something else that's an interesting point... the DOS 4.0 source code is on github, and includes usages of these directives -- and DOS4 shipped around 86-88?
https://github.com/microsoft/MS-DOS/blob/2d04cacc5322951f187bb17e017c12920ac8ebe2/v4.0/src/SELECT/S_DISPLY.ASM#L64
-
Oh, here's the file from the DOS repo that contains the macro definitions for .if and .endif
https://github.com/microsoft/MS-DOS/blob/2d04cacc5322951f187bb17e017c12920ac8ebe2/v4.0/src/INC/STRUC.INC#L320
Perhaps this file or something like it was distributed with earlier MASM versions? Or maybe Sierra had their own, or was given it from Microsoft?
-
Yeah, it is starting to look like these were defined as macros, given that that example you found has shown that it is possible to define very similar macros. I think we can deduce what the macros contained based on comparing the usage of these macros with a disassembly of the AGI interpreter.
-
I have a memory of Ken or one of the early Sierra devs mentioning using MASM.
-
I have a memory of Ken or one of the early Sierra devs mentioning using MASM.
There's also the SCI changelogs (but this is of course later), which mention a change from MASM 5.1 to 6.0.
-
And I recall adding some .if uses to what might be slightly older parts of SCI 1.001.100. To improve readability, dig?
-
Yeah, it is starting to look like these were defined as macros, given that that example you found has shown that it is possible to define very similar macros. I think we can deduce what the macros contained based on comparing the usage of these macros with a disassembly of the AGI interpreter.
I'm working on putting together some MASM macros that hopefully behave like the ones that the AGI interpreter source is using. Took a while to get my head around what was possible in MASM 4, which is what I'm targeting, since I am assuming that that version of MASM was what was used during the development of most of the code that we have. They probably would have switched to MASM 5 as soon as it was released, but I still think that their macros would have originally been for MASM 4.
Most of the useful documentation online is for 6, and some for 5. I did find a couple of manuals for 4 and earlier as well. That STRUC.INC file from the MS DOS 4 repo is quite useful in seeing what is possible. The trick of creating symbols to represent a stack that things can be pushed to is quite clever and I think also necessary in order to properly handle the nesting of the various block types (loops, ifs). The equivalent macros in that STRUC.INC file are more complex than required though, e.g. they handle testing a condition, but the AGI ones do the test before calling the macro, so the macro doesn't need to be as complex. So I have tried to come up with something a bit simpler. Hopefully it will work. I'll report back on progress after testing it out a bit.
-
Very cool. Let me know if you want any help with that. You should also be able to mostly reconstruct the missing files by ripping the asm straight out of the executable. For a lot of the missing functions, you will probably find source code that calls into those procedures, and so you can cross reference the source code with the disassembled executable in order to know what the function's symbol is.
-
Very cool. Let me know if you want any help with that. You should also be able to mostly reconstruct the missing files by ripping the asm straight out of the executable. For a lot of the missing functions, you will probably find source code that calls into those procedures, and so you can cross reference the source code with the disassembled executable in order to know what the function's symbol is.
Yeah, that is the plan, to reconstruct the missing bits, and then the long term plan is to set up a github action to build it using dosbox with MWC, MASM and PLINK86.
Regarding the symbols, the names are also in the AGI.MAP file, which contains every module. Those are all in uppercase though, whereas the source code references would show the original names, usually upper or lower camel case. I think between both, most symbols could be worked out. The data that is internal to a module isn't included in the AGI.MAP though, so there would be some internal bits with guesses as to the names.
I was discussing the code with AGKorson recently, who did a detailed analysis of the AGI.MAP, and we think this source is earlier than the 2.903 interpreter. Its an unreleased test version between 2.440 and 2.903. Most of it matches 2.903 but there are some bits that still match 2.440. The latest date mentioned in a source code comment is Sept 87. The AGI.MAP is dated the 7th Oct. A game using 2.903 was released about 2 weeks after that.
-
Any chance that some of the source files from the disk slack came from different versions? ie, an old version to reference against a newer version?
-
Any chance that some of the source files from the disk slack came from different versions? ie, an old version to reference against a newer version?
Yeah, that's what I'd think too. If I understand correctly, the files were taken from two very different AGI games.
-
Yeah, you're right, and to prove it, I just compared the IOBJSBRS.ASM file between the SQ2 and KQ3 disks, which is one of the files that appears in both. The SQ2 version has two additional lines:
$ diff IOBJSBRS.ASM.SQ2 IOBJSBRS.ASM.KQ3
18,19d17
< data segment byte public 'data'
< data ends ;dummy segment
Haven't compared others yet but they'd also likely be slightly different. I guess we can compare all the files extracted from the KQ3 disk with the 2.903 disassembly (being the closest released interpreter version) so see what differences there are and tweak as required. The KQ3 files are still useful with regards to things like comments and symbol names and general code layout.
-
Any chance that some of the source files from the disk slack came from different versions? ie, an old version to reference against a newer version?
That was one of the first questions I had when I started looking at the source files in detail. Based on what I've found so far, I'm inclined to believe these are more than likely from the same version, at least most of them. The files appear to match the memory map in size and position, and also match version 2.903 except for a few places where there is code matching 2.440.
There are over 30 dated comments in the source. I sorted those and compared them to the disassembled code from all versions to build a timeline of versions 2.272 (which appears to be the base version of these source files) through 2.903:
2.272: < 17 FEB 1987
2.411: >= 17 APR 1987 < 30 APR 1987
2.425: >= 30 APR 1987 < 11 MAY 1987
2.426: >= 30 APR 1987 < 11 MAY 1987*
2.435: >= 11 MAY 1987 < 27 MAY 1987
2.439: >= 27 MAY 1987 < 01 JUN 1987
2.440: >= 01 JUN 1987 < 14 JUL 1987
2.903: > 01 SEP 1987
*No comments found that are attributable to this version because the only change is a very minor adjustment to the RenderPic function which is in the file PICDRAW.ASM, one of the missing source files.
-
Yeah, you're right, and to prove it, I just compared the IOBJSBRS.ASM file between the SQ2 and KQ3 disks, which is one of the files that appears in both. The SQ2 version has two additional lines:
As AGKorson said, the files on the SQ2 disk are probably from the same date, but the files on the KQ3 disk appear to be from a different time period.
That IOBJSBRS.ASM file might be the only one that is in both actually. I can't spot any others. There is less overlap than I thought. There are some asm header files on the KQ3 disk for which there are C header files on the SQ2 disk. I think Sierra may have maintained C and ASM versions of at least some of the header files. I can see one example where Jeff Stephenson added a comment to the C version only a minute after the ASM version:
;GAME.AH
;Header file for King's Quest
;
;Change History
;18/02/87 15:02:16 JAS
; Increased NUMCONTROL and KEYMAPSIZE to 50.
/* GAME.H
** Header file for King's Quest
**
** Change History:
** 87.04.30 9:58 JAS
** Added NO_PRMPT_RSTRT flag to not prompt user on a restart (allows
** programmer to restart game from logics).
** 18/02/87 15:03:27 JAS
** Increased NUMCONTROL to 50.
*/
(I don't think we can read too much into the "Header file for King's Quest" comment. I don't think this file has anything game specific as such, so that might be a very old comment line that they didn't bother to update)
Interesting that the later comment from April 87 does not have an equivalent in the .AH file, suggesting either that it wasn't required, or that the .AH version predates that change.
-
That NO_PRMPT_RSTRT system flag isn't the only one that the GAME.AH is missing though. All of the following appear in the C version but not the .AH version:
#define TRACE_ENABLE 10 /* to enable tracing */
#define HAS_NOISE 11 /* does machine have noise channel */
#define RESTORE 12 /* restore game in progress */
#define ENABLE_SELECT 13 /* allow selection of objects from inventory screen */
#define ENABLE_MENU 14
#define LEAVE_WIN 15 /* leave windows on the screen */
#define NO_PRMPT_RSTRT 16 /* don't prompt on restart */
Its starting to look like the KQ3 code is quite a bit older, unfortunately.
-
That was one of the first questions I had when I started looking at the source files in detail. Based on what I've found so far, I'm inclined to believe these are more than likely from the same version, at least most of them. The files appear to match the memory map in size and position
There is something unusual about these files that I have noticed many times but haven't really stopped to think about the implications of until now. The observation is that all these files on the SQ2 disk are not cleanly separated by the start of a cluster, i.e. one file runs immediately into the next and that happens within the same cluster, literally the next byte! The logical conclusion is that they were all in the same file. My current assumption is that a backup tool was used to join them all into a single file but where no compression was involved.
This is good for us because it would seem to suggest two things:
- The source files on the SQ2 disk are all from the same point in time, since they were archived together at the same time.
- The AGI.MAP memory map file is also likely to be from the same point in time, e.g. from the last time they compiled that source.
-
Its starting to look like the KQ3 code is quite a bit older, unfortunately.
I have found more evidence that the code from the KQ3 disk is older. I started out with the MACRO.AH file as it appears on the KQ3 disk, but it is clear now that the "bios" macro isn't correct, since one of the uses of that macro assumes that the second parameter is optional, which it isn't. But compare that with the "bios" macro found in the SCI source, where that parameter is optional, and we probably have what the code on the SQ2 disk is expecting.
I was initially thinking that these other macros, such as ".if", "pcall", "break", "while", "until", "repeat", "enter", "exit", "breakif", "retif", "continue", etc. were perhaps in another macro file, but now that it is obvious that the MACRO.AH file on the KQ3 disk is older, then it could be that all those macros were in the version of the MACRO.AH file that the code on the SQ2 disk expects.
-
(I don't think we can read too much into the "Header file for King's Quest" comment. I don't think this file has anything game specific as such, so that might be a very old comment line that they didn't bother to update)
It is interesting nonetheless. If it really came from King's Quest it would have to be the GAL version, since by the time of AGI, Sierra was all about generic code. We have established that GAL is not simply an early kind of AGI. And yet, the file wasn't rewritten from scratch.
I noticed another macro-related thing. They use 'enter' as an assembly macro, but 'enter' is an opcode on 80186 and later processors. So it seems their assembler didn't complain about this.
-
It is interesting nonetheless. If it really came from King's Quest it would have to be the GAL version, since by the time of AGI, Sierra was all about generic code. We have established that GAL is not simply an early kind of AGI. And yet, the file wasn't rewritten from scratch.
Yes, very interesting, in fact it isn't the only reference to a specific game. I'd like to explore that a bit more in a later post. I recall seeing a KQ II reference as well.
I noticed another macro-related thing. They use 'enter' as an assembly macro, but 'enter' is an opcode on 80186 and later processors. So it seems their assembler didn't complain about this.
I've tried MASM 4.0 with every one of the ASM files now, using my implementations of the missing macros. Most of the files are producing .OBJ files without error. Obviously not tested though, so there could very well be errors in my macro implementations. I won't really be able to test them properly until I've fully linked it together. I have successfully compiled all the C source using the MWC compiler (version 4 for that one as well). Its going to be some time before the missing bits are ready though, as I'll have to reconstruct each of those. I've done it already for a couple of files, one of which I haven't yet pushed to the repo.
MASM had no problems with the "enter" macro. It does have problems with a few things though. I think I have three separate issues at the moment, or maybe its four if I'm being picky.
1. The TYPE keyword as understood by MASM is clashing with a "type" symbol in one of the source files. I think there might be an option to support using keywords for symbols. Haven't tried it yet though.
2. The "return" macro is clashing with the RETURN constant that is defined in one of the source files.
3. Some of the "pcall" macro calls are using a & char in front of a string parameter, presumably to get the address of the string. MASM doesn't seem to like this. It outputs a warning and uses a value of 0, which doesn't sound good.
4. Some of the "pcall" macro calls are using a # char in front of either a number (e.g. #15) or a numeric constant (e.g. #TEXTRGHT"). This is presumably meant to indicate an immediate value but MASM doesn't like this either. It doesn't seem to recognise the # char being used in this way, in fact the macro calls would probably work without the #, so not sure why they are needed.
I've searched online a bit, and read through bits of a few MASM books online, but can't see anywhere where # or & are used in this way. It suggests that something else other than MASM was looking at these.
Any ideas?
-
1. The TYPE keyword as understood by MASM is clashing with a "type" symbol in one of the source files. I think there might be an option to support using keywords for symbols. Haven't tried it yet though.
2. The "return" macro is clashing with the RETURN constant that is defined in one of the source files.
Actually, rather than there being an option to support using keyword names as symbol names (I thought I saw one but can't spot it now in the MASM 4 command line options), there is instead an /ML option that allows symbols to retain their case. Using that option allows, for example, "type" to avoid clashing with TYPE.
I don't think the RETURN constant was clashing with the return macro btw. I misinterpreted the error. It's another example of the 4th issue. This is in the SCROUT.ASM file:
pcall PutChar,#RETURN
where RETURN is defined as follows:
RETURN equ 13
-
So, yeah, I think it is mainly the "pcall" macro uses below that are causing issues:
DOTEST.ASM: pcall Error,#15,ax ;bad test number
EQUIPCHK.ASM: pcall _SetTextAtr,#WHITE,#BLACK
EQUIPCHK.ASM: pcall open,&fontFile,#INPUT0
EQUIPCHK.ASM: pcall printf,&noFontMsg
EQUIPCHK.ASM: pcall read,ax,heapBase,#FONTFILESIZE
EQUIPCHK.ASM: pcall printf,&noMemMsg
INTRPT.ASM: pcall WindowNoWait,&badStackMsg
SCROUT.ASM: pcall PutChar,#RETURN ;out of window -- recurse on
SCROUT.ASM: pcall ClearRect,topline,#TEXTLEFT,bottomline,#TEXTRGHT,attribute
i.e. any usage that uses # or &.
The following are working fine:
DOTEST.ASM: pcall TraceTest,ax,savedTestPtr
DOTEST.ASM: pcall CompareStrings,ax,bx
EQUIPCHK.ASM: pcall close,ax
EQUIPCHK.ASM: pcall ErrBeep
SCROUT.ASM: pcall ClrLines, theLine, theLine, clrAtr
There must be a way to translate the # and & into a form that MASM understands. For example, if & could be translated into "OFFSET " then maybe that might work (?). Maybe there is a way to define/override what these chars do. The & char is a tricky one though, as that is by default used for substitution, which I'm making heavy use of in the macro definitions.
-
I think the logic for this is all inside the missing macro. The # and & is indicating to the macro what type of parameter it is, and then the macro must be somehow detecting that and behaving differently. If I compare some pcall usages with the disassembly of the same code, then these are some examples of what gets generated:
#15
becomes:
mov [bp+var_2], 15
push [bp+var_2]
and:
&noMemMsg
becomes:
push ax
lea ax, noMemMsg
mov [bp+var_2], ax
pop ax
push [bp+var_2]
where the [bp+var_2] is a local variable (var_2 is negative).
And then there is the normal case where it pushes the macro parameter as is:
ax
becomes:
push ax
Three different behaviours, based on whether the first char is #, & or neither.
Now I need to read up more on macros to see how this kind of determination logic and be implemented.
-
In the MASM manual there is talk of a "TYPE" operator that can be applied to an expression to see what it is. It says it returns "a number" but I don't see what the possible numbers are. I would make a macro that takes a parameter and print out the TYPE of the parameter, and then I bet you could use that with some IF directives to select the code to emit based on the type of thing (register / word / pointer?) passed to the macro
-
I did this test in the latest masm:
.data
stuff db 54
extrn other_foo:near
.code
foo proc
push TYPE stuff
push TYPE other_foo
push TYPE rax
foo endp
end
Results in:
0000000000000000: 6A 01 push 1
0000000000000002: 68 08 FF 00 00 push 0FF08h
0000000000000007: 6A 08 push 8
-
(it also gives me a syntax error if I try to write &other_foo so I'm not entirely sure what that's about)
-
Version 5.1 of MASM has some new features apparently not in MASM 5.0 or 4 that look like they could be useful, but MASM 5.1 wasn't released until 1988, whereas the code is dated the 7th October 1987. I'm trying to use 4 since this is likely what they used at the time.
-
Have you seen anything that indicates what version interpreter the source is from?
-
Have you seen anything that indicates what version interpreter the source is from?
AGKorson answered that question earlier in this thread:
The files appear to match the memory map in size and position, and also match version 2.903 except for a few places where there is code matching 2.440.
The source code appears to date to a few weeks before the first game using 2.903 was released. The memory map file is dated the 7th October 1987. Version 2.903 was used for an early Police Quest release. I don't have that disk image, so I can't check myself what the timestamp is for the AGI and AGIDATA.OVL files on that disk. I found a date of 23rd October 1987 mentioned online though.
AGKorson has done a detailed analysis of the sizes of the various parts of the AGI.MAP memory map file and deduced that it predates 2.903, but is mostly a match for 2.903, with, as he mentioned in the quote above, a few bits still matching what was in 2.440.
-
Yeah, the only PQ1 I have is v2, the same used for the 4 Most Wanted.
-
In the MASM manual there is talk of a "TYPE" operator that can be applied to an expression to see what it is. It says it returns "a number" but I don't see what the possible numbers are. I would make a macro that takes a parameter and print out the TYPE of the parameter, and then I bet you could use that with some IF directives to select the code to emit based on the type of thing (register / word / pointer?) passed to the macro
I couldn't see how this would work with the values starting with # and &. I think TYPE must be for determining the type of data. The documentation talks about the size being returned for simple types like byte, word, etc. The problem with these values that are prefixed with & and # is that they aren't valid symbols. They don't actually refer to anything as such. Without the & and #, then do refer to something. I think that when these values, e.g. &noMemMsg or #TEXTLEFT, are passed into the macro, they must look like string values at that point, rather than a symbol that references data, since # and & do not have the meaning within MASM that these pcall usages appear to convey. If the macro uses them as-is with an asm instruction, then it wouldn't be valid. When I try to do that, I get the message "error 101: Missing data; zero assumed". It treats it as a warning, which is quite strange actually, as it clearly isn't correct for it to assume a zero value.
I'm stumped to be honest. I've skimmed through the MASM 4 manuals a few times and can't see anything that would be able to do what needs to be done here. It needs two different mechanisms: One is to recognise the &, #, or neither of those based on the first character, and the other mechanism is to get the substring from the second character to the end, which would give the actual symbol name in the case of & or # being present. Later versions of MASM do support INSTR and SUBSTR, which would be useful, because they could presumably be used to detect the # and & characters, and then to get the substring after that character. I looked at the strings in versions 5.0 and 5.1 of the MASM.EXE. Version 5.0 has no mention of INSTR and SUBSTR but version 5.1 does. I feel like it would be cheating though to write something that targets MASM 5.1, since that version wasn't yet available (the two 5.1 versions are dated 31st Jan 1988 and 1st Feb 1988). I may end up having to target version 5.1 though, as I can't see how it can be done in MASM 4.
I did wonder whether maybe they used more than one assembler. As was mentioned earlier in this thread, one of the files, SCROUT.ASM, mentions MASM by name in one of the comments. SCROUT.ASM also happens to be one of the files that uses these strange pcall macro calls, so it is difficult to suggest that a different assembler was used for that file when MASM is mentioned within the file. Of course, it could be the case that the comment is old and that they were using a different assembler at this time and hadn't bothered to remove that comment. It is worth considering. I'm not sure what other assemblers would have been around that might have handled this type of syntax. Or is it possible that some kind of preprocessor was used on the .ASM file before MASM was used?
One thing I'm trying not to think too much about is that the SQ2 disk itself was prepared in mid-March 1988, i.e. that is when the AGI interpreter and game files for SQ2 were copied onto the master disk. So it is theoretically possible that the AGI interpreter source is more recent than the October 1987 date. The problem with this though is that AGKorson's analysis of AGI versions, i.e. the evolution of the different sizes of the various internal data and code parts within the interpreter, places the source code around that 2.903 version, which can't be any later than early Nov 1987, since the 2.911/2.912/2.915 are from Nov 1987, and 2.917 is early Dec 1987. The point to make though is that MASM 5.1 was available prior to March 1988 when the SQ2 disk was prepared, thus the reason why it being theoretically possible that the AGI interpreter source is more recent is of relevance, as it would then bring MASM 5.1 into the picture. I feel like that is stretching the evidence a bit though.
-
I may have finally found something available in MASM 4 that should do the trick. It's the IRPC directive. It loops over the characters in a string. So it should be possible to use that to check the first char and split off the rest. There is an example of doing something similar to that in the MIXED.INC macro file provided with MASM 5.0.
; Split argment into argNameCur and typeNameCur
fetType macro arg
cbType=0
fColon=0
argNameCur equ < >
typeNameCur equ < >
.xcref fColon, argNameCur
irpc aChar,arg
if fColon
conCat typeNameCur, %typeNameCur, aChar
else
ifidni <aChar>,<:>
fColon=1
else
conCat argNameCur,%argNameCur,aChar
endif
endif
endm
adjustType %typeNameCur
endm
This example is splitting strings like the following:
rectype:byte
It looks for the colon in the middle to switch between concatenating to the arg name vs the arg type name.
The cool thing is that the macro code it is using should also work in MASM 4. I'll give this approach a go tomorrow.
-
I managed to download a copy of the Mark Williams compiler owner's manual (it was called "Let's C").
According to the manual, (Version 4, published in 1987 [I don't know the exact date though]) it included an assembler (called 'as'). According to the documentation, you could compile C source files and assembly files at the same time as part of a single project. Maybe that's the assembler they used, instead of MASM.
I haven't read the documentation on as to see if it handles the prefix problem you are seeing. It does say that it support macros.
-
After more reading, it looks like 'Let's C' will use their internal assembler if the source file ends in '.s' and it will use MASM if the source file ends in '.asm'. Since all the assembly source files end in '.asm', it looks like Sierra wasn't using the MWC built in assembler. So I guess that theory is debunked.
-
After more reading, it looks like 'Let's C' will use their internal assembler if the source file ends in '.s' and it will use MASM if the source file ends in '.asm'. Since all the assembly source files end in '.asm', it looks like Sierra wasn't using the MWC built in assembler. So I guess that theory is debunked.
The fact that MWC by default uses MASM when the file extension is .ASM adds further support to MASM being what Sierra used for those files. So we have the comment in the SCROUT.ASM file that mentions MASM, and now also the fact that MWC uses MASM by default for .ASM files. We don't have a copy of their make file for AGI, so its hard to say if they used MWC for everything (with the .ASM files being delegated to MASM), or whether they built the .ASM files directly with MASM. I guess there wouldn't be much difference.
Further support that MASM is what they were using (can't remember if I mentioned this already) is that the floppy disk slack space for some games has directory entries for MASM. I checked the timestamp and file size mentioned in that dir entry and it matches MASM 4 exactly, which is the main reason I started out with using MASM 4.
I think that this IRPC directive is most likely going to work. I've drafted an initial implementation of the pcall macro using IRPC and logically it appears that it should work, from reading through the macro code I've come up with. I just need to work through the various syntax errors I've got in there at the moment, but I have a good feeling I should be able to resolve those and have something that works. Hopefully there will be good news later on.
-
Unfortunately its bad news. The syntax shown in that MASM 5.0 MIXED.INC file doesn't' work in MASM 4. The IRPC bit to loop over the chars does work but not much after that. I'm going to try MASM 5.0 next using this approach. If it works, then I guess I can switch to MASM 5.0, since that was released at the end of July 1987. It is possible that they switched to MASM 5.0 at that time and started to make use of new features. For that to be true, it would mean that they weren't previously using a pcall macro.
-
I'm certainly making more progress with MASM 5.0.
The MASM 5.0 manual does mention a couple of new features not in MASM 4 that would explain why I was finding it hard to get it working under MASM 4:
The use of angle brackets to force string evaluation is a new feature of Version 5.0 of the Macro Assembler. Previous versions tried to evaluate equates as expressions. If the string did not evaluate to a valid expression, MASM evaluated it as a string. This behavior sometimes caused unexpected consequences.
I was aware that MASM 4 didn't support the <> syntax for string equates, but this quote stresses the difficulty that people (like me I guess) had with trying to make MASM 4 treat something as a string. It is possible to define a string equate, by relying on that fallback mechanism described above, but I'm now thinking that there isn't a way in MASM 4 to perform string concatenation, which is certainly possible in MASM 5.0, as I have that bit working now, i.e. building up a string character by character (which seems to be what is required to perform a substring in MASM 5.0 in the absence of the SUBSTR directive, which doesn't become available until 5.1).
The other big difference that make it difficult/impossible under MASM 4 is this one:
%text
MASM computes the expression's value and replaces text with the result. The expression can be either a numeric expression or a text equate. Handling text equates with this operator is a new feature in Version 5.0. Previous versions handled numeric expressions only.
It's that last point, i.e. that MASM 4 handled numeric expressions only, that causes issues with what I'm trying to do in the pcall macro.
I still don't have it working in MASM 5.0 yet. The biggest blocker is with the & character. It is certainly possible to loop over the characters in a string, and it is possible to test for the # character and other normal alphanumeric characters, but trying to test for & isn't working. I think it must be trying to do a substitution, which is what & does within a macro, by default. It is apparently possible to create a & literal using <>, e.g. <&>, but I haven't yet worked out how to make use of that. There is also something going on with types. I am wondering whether internally it treats a single char and a string as different types when doing an identical test, since my logging within the macro shows that I am comparing a character of & (from the irpc loop) with a & string constant (since I can't seem to use & directly in the comparison, due to the substitution issue I mentioned), but the comparison evaluates to false, even though my logging shows that both sides contain &.
Anyway, this is where I am at the moment. I have a couple more ideas to try, but they're starting to become work arounds and hacks. If one of them works though, I'll go with it.
-
Have you looked into the FILEIO source from the SCI interpreter (https://github.com/OmerMor/SCI16/tree/master/INTERP)? Perhaps it's similar.
Omer, I probably should have asked this before now, but I assume that you don't have any original AGI interpreter source, other than what was extracted from the game disks?
Would save some time ;D
-
The biggest blocker is with the & character. It is certainly possible to loop over the characters in a string, and it is possible to test for the # character and other normal alphanumeric characters, but trying to test for & isn't working. I think it must be trying to do a substitution, which is what & does within a macro, by default. It is apparently possible to create a & literal using <>, e.g. <&>, but I haven't yet worked out how to make use of that. There is also something going on with types. I am wondering whether internally it treats a single char and a string as different types when doing an identical test, since my logging within the macro shows that I am comparing a character of & (from the irpc loop) with a & string constant (since I can't seem to use & directly in the comparison, due to the substitution issue I mentioned), but the comparison evaluates to false, even though my logging shows that both sides contain &.
I think I've come up with a convoluted way of working around the above (assuming my hunch on the above is correct). I've got an equate constant for the & char, defined as <&>, and then for each character in the IRPC loop, I used a macro to convert it to a string, by defining another string equate that uses the char's value. I then use another macro to compare the two, and this seems to match for the & char. Now I need to wire that up into the main logic I have. I think I might have it working later today, but I've been there before. I feel a little more confident this time though. Let's see.
-
This would be pretty cool if you got it to work, although I would say it seems like an awful lot of effort in order to enable that syntax, I wonder if Sierra really would have put in that much effort in order to use that syntax. Certainly possible, just seems a lot easier to use something else especially considering there's only a handful of pcall uses in the code?
-
This would be pretty cool if you got it to work, although I would say it seems like an awful lot of effort in order to enable that syntax, I wonder if Sierra really would have put in that much effort in order to use that syntax.
I have got my pcall macro fully working now in MASM 5.0, for all three types of argument. I am convinced that it wouldn't have been possible in MASM 4.
Certainly possible, just seems a lot easier to use something else especially considering there's only a handful of pcall uses in the code?
The fact that there is only a handful of them adds a little weight to it being something new that they were trying out. I can see 15 uses of the pcall macro in the code on the SQ2 disk but no uses in the code on the KQ3 disk. As mentioned recently, the KQ3 code appears to be older by at least half a year. Its not absolute proof obviously. Finding a pcall use in the code on the KQ3 disk would probably have eliminated MASM 5.0 being available to them, but since the code on the SQ2 disk is from Oct 1987, and MASM 5.0 came out at the end of July 1987, it does allow for them to be using it and potentially trying out a few new features and more powerful custom macros.
The most recent change history comment from Jeff Stephenson that I can see is from mid July 1987, and since he appears to be the person who most often added change history comments, I'm wondering whether maybe Jeff moved on to the SCI development at that point and others took over the AGI development, as Jeff is likely to have added further change history comments if he was still working on it. The initials pmk appear for Aug 1987 and Sept 1987. I think this must be Paul Krasno who is credited with working on the game development system for the AGI KQ4 release. I wonder if he was the one introducing the pcall usage. I guess we're unlikely to ever know that.
-
Omer, I probably should have asked this before now, but I assume that you don't have any original AGI interpreter source, other than what was extracted from the game disks?
Would save some time ;D
I'm afraid not... ?\_(ツ)_/?
-
I love how utterly broken Unicode support is on this damn board X3
-
As mentioned recently, the KQ3 code appears to be older by at least half a year.
I have been able to work out the date of the AGI interpreter source code files on the KQ3 2.14 720K Disk 1 floppy disk. Not only are the files on there, but there is also a section that has details about the files, including their name, size, start offset, and datetime. It isn't a standard DOS DIR table format, but it is similar, most likely part of the data written by whatever backup tool they used to make backups of the code.
So, from that, I've been able to determine that most of the files are dated the 5th Feb 1987, except for the MACRO.AH file, which is dated the 24th Mar 1987, and the GAME.AH file, which is dated the 31st March 1987. This range of dates is therefore about 6-8 months before the date of the code on the SQ2 2.0D disk.
The SQ2 disk doesn't have the MACRO.AH file, but given the gap in time, it is certainly possible that many more macros were added in between. I'm just guessing though as to how they were implemented, but I figure that if from a usage perspective they behave the same, then it should be fine.
The dates would associate most of the code extracted from the KQ3 disk to roughly the 2.272 AGI interpreter version, whereas the GAME.AH and MACRO.AH files are getting close to the date of the 2.411 version. They would obviously have had lots of internal builds between Feb 1987 to April 1987 to account for the jump in version number over that time. All we see are the released versions.
I can't remember if we have previously discussed the big jump in version number from the 2.440 version to the 2.903 version. It suggests many internal builds, and therefore a fair bit of development. There is a roughly 4 month gap between those two versions.
-
I can see 15 uses of the pcall macro in the code on the SQ2 disk but no uses in the code on the KQ3 disk. As mentioned recently, the KQ3 code appears to be older by at least half a year. Its not absolute proof obviously. Finding a pcall use in the code on the KQ3 disk would probably have eliminated MASM 5.0 being available to them, but since the code on the SQ2 disk is from Oct 1987, and MASM 5.0 came out at the end of July 1987, it does allow for them to be using it and potentially trying out a few new features and more powerful custom macros.
This may be true of the pcall macro, but I've been looking over the code on the KQ3 disk more closely and can see that it uses the following macros that are not in its copy of the MACRO.AH file: enter, exit, do, until, dloop, .if, .else, .end. I already had all of those working under MASM 4, and since there are no uses of pcall in the code on the KQ3 disk, it doesn't rule out a switch to MASM 5.0 by Oct 1987. It is interesting that these macros (enter, exit, until, dloop, do, .if, .else, .end) are not defined in the MACRO.AH file. That suggests that there was another macro file somewhere, but the problem is that the source files that use these macros only include the MACRO.AH file, so there would have to have been another way that MASM pulled in the definitions of these macros.
I have been able to work out the date of the AGI interpreter source code files on the KQ3 2.14 720K Disk 1 floppy disk. Not only are the files on there, but there is also a section that has details about the files, including their name, size, start offset, and datetime. It isn't a standard DOS DIR table format, but it is similar, most likely part of the data written by whatever backup tool they used to make backups of the code.
I have noticed that the SQ2 disk has a similar section with details about some of the files, which will be interesting to look at, as this would give timestamps to a few of the files. Up to now, I've assumed that the code is all from the date of the memory map file, but if any have a timestamp newer than that, then that obviously wouldn't be true. We'd expect them all to be on or before the date of the memory map file. I'll take a look at that later today.
-
I have decoded the details about the files on the SQ2 disk. It turns out that there are details for 85 distinct files, which covers most of the ones for which there is source on the disk. In the process of decoding this, I've discovered that the format is the DOS BACKUP format, i.e. the format created by the BACKUP DOS command. The fields match exactly. So that answers the question as to what tool they used. The physical floppy disk must have been used with the BACKUP command to create an archive of the AGI source, plus a few extras.
The following is the full list of files mentioned, their dates and sizes. Notice that there are two sizes: One is the total size of the original file. The other is the amount of the file in bytes actually stored on the disk. In most cases, the two sizes are the same, but for one of them, JRGRAPHX.ASM, the second size is smaller. This is because the disk ends before the whole file can be completed, so it is only a part of the file. The rest of it would presumably continue on the next floppy.
1987-02-05 ANIOBJ.H 2892 bytes 2892 bytes
1987-02-05 BEEP.ASM 758 bytes 758 bytes
1987-02-05 DIRFLAGS.H 87 bytes 87 bytes
1987-02-05 DOSFLAGS.H 89 bytes 89 bytes
1987-02-05 ERRMSG.H 1046 bytes 1046 bytes
1987-02-05 EVENT.H 642 bytes 642 bytes
1987-02-05 FLAG.ASM 2624 bytes 2624 bytes
1987-02-05 FLAGS.H 89 bytes 89 bytes
1987-02-05 HDIALOG.C 1593 bytes 1593 bytes
1987-02-05 INITDIR.ASM 696 bytes 696 bytes
1987-02-05 JRJMPTBL.ASM 415 bytes 415 bytes
1987-02-05 KBDDRV.C 1331 bytes 1331 bytes
1987-02-05 KEYDEFS.H 777 bytes 777 bytes
1987-02-05 KEYMAP.C 481 bytes 481 bytes
1987-02-05 LOG.C 1352 bytes 1352 bytes
1987-02-05 LOGIC.H 528 bytes 528 bytes
1987-02-05 MISC.ASM 3091 bytes 3091 bytes
1987-02-05 MOVEOBJS.C 2608 bytes 2608 bytes
1987-02-05 NEWROOM.C 2170 bytes 2170 bytes
1987-02-05 OBJACT.C 1421 bytes 1421 bytes
1987-02-05 OBJECT.H 494 bytes 494 bytes
1987-02-05 OBJLIST.H 627 bytes 627 bytes
1987-02-05 PICOBJ.C 1981 bytes 1981 bytes
1987-02-05 PICOBJ.H 317 bytes 317 bytes
1987-02-05 PICTURE.H 207 bytes 207 bytes
1987-02-05 POSITION.C 2721 bytes 2721 bytes
1987-02-05 PRIORITY.C 904 bytes 904 bytes
1987-02-05 PRODIO.H 450 bytes 450 bytes
1987-02-05 SAVEAREA.C 1218 bytes 1218 bytes
1987-02-05 SCRIPT.H 400 bytes 400 bytes
1987-02-05 SEGERR.H 371 bytes 371 bytes
1987-02-05 SEGIO.H 482 bytes 482 bytes
1987-02-05 SOUND.C 2199 bytes 2199 bytes
1987-02-05 SOUND.H 209 bytes 209 bytes
1987-02-05 TEXTWIN.H 148 bytes 148 bytes
1987-02-05 TRACE.H 92 bytes 92 bytes
1987-02-05 TYPES.H 774 bytes 774 bytes
1987-02-05 VIEW.C 6793 bytes 6793 bytes
1987-02-05 VIEW.H 990 bytes 990 bytes
1987-02-18 SEGPTR.C 3558 bytes 3558 bytes
1987-02-19 PICTURE.C 3123 bytes 3123 bytes
1987-02-19 TRACE.C 5163 bytes 5163 bytes
1987-02-20 PRINT.C 10989 bytes 10989 bytes
1987-03-24 JOYREAD.ASM 2478 bytes 2478 bytes
1987-03-25 MOVETO.C 2132 bytes 2132 bytes
1987-03-25 OBJLIST.C 4585 bytes 4585 bytes
1987-03-25 STATUS.C 4537 bytes 4537 bytes
1987-03-25 WANDER.C 846 bytes 846 bytes
1987-04-07 INTRPT.ASM 6760 bytes 6760 bytes
1987-04-07 MOTION.C 3990 bytes 3990 bytes
1987-04-30 CMDATA.ASM 937 bytes 937 bytes
1987-04-30 GAME.H 3867 bytes 3867 bytes
1987-04-30 INIT.C 3805 bytes 3805 bytes
1987-04-30 RESTART.C 862 bytes 862 bytes
1987-05-11 TIMERINT.C 1412 bytes 1412 bytes
1987-05-12 JOYDRV.C 3732 bytes 3732 bytes
1987-05-12 MSGSTR.C 1019 bytes 1019 bytes
1987-05-12 SEGMENT.C 6643 bytes 6643 bytes
1987-07-14 STRING.C 3946 bytes 3946 bytes
1987-07-23 CMOBJSBR.ASM 18448 bytes 18448 bytes
1987-07-24 MEMMGR.C 2299 bytes 2299 bytes
1987-07-27 PARSE.C 5531 bytes 5531 bytes
1987-08-16 VGJMPTBL.ASM 578 bytes 578 bytes
1987-08-21 DOTEST.ASM 8701 bytes 8701 bytes
1987-08-24 INITMACH.VGA 1866 bytes 1866 bytes
1987-08-24 IOBJSBRS.ASM 6853 bytes 6853 bytes
1987-08-24 RANDOM.ASM 758 bytes 758 bytes
1987-08-24 SAVENAME.C 1951 bytes 1951 bytes
1987-08-25 INITDIR.OBJ 118 bytes 118 bytes
1987-08-31 MAIN.C 2800 bytes 2800 bytes
1987-08-31 SQSG.3 2774 bytes 2774 bytes
1987-09-03 GETGAME.C 9437 bytes 9437 bytes
1987-09-03 HGRAPHX.OBJ 1852 bytes 1852 bytes
1987-09-03 HOBJSBRS.OBJ 963 bytes 963 bytes
1987-09-03 INTRPT.OBJ 1442 bytes 1442 bytes
1987-09-03 RESTGAME.C 2611 bytes 2611 bytes
1987-09-03 SENDIT.BAT 39 bytes 39 bytes
1987-09-04 CMGRAPHX.ASM 16569 bytes 16569 bytes
1987-09-09 MENU.C 8700 bytes 8700 bytes
1987-09-29 SCRIPT.C 2195 bytes 2195 bytes
1987-09-30 EQUIPCHK.ASM 8204 bytes 8204 bytes
1987-09-30 JRGRAPHX.ASM 7875 bytes 6144 bytes
1987-09-30 SCRACT.C 4869 bytes 4869 bytes
1987-10-01 SCROUT.ASM 7185 bytes 7185 bytes
1987-10-07 PRODFLAG.H 27 bytes 27 bytes
There are a number of things to draw attention to with regards to the dates:
- The earliest and most common date mentioned is the 5th Feb 1987, which is the same date as most of the files on the KQ3 disk.
- The most recent date is the 7th October 1987, i.e. the same date as included in the memory map file. There isn't any later date. That's a strong indication that the source files are a match for the memory map file, which we were already fairly confident in.
- Many of the files were updated during July, August, and September of 1987. This must represent the internal builds that progressed the versioning from the 2.4XX versions up to the 2.9XX versions
I have checked the file sizes mentioned and they match the actual sizes of the files that were extracted from the SQ2 disk, which supports the content being a single DOS BACKUP file, and that these dates are highly likely to represent the actual dates of each extracted file. It all ties together.
There is one thing about the dates that worries me, and that is the timestamp of the INTRPT.ASM file. There are four files that use the pcall macro. Three of those have a date after the release of MASM 5.0, but INTRPT.ASM is dated the 7th April 1987, nearly 4 months before MASM 5.0 was released. That would seem to eliminate the idea that they wrote the pcall macro after switching to MASM 5.0, unless they got an early release. It could mean that they somehow had it working in MASM 4.
-
I have been able to successfully build all "complete" files of those extracted from the SQ2 & KQ3 disks. As mentioned, one of them is truncated due to the end of the disk.
This is a link to the macro file containing my implementations of the missing macros:
https://github.com/lanceewing/agi/blob/main/missing/MACRO.AH2
This could be quite different to what they had but it seems to work. I haven't been able to execute the code yet, as I can't really do that until all the missing modules are reconstructed, and everything then linked together, but a quick visual check seems to confirm they're doing what they should, the macros I mean.
I am currently in the middle of reconstructing the ACTION module. This is the one that contains the table of details about the AGI actions (i.e. commands) and also one subroutine to use that table to execute an action. Reconstruction is going well so far. I'll most likely look at the VAR module after that.
-
Another interesting idea might be to compare the macro use between AGI and SCI - if they were using the same, or nearly the same, versions of MASM. The MACRO.I file in SCI contains none of the things you've been talking about. I have only dipped my toes in that, though. I also looked briefly at the version of MASM in Kawa's SCI build environment - and it contains some interesting strings, like .if-.repeat-.while and carry, zero, overFlow, sign, parity, but is missing some others.
-
I think that the SCI source would be using a newer version of MASM (6.xx) that has those built into MASM itself. The interesting thing is that the .if control flow directive is quite different from what the AGI source is using. One difference is that it ends with a .endif, whereas the one used in AGI ends with .end. Another is that the condition that appears after .if in the SCI source (and supported directly in MASM 6.xx) is more complex. It supports complex conditional expressions, e.g. .if ah && (al == eofCode). The one used in AGI only supports things like: .if not_equal, all variations of which translate to a conditional jump. It relies on the state of the processor already being set, so usually has a cmp immediately before. The SCI code does show examples of that simpler form as well, e.g. the zero, sign, etc. that you mentioned. The MASM 6 documentation mentions the two cases, i.e. "C-style" expressions, and simply checking the state of processor flags. I don't think the one used with AGI supports the C-style expression format.
Some other differences: The AGI code uses "repeat", "while", "until", "break", etc. without the dot at the start.
Given that the MACRO.AH file extracted from the KQ3 disk is dated the 24th March 1987:
1987-03-24 14:24:02 MACRO.AH 728 bytes 728 bytes From KQ3 2.14 (720K disk 1)
and the INTRPT.ASM file that uses pcall (and several of the other missing macros) is dated only a couple of weeks later:
1987-04-07 09:00:32 INTRPT.ASM 6760 bytes 6760 bytes From SQ2 2.0D (720K disk 1)
then it seems quite likely that the missing macros were never in the MACRO.AH file and instead came from somewhere else. Although having said that, the fact that the MACRO.AH file on the KQ3 disk has a datetime that isn't the 5th Feb 1987 means that they added something to it, the most recent update being on the 24th March 1987, so they were working on it to some degree. Is it possible that they coded all those missing macros in a couple of weeks? It seems unlikely, especially given that an implementation for pcall does not seem possible until MASM 5.0.
I have noticed that MASM 4 had a beta release that presumably came out before the official release date of MASM 4. That sets some precedent for MASM 5.0 being available in an early release of some kind, but I can find no mention of that, so currently we might have to assume that MASM 5.0 wasn't available until the 31st July 1987.
So if they weren't defined in the MACRO.AH, then what does that leave us? It seems unlikely to be honest that they were defined in a different macro file. If they were custom macros that Sierra wrote, then I'd expect them to be in that file. I have seen that the IBM Macro Assembler supported a pre-processor called SALUT but there is nothing in there that looks like these. However, it did give me the thought that something similar may have been used by Sierra, i.e. a pre-processor of some kind. If it was a standard pre-processor product though, that provided direct support for these directives, then I would expect to find some other examples of code online that uses them, but I haven't turned up anything so far.
Oooohhh, interesting. I just did a search for "masm" across the SCI interpreter code base. Some very interesting comments turned up in the SCI.DOC file that I believe has finally answered the question!!
The assembly language preprocessor for Intel code (as.exe and ap86.exe)
are now obsolete. All assembly source code for the interpreter has been
converted to use the MASM 6.0 structured assembly constructs, which are
far superior.
***** PLEASE use these constructs rather than labels and jumps! *****
If you don't have a copy of MASM 6.0, get one from Larry.
Jackpot!!
So that does indeed solve the mystery. It confirms that they switched over to using the MASM 6.0 structured assembly constructs, but it also confirms the name of the pre-processor they were using prior to that, i.e. AS.EXE and AP86.EXE. I have seen both of these mentioned in the slack space of original AGI game disks, but wasn't able to turn up much on them. The comment above appears to suggest that SCI was using the same as AGI for these "structured assembly constructs" in early versions, i.e. before changing to MASM 6.
The comment in SCI.DOC is from Jeff, dated 3/3/92. So this is roughly when they switched from MASM 5.1 to MASM 6, or shortly before that.
I assume we don't have access to an earlier version of the SCI source? Or better yet, AS.EXE and AP86.EXE hiding somewhere?
-
I think that the SCI source would be using a newer version of MASM (6.xx) that has those built into MASM itself.
6.11 in my case.
-
It looks like they had various preprocessors for the different platforms, which makes sense. I found mention of:
AP65.EXE
AP68K.EXE
AP86.EXE
...in the slack space of a KQ3 v1.01 int 2.272 disk. So these would be the preprocessors for the 6502 (for Apple II), 68K processors (Amiga, Atari ST and Mac), and the 8086 processors (DOS). The question now is: What product did these tools come with? Was it a product? Or did they write these themselves? I'm not finding anything so far online.
I have also found a file in slack space called AP86.LNK, which is a bit suggestive. In addition to that .LNK file, I have also found mentioned:
BILOAD.LNK
SIERRA.LNK
-
Maybe this (make file?) would give more clues:
https://github.com/historicalsource/leisure-suit-larry-1-alt/blob/8d7352d4d08336a97d92432a66c22a950a50aedc/SND/READMIDI#L5
readmidi.obj: readmidi.c
msc readmidi /Zi /Od;
midi_int.obj: midi_int.s
ap86 midi_int.s midi_int.tmp
masm midi_int.tmp;
readmidi.exe: readmidi.obj midi_int.obj
link readmidi+midi_int,readmidi,, /co;
-
My God, there was something I'd missed in that package. I thought I'd seen everything there was to find.
-
Maybe this (make file?) would give more clues:
https://github.com/historicalsource/leisure-suit-larry-1-alt/blob/8d7352d4d08336a97d92432a66c22a950a50aedc/SND/READMIDI#L5
readmidi.obj: readmidi.c
msc readmidi /Zi /Od;
midi_int.obj: midi_int.s
ap86 midi_int.s midi_int.tmp
masm midi_int.tmp;
readmidi.exe: readmidi.obj midi_int.obj
link readmidi+midi_int,readmidi,, /co;
Thanks Omer. Very interesting. So that seems pretty clear then. They passed at least some assembler files through AP86 first, then used MASM. A shame that the .TMP file wasn't in that repo. That would have been interesting, to see the in between code.
Still not sure if AP86 was something that they wrote themselves but I'm currently assuming that. That other BILOAD.LNK file certainly appears to have been associated with them building something called BILOAD.COM. The same fragment of slack space on the floppy where this appears (its a fragment of a DOS directory table) also includes the files: MKBI.BAT, BILOAD.MAP, BILOAD.LNK and BILOAD.COM. The .LNK file appears to be part of the output of building certain files. So the existence of an AP86.LNK file in the slack space on another original Sierra game disk suggests that AP86 was also built by them.
What it is looking like then is that I have written the equivalent of what AP86 did but in MASM macro form. In the case of pcall, it only works with MASM 5.0 and above, but they were most likely still using MASM 4. I'm in two minds what to do then. It doesn't feel right to be using MASM 5.0. I might have to build my own equivalent to AP86 that isn't part of the main MASM execution step.
-
There is this tidbit from the SCI changelog:
3/3/92 - Jeff Stephenson
[...]
The assembly language preprocessor for Intel code (as.exe and ap86.exe)
are now obsolete. All assembly source code for the interpreter has been
converted to use the MASM 6.0 structured assembly constructs, which are
far superior.
***** PLEASE use these constructs rather than labels and jumps! *****
there's at least one commercially available preprocessor tool, made by HP, by the name ap86. But no references to ap65 or ap68k, so I think you're right. I wonder what BILOAD.LNK is (you didn't post it) - could be either the standard Sierra loader or a special loader for GAL (and other booters) so you wouldn't have to boot all the time during development. Or something else entirely.
-
there's at least one commercially available preprocessor tool, made by HP, by the name ap86.
I found this document that mentions it:
http://www.bitsavers.org/pdf/hp/64700/software_toolchain/B1449-97000_8086_Assembler_Apr93.pdf#page=54
ap86 accepts the macro preprocessor language that is described in the Intel 8086 Assembler Reference Manual. This macro language allows the definition and use of macros, evaluation and replacement of expressions, loop control, and including of other text files. Correct use of a macro preprocessor can simplify the task of writing assembly language source when redundant operations are performed or code is shared between files.
I believe this is referring to the following:
http://www.bitsavers.org/pdf/intel/ISIS_II/121703-003_ASM86_Language_Reference_Manual_Mar85.pdf#page=285
-
It is interesting to compare the various Sierra booter games. Many of them share the two-stage loader scheme (which would generally be required on an IBM PC) but with different and comparable version numbers. Others have written about the LOADER 2.0 versus LOADER 3.0 in DOS AGI games, but there are earlier ones out there. These were used in non-AGI games as well. In KQ1, the main executable is an EXE file, but is not run in a DOS environment. So the loader has to implement the EXE file parser itself. Even the Championship Boxing diskette uses these tools (now known as ABSBOOT and ABSLOAD) - but I don't think there's an EXE file there.
-
there's at least one commercially available preprocessor tool, made by HP, by the name ap86.
I found this document that mentions it:
http://www.bitsavers.org/pdf/hp/64700/software_toolchain/B1449-97000_8086_Assembler_Apr93.pdf#page=54
ap86 accepts the macro preprocessor language that is described in the Intel 8086 Assembler Reference Manual. This macro language allows the definition and use of macros, evaluation and replacement of expressions, loop control, and including of other text files. Correct use of a macro preprocessor can simplify the task of writing assembly language source when redundant operations are performed or code is shared between files.
I believe this is referring to the following:
http://www.bitsavers.org/pdf/intel/ISIS_II/121703-003_ASM86_Language_Reference_Manual_Mar85.pdf#page=285
I am fairly sure that this HP ap86 is something different. The macro calling syntax is quite different. It appears that this HP tool requires a meta character at the start of the call and for the parameters to be in brackets.
%MOVE_ADD_GEN(INPUT, STORE, 100H)
The calls within the AGI source do not require a meta character at the start and the parameters appear without brackets, e.g.
pcall ClearRect,topline,#TEXTLEFT,bottomline,#TEXTRGHT,attribute
pcall open,&fontFile,#INPUT0
-
I wonder what BILOAD.LNK is (you didn't post it) - could be either the standard Sierra loader or a special loader for GAL (and other booters) so you wouldn't have to boot all the time during development. Or something else entirely.
I'm not sure what it is, as there is only DOS directory table entries that I have found, not any part of the file itself. Let's dig a bit into that though. Where I found it was in a slack space fragment on a PQ1 disk:
Police Quest (1987) (v2.0G, Int. 2.917) (360K) (Disk 1)
...around offset $13F00. The DOS directory table fragment also contains the following files, which are also very interesting:
MKNONPRO.BAT
TESTSQ .BAT
MESSAGE.O
DOSRELOC.S
AGILOAD.O
BILOAD .MAP
MKBI.BAT
CHECK.MAP
MKCHECK.BAT
BILOAD.LNK
LOAD.COM
BILOAD.COM
GRAPHICS.O
I wonder if, given they appear in the same directory, we can assume that they are related. Obviously the AGILOAD.O implies a connection to AGI, and TESTSQ.BAT suggests a connection to Space Quest (despite this being a PQ disk, but that isn't unusual really, as these kinds of fragments usually came from a completely different disk, quite often a hard disk). What we can do is decode the other details of these DOS directory entries, such as the datetime and size, to reveal a bit more info. For example:
BILOAD.COM 2797 30-Jun-1987 08:08:02 [cluster=6694]
AGILOAD.O 1546 16-Nov-1987 13:31:58 [cluster=4838]
I'll check the dates for the others later today but the dates of the two above suggest it is the loader used by AGI. The cluster numbers definitely suggest the origin was a hard disk.
-
The assembler program that was bundled with their C compiler, Let's C, was named 'as.exe'. This is what the manual has to say about it:
as is a multipass assembler that will assemble functions written in i8086 assembly language. as will assemble programs into either SMALL or LARGE model, and will generate an object module in MS-DOS object format. It also supports i8087 opcodes, and it allows you to write functions in a model-independent manner.
as is not intended to be used for full-scale assembly-language programming; therefore, it does not include some of the more elaborate features found in full-fledged assemblers. For example, it has no facility for conditional compilation or user-defined macros. However, Let?s C allows you to use preprocessor instructions to perform conditional assembly and expand macros. In addition, as optimizes branches to take advantage of short addressing forms, where the span of the branch permits.
It says it doesn't support user-defined macros, but it will expand macros. But then it goes on to say that files must be named *.s or as won't run.
Is it just a coincidence that this file is named as.exe?
-
I'll check the dates for the others later today but the dates of the two above suggest it is the loader used by AGI. The cluster numbers definitely suggest the origin was a hard disk.
These are the ones that I think are worth highlighting at this point:
MKBI.BAT 178 29-Jun-1987 10:42:02 [cluster=6685]
BILOAD.LNK 101 29-Jun-1987 10:42:38 [cluster=6689]
BILOAD.COM 2797 30-Jun-1987 08:08:02 [cluster=6694]
BILOAD.MAP 2736 30-Jun-1987 08:08:02 [cluster=6682]
LOAD.COM 1833 29-Jun-1987 12:01:04 [cluster=6690]
LOAD.MAP 2092 29-Jun-1987 12:01:02 [cluster=6710]
MKCHECK.BAT 146 29-Jun-1987 10:40:20 [cluster=6688]
CHECK.COM 2686 29-Jun-1987 12:02:26 [cluster=6715]
CHECK.MAP 2622 29-Jun-1987 12:02:26 [cluster=6686]
LOADCHK.EXE 3457 29-Jun-1987 10:45:00 [cluster=6702]
MKCRYPT.BAT 172 29-Jun-1987 10:37:48 [cluster=6719]
DECRYPT.MAP 2620 29-Jun-1987 12:01:30 [cluster=6704]
DECRYPT.COM 2689 29-Jun-1987 12:01:30 [cluster=6724]
SIERRA.LNK 88 29-Jun-1987 10:27:00 [cluster=6718]
MKNONPRO.BAT 197 05-Nov-1987 10:52:44 [cluster=1709]
PROTECT.COM 3121 05-Nov-1987 10:54:38 [cluster=5083]
PROTECT.MAP 2807 05-Nov-1987 10:54:36 [cluster=4961]
AGILOAD.O 1546 16-Nov-1987 13:31:58 [cluster=4838]
GRAPHICS.O 441 30-Sept-1986 10:19:22 [cluster=6696]
MESSAGE.O 1070 16-Apr-1987 14:09:22 [cluster=6676]
DOSRELOC.S 5048 04-Nov-1987 10:17:02 [cluster=1655]
DOSCPC.O 1469 30-Jun-1987 08:07:28 [cluster=6728]
DOSRELOC.O 747 16-Nov-1987 13:31:30 [cluster=3484]
DOSERR.O 308 28-Apr-1987 08:38:08 [cluster=6717]
Particularly the first batch, which are all dated around the same date and appear to show several small executables and the artifacts related to building those executables. I wanted to highlight again the .LNK files, both the BILOAD.LNK and SIERRA.LNK files. It is relevant to AP86 because of the existence of AP86.LNK in a DOS directory table entry in the slack space of one of the disks.
AP86.LNK 21 07-Sept-1986 17:02:58 [cluster=342]
Hmmm, didn't expect it to be so small though. Whatever these .LNK files are, they're always quite small. I also found the details of the AP86.EXE file on one of the disks:
AP86.EXE 17742 28-Sept-1987 11:35:04 [cluster=1511]
Edit: I found another occurrence of AP86.EXE in a slack space DOS directory table entry, which gives a slightly different size and date:
AP86.EXE 18070 26-Mar-1987 16:36:38 [cluster=4876]
But its still roughly the same ballpark with regards to size.
Edit 2: I have found yet another occurrence of AP86.EXE in a slack space DOS directory table entry:
AP86.EXE 14848 01-Oct-1986 12:10:02 [cluster=52]
-
The assembler program that was bundled with their C compiler, Let's C, was named 'as.exe'. This is what the manual has to say about it:
as is a multipass assembler that will assemble functions written in i8086 assembly language. as will assemble programs into either SMALL or LARGE model, and will generate an object module in MS-DOS object format. It also supports i8087 opcodes, and it allows you to write functions in a model-independent manner.
as is not intended to be used for full-scale assembly-language programming; therefore, it does not include some of the more elaborate features found in full-fledged assemblers. For example, it has no facility for conditional compilation or user-defined macros. However, Let?s C allows you to use preprocessor instructions to perform conditional assembly and expand macros. In addition, as optimizes branches to take advantage of short addressing forms, where the span of the branch permits.
It says it doesn't support user-defined macros, but it will expand macros. But then it goes on to say that files must be named *.s or as won't run.
Is it just a coincidence that this file is named as.exe?
Yeah, I did wonder whether Jeff meant the AS.EXE tool that comes with MWC when his comment in the SCI change log says:
The assembly language preprocessor for Intel code (as.exe and ap86.exe)
are now obsolete.
I wonder why two tools would have been required. Wouldn't it have been possible to write a small pre-processor as a single executable? - It seems clear that they used MASM for the actual assembler, and that snippet that Omer found appears to show that, i.e. ap86 followed by masm. No need for AS.EXE.
I think that they would have had the AS.EXE tool from the MWC compiler on their machines, since it comes in the BIN directory of the distribution disks, the same BIN directory that has the C compiler EXE files. The various EXE files for the C compiler steps appear in slack space DOS directory table entries across multiple original AGI game disks, and there are also occurrences of AS.EXE. The question is whether it really is the one from MWC, or is the one Jeff is referring to, or are they're both the same tool... and if its the latter, then how was it used with AP86?
Edit: I have found two occurrences of AS.EXE in DOS directory table entries in the slack space of original AGI games disks:
AS.EXE 31409 29-Nov-1985 09:09:26 [cluster=29]
AS.EXE 13980 23-Jul-1987 11:36:50 [cluster=9128]
It is curious that the older occurrence is larger in size.
-
I wanted to highlight again the .LNK files, both the BILOAD.LNK and SIERRA.LNK files. It is relevant to AP86 because of the existence of AP86.LNK in a DOS directory table entry in the slack space of one of the disks.
AP86.LNK 21 07-Sept-1986 17:02:58 [cluster=342]
Hmmm, didn't expect it to be so small though. Whatever these .LNK files are, they're always quite small.
I am not sure if this is true of the MS DOS LINK tool, but certainly for the PLINK86 tool that was used as the linker for the AGI executable and overlays, the .LNK file extension is the default extension for the linker "command" file. I think that Sierra would have used PLINK86 for some things, like AGI where the PLINK overlay system was used, but also LINK for other tools. LINK did also support a command input file but I'm not sure if .LNK was also the default file extension for that. Regardless, it seems very likely that these .LNK files were to instruct the linker, and therefore AP86.LNK would have been for defining how to link AP86.EXE. With a size of 21 bytes, it wouldn't have said much at all, perhaps only enough to specify the name of the input and output files. I am assuming it was a simple one module program, so maybe there was an AP86.OBJ that was linked into AP86.EXE.