Community

AGI Programming => AGI Development Tools => Topic started by: Collector on November 02, 2018, 11:32:17 PM

Title: C# AGI Interpreter
Post by: Collector on November 02, 2018, 11:32:17 PM
BTW, lance, not to hijack the thread, but did you get any further with your C# interpreter?
Title: Re: C# AGI Interpreter
Post by: lance.ewing on November 03, 2018, 08:38:56 AM
I haven't worked on it since May 2017, but I really must get back to it. I had just finished the save game feature at that point, so should be able to try playing through a few games now.

Title: Re: C# AGI Interpreter
Post by: Collector on November 03, 2018, 10:13:20 AM
Perhaps I should split this off its own thread and leave this thread for Nickyv2003's question.
Title: Re: C# AGI Interpreter
Post by: Collector on November 04, 2018, 10:22:34 AM
Split from Nickyv2003's walkthrough thread.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on November 04, 2018, 11:42:01 AM
I haven't worked on it since May 2017, but I really must get back to it. I had just finished the save game feature at that point, so should be able to try playing through a few games now.

I've reinstalled Visual Studio, checked out the source code again, and started trying to work out where I got to. It looks like I was starting to work on decoding the sound resources. The AGI Library didn't support that, so I was starting to add that in. I lost my old local checkout of the code when my Windows VM died at some point mid last year. A good thing I was regularly pushing the code to bitbucket.

I think I still had to return to the rendering at some point. It was mostly in line with the original AGI interpreter, but there were a few areas where it wasn't quite right. To be honest though, it would only have been in quite obscure scenarios and probably difficult to notice.
Title: Re: C# AGI Interpreter
Post by: Collector on November 04, 2018, 04:02:36 PM
Did I ever give you the source to the Visual AGI sound editor? They never got around porting it to C# from C++, but it may be of some help. And if you get anywhere with adding it to the AGI Library it could help with adding it to the rest of Visual AGI IDE. At that point the only big thing left for it would be the compiler. I did contact Gustaf Wennerholm on Facebook a while back. He probably be interested if we managed to complete Visual AGI.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on November 05, 2018, 02:17:43 PM
Yes, I think you did send it to me, and I have got it somewhere. I will probably take a look at it to get some ideas.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on December 26, 2018, 01:33:07 PM
I've been making some progress on the C# AGI interpreter. The sounds are now playing. I went down the route of using some SN76489 emulation code to generate 16 bit  sample data at 44100 samples per second. The SN76489 is the sound chip used in the IBM PC Jr and therefore what Sierra coded the original AGI interpreter to output to. As you probably know, all the AGI sounds had four part sound as a consequence of this, the fourth voice being noise. I found some SN76489 emulation code already available on the Internet written in C#. It was part of a VGM Player written in C#. The author had ported the code to C# from another project. I thought I'd see if I could make use of it in this C# AGI interpreter. Seems to do the job well. When an AGI sound is played, the interpreter now uses the raw SOUND data to feed into the SN76489 emulation to produce the samples, then wraps it up with Wave file headers, and then plays it as a Wave file memory stream. I've tried playing the generated Wave file data with three different media playing libraries. One is the built in C# SoundPlayer, which works but isn't as flexible for checking whether the background sound has finished playing. I've also tried the NAudio and CSCore libraries. CSCore seems the nicest from what I've seen so far.

So its coming along. I'm now starting to play through a few games to pick up more bugs. Found a few already. I'll keep doing that for a few weeks I think.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on December 27, 2018, 06:19:48 AM
I've finally managed to get to the top of the beanstalk in KQ1, with the help of the saved game functionality. Even with it it took me about 15 minutes to get to the top.  :)

So this is now as far as I've got through playing KQ1 with the new C# interpreter. In the process of climbing the beanstalk, I exhausted all the free saved game slots and wrapped around to overwrite the first one again. Seems I have a bug in there somewhere, because it then falls over when I try to restore it. Should be easy to track down. I suspect that it is incorrectly keeping track of the number of saved games and doesn't account for the limit. Something like that. I'm going to fix that first then continue on with playing KQ1 through to the end. I'm about two thirds through the game I'd say.
Title: Re: C# AGI Interpreter
Post by: Collector on December 27, 2018, 09:12:29 AM
Nice to see progress.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on December 28, 2018, 11:19:33 AM
Seems I have a bug in there somewhere, because it then falls over when I try to restore it. Should be easy to track down. I suspect that it is incorrectly keeping track of the number of saved games and doesn't account for the limit. Something like that. I'm going to fix that first then continue on with playing KQ1 through to the end. I'm about two thirds through the game I'd say.

Turns out that there were two bugs in there. One bug exposed the other one. Both are fixed now, and saving and restoring games seems to be fully working.

And I just finished playing through to the end of KQ1!    :)

The strange thing is that I've noticed a few issues when playing my Ruby Cast  demo. I'll have to check those out. I'm surprised I didn't see similar issues in KQ1. They are the kind of issues, to do with sentence recognition, that I would have thought would have affected KQ1 as well. But maybe Ruby Cast is more broken than I thought. I'll have to try it in the original AGI interpreter and compare.

Title: Re: C# AGI Interpreter
Post by: lance.ewing on December 30, 2018, 04:27:14 PM
The strange thing is that I've noticed a few issues when playing my Ruby Cast  demo. I'll have to check those out. I'm surprised I didn't see similar issues in KQ1. They are the kind of issues, to do with sentence recognition, that I would have thought would have affected KQ1 as well. But maybe Ruby Cast is more broken than I thought. I'll have to try it in the original AGI interpreter and compare.

I have now fixed the parsing issues that I had noticed in The Ruby Cast. I can see why they didn't affect KQ1. I did find some rooms in SQ2 where the bugs could be seen though. But all fixed now.

I fixed a number of other bugs as well, including a long standing picture bug with the Space Quest 2 title screen. Turns out that the Visual AGI library had a bug in it where it wasn't executing "Fill" commands in some scenarios where it should have been.

The only bug I'm aware of that is currently outstanding is associated with sound playing. It seems that it holds down some notes for too long. It isn't obvious with most sounds, but there are a few where a note is held down for far too long. I'll start looking at that now.

And then after that I'll be adding AGI v3 support to Visual AGI's AGI Library. Should be straight forward.
Title: Re: C# AGI Interpreter
Post by: Collector on December 31, 2018, 03:27:33 AM
I'll be interested to see the changes to the library.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 01, 2019, 05:17:44 PM
The only bug I'm aware of that is currently outstanding is associated with sound playing. It seems that it holds down some notes for too long. It isn't obvious with most sounds, but there are a few where a note is held down for far too long. I'll start looking at that now.

And then after that I'll be adding AGI v3 support to Visual AGI's AGI Library. Should be straight forward.

The sound bug has been fixed now, and I have also added the ability to load AGI V3 games.

So far I have tried KQ4, Gold Rush, MH1, and MH2. Haven't tried the AGI V3 build of BC yet. - For KQ4, the whole opening story played through fine, with the music playing nicely in the background. Didn't spot any issues. But after the game itself started, ego's loop wasn't changing when I changed walking direction. It did, however, change loops properly and as expected when ego walked into water, which means the mechanism itself is fine, but for some reason the default ego View isn't switching loops. I will take a look at that over the next day or so.

In Gold Rush, I spotted a minor issue to do with the user input prompt. Might take a look at that one tonight.

In MH1 and MH2, both games fell over with an Exception. Haven't taken a look to see why yet. In the case of MH1, the opening story/credits played through fine, but in MH2, it crashes almost immediately. I haven't yet implemented the AGI V3 specific commands, so might be to do with that. I guess I'll find out soon.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 05, 2019, 07:10:41 AM
For KQ4, the whole opening story played through fine, with the music playing nicely in the background. Didn't spot any issues. But after the game itself started, ego's loop wasn't changing when I changed walking direction. It did, however, change loops properly and as expected when ego walked into water, which means the mechanism itself is fine, but for some reason the default ego View isn't switching loops. I will take a look at that over the next day or so.

I've fixed this one but it was an interesting problem. The main Rosella ego View has 5 loops. Not certain yet whether the extra one was in the original game or whether it inadvertently was added at some point when loaded into an editor. I'll try to rule that out soon.

I discovered another issue to do with non-default base priority values. KQ4 uses a base priority of 73. My priority band calculations were incorrectly using this. It has now been fixed.

I'm quite keen to try playing the whole of KQ4 through now and see where I get to.

In Gold Rush, I spotted a minor issue to do with the user input prompt. Might take a look at that one tonight.

Also fixed.

MH1 and MH2, both games fell over with an Exception. Haven't taken a look to see why yet. In the case of MH1, the opening story/credits played through fine, but in MH2, it crashes almost immediately. I haven't yet implemented the AGI V3 specific commands, so might be to do with that. I guess I'll find out soon.

There were several issues with the Manhunter games. Their logic scripts seem to do things that pretty much none of the other games do. This revealed a couple of areas where I wasn't quite doing things the same as in the original AGI interpreter, particularly around room changing and game clock updates. That is fixed now.

It is now possible to play both MH games. The main thing outstanding is the hold.key and release.key AGI v3 commands. Currently without them it means the round ball "ego" that you move around on the maps doesn't stop moving if you stop pressing the cursor key. Shouldn't take much to implement hold.key and release.key to resolve this.

Title: Re: C# AGI Interpreter
Post by: Collector on January 05, 2019, 10:05:13 AM
If you are talking about view.000 being the main view, then yes version 2 only has four loops. It has been some time since I have last looked at Visual AGI, but my memory is that it could not open any AGI3 resources. I am assuming that you added support for decoding these too, not just AGI3 logic support. I need to go through the project again to refamiliarize myself with it.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 06, 2019, 04:21:27 PM
Yes, it does say version 2, but the strange thing is that I've checked the timestamps of the vol files and they're all still 1988, which suggests they haven't been modified.

I have found another issue with KQ4. It only seems to affect KQ4. There are a few pictures that the AGI Library does not load, complaining that in the 0xF7 "relative draw" command that Y hits a value of 168, which it says is invalid. That sounds fair enough, but if I comment out that check, these pictures load fine and are drawn fine by my Interpreter. Not sure at this point whether this is somehow "bad" data that the original AGI Interpreter (and my Interpreter) is handling without issue, or whether there is something wrong in the AGI Library's Picture drawing code. I'm tempted to leave this validation check commented out since the Pictures are working fine like that.

Since my last update, I discovered an omission in my SOUND resource decoding that I have fixed. It wasn't catering for early termination of a voice with two consecutive 0xFF bytes.

And hold.key and release.key have been implemented, so the MH games are feeling like the real thing now.

I've been looking back over games that I thought I had working fine, and noticed that MUMG has a number of rendering issues. I recall discovering this back in early 2017. I thought I had fixed it, but perhaps I was partway through trying to resolve this when I was distracting by other things. It is to do with when animated objects are drawn. What MUMG does is to draw some animated objects to screen (e.g. speech bubble), and then immediately sets them to stop updating. The main Interpreter cycle doesn't redraw "stopped" animated objects; it only redraws those that are updating. So what MUMG does is to then position text on top of the "stopped" animated object, which it expects to stay on the screen. Currently my Interpreter is redrawing the stopped animated objects, which it shouldn't be in order to be compatible. So I'll need to take a look at that now. The way the original AGI Interpreter manages and redraws its "stopped" and "updating" objects though is like a ball of spaghetti, so its a matter of extracting the key parts in a cleaner design. I've got it all written out in a notebook, so hopefully I can get this solved soon.

Edit: Oh, forgot to mention that yes, I added AGIV3 support for loading of all resources. Mainly its just the LZW compression, and the Picture compression.
Title: Re: C# AGI Interpreter
Post by: lskovlun on January 06, 2019, 05:58:44 PM
The way the original AGI Interpreter manages and redraws its "stopped" and "updating" objects though is like a ball of spaghetti, so its a matter of extracting the key parts in a cleaner design.
Oh yes, oh yes. That was one of the major sticking points in SCI as well. I suspect much of it was brought over unchanged.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 10, 2019, 02:23:16 AM
If you are talking about view.000 being the main view, then yes version 2 only has four loops. It has been some time since I have last looked at Visual AGI, but my memory is that it could not open any AGI3 resources. I am assuming that you added support for decoding these too, not just AGI3 logic support. I need to go through the project again to refamiliarize myself with it.

There is definitely something weird going on with VIEW.0 in KQ4. My first workaround for this broke MUMG. Since the original AGI interpreter handles playing KQ4 fine, then it is coping with this extra Loop, even though the fragments of original AGI interpreter source indicate that AGI V2 would not have handled it. So I started wondering how scummvm handles this (since I've already ruled out the extra Loop having been introduced by an editor, due to timestamps on the KQ4 files still being 1988). Turns out that it has a specific workaround coded for KQ4:

https://github.com/scummvm/scummvm/blob/master/engines/agi/view.cpp

Code: [Select]
if (!(screenObj->flags & fFixLoop)) {
switch (screenObj->loopCount) {
case 2:
case 3:
loopNr = loopTable2[screenObj->direction];
break;
case 4:
loopNr = loopTable4[screenObj->direction];
break;
default:
// for KQ4
if (getVersion() == 0x3086 || getGameID() == GID_KQ4)
loopNr = loopTable4[screenObj->direction];
break;
}
}

My workaround was almost the same but without the AGI interpreter version and game ID check. This breaks MUMG though, which is why it seems that it is necessary to have a game specific check.
Title: Re: C# AGI Interpreter
Post by: Collector on January 10, 2019, 09:50:03 AM
ScummVM does use an abbreviated MD5 check (from the first 5000 bytes), usually on the logdir file for AGI games to determine not just the game, but the game version as well. I used this for my Sierra game version detector tool. Wouldn't such an approach cause a problem with using it on the fly to run a new game, especially one still under development where the target file for the check would be constantly changing.
Title: Re: C# AGI Interpreter
Post by: OmerMor on January 10, 2019, 12:03:17 PM
So I started wondering how scummvm handles this (since I've already ruled out the extra Loop having been introduced by an editor, due to timestamps on the KQ4 files still being 1988). Turns out that it has a specific workaround coded for KQ4:

https://github.com/scummvm/scummvm/blob/master/engines/agi/view.cpp


Turns out this fix goes all the way back to Sarien (which was imported to ScummVm), some 18 years ago:
https://github.com/cmatsuoka/sarien/commit/2e38cdc (https://github.com/cmatsuoka/sarien/commit/2e38cdc)
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 10, 2019, 04:03:20 PM
So I started wondering how scummvm handles this (since I've already ruled out the extra Loop having been introduced by an editor, due to timestamps on the KQ4 files still being 1988). Turns out that it has a specific workaround coded for KQ4:

https://github.com/scummvm/scummvm/blob/master/engines/agi/view.cpp


Turns out this fix goes all the way back to Sarien (which was imported to ScummVm), some 18 years ago:
https://github.com/cmatsuoka/sarien/commit/2e38cdc (https://github.com/cmatsuoka/sarien/commit/2e38cdc)

I suspected as much. It wouldn't have taken long to see the issue in KQ4, in fact its right there in the first room.  I like the commit comment.    :)
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 10, 2019, 04:12:20 PM
ScummVM does use an abbreviated MD5 check (from the first 5000 bytes), usually on the logdir file for AGI games to determine not just the game, but the game version as well. I used this for my Sierra game version detector tool. Wouldn't such an approach cause a problem with using it on the fly to run a new game, especially one still under development where the target file for the check would be constantly changing.

I've decided to go with simply the Game ID check for now, i.e. "KQ4" in this case. Its the same string they use when building saved game filenames. It seems that there is no need to check the specific AGI version because KQ4 was only released with the one version of Interpreter as far as I know.

Title: Re: C# AGI Interpreter
Post by: AGKorson on January 10, 2019, 06:32:34 PM
Which interpreter version does your KQ4 game use? If it's 3.002.102, that version included a new reserved flag, f20, which if set would allow views with more than 4 loops to automatically choose loops without problem. Could that be the problem you've encountered?
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 11, 2019, 02:25:00 AM
Yeah, that would definitely explain it. Let me check the version...  its 3.002.086 apparently, but it certainly behaves differently with regards to the automatic loop choosing than what AGI V2 does. Is it possible that f20 feature is there for all of AGI V3? I haven't checked yet whether f20 is set at this point in the version I have, but will do soon.

Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 11, 2019, 02:35:46 PM
I've checked this now, and flag 20 is not set. So I guess that agrees with the fact that only 3.002.102 and above use flag 20 for controlling this.

Version 3.002.086 must be doing something similar but not controlled by flag 20.
Title: Re: C# AGI Interpreter
Post by: Collector on January 11, 2019, 02:46:06 PM
For PC I am aware of only three versions of the game; 2.0, 2.2, and 2.3, but all use 3.002.086. The demo uses 3.002.102.

Do the Easter eggs in 2.0 work OK in your interpreter?
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 11, 2019, 03:39:10 PM
I assume they would. I've tried a couple of the easy to get to ones, and they work. I haven't played through much of KQ4 yet though.

I'm looking at my list of known issues at the moment. For some reason saved games don't work at all in Gold Rush. It completely ignores it when I select Save game in the menu. Really weird. I guess as soon as I start debugging that, all will become obvious.  That's probably the main one at the moment, but I haven't played through much of any game other than KQ1 so far.

Title: Re: C# AGI Interpreter
Post by: Collector on January 11, 2019, 04:34:57 PM
Gold Rush seems to be another one that has multiple versions, but all use the same interpreter, 3.002.149.
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 11, 2019, 05:15:24 PM
I've checked this now, and flag 20 is not set. So I guess that agrees with the fact that only 3.002.102 and above use flag 20 for controlling this.

Version 3.002.086 must be doing something similar but not controlled by flag 20.

If you have verified that it works correctly with the original DOS interpreter, I would love to get a copy of that interpreter file; I've decompiled all AGI versions that I have, including all v3 versions. I can confirm that only 3.002.102 and above use flag 20. I've examined the code in the 3.002.086 version I have, and there is nothing there that I can see that would change loops (without specific code in the logic), so I would love to know if the interpreter file I have is authentic (the date/time stamp is 8/31/1988 12:38PM). Would you mind sharing your file?

Also, I don't have access to my original disks (I'm on temporary assignment for work away from home for several months). Could I ask for a copy of the game files for your KQ4 version as well?  Thx!
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 11, 2019, 05:17:29 PM
Also, is your source code available for your C# interpreter? I'd be happy to take a look at any sections where you have questions/concerns; I might be able to find issues quickly based on my decompilation efforts.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 12, 2019, 05:23:21 AM
Also, is your source code available for your C# interpreter? I'd be happy to take a look at any sections where you have questions/concerns; I might be able to find issues quickly based on my decompilation efforts.

The project is currently a fork of Collector's AGI/SCI Developer repo in Bitbucket. That repo is described as a merger of SCI Developer and Visual AGI. It has a separate AGI Library that Visual AGI uses to load and save resources, although it didn't have anything to deal with SOUND or LOGIC resources when I started out with the Interpreter, and it didn't have anything for loading AGI V3 games either. The reason I forked Collector's repo is because one of the end goals is that it would be possible to run an AGI game from within the Visual AGI part of the IDE, so it is built on top of that AGI Library.

So having said all that, Collector's repo is currently private, and my fork of his repo is also private. If you create a Bitbucket account though, or already have one, then I can share my repo with you so that you can take a look. It would be very helpful actually. It would be good to get your feedback on what you see in there. I know there are still a number of outstanding issues. I was playing through SQ2 last night (nearly half way through I think) and spotted another four issues I need to take a look at. There are also several issues in KQ4.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 12, 2019, 11:26:40 AM
If you have verified that it works correctly with the original DOS interpreter, I would love to get a copy of that interpreter file; I've decompiled all AGI versions that I have, including all v3 versions. I can confirm that only 3.002.102 and above use flag 20. I've examined the code in the 3.002.086 version I have, and there is nothing there that I can see that would change loops (without specific code in the logic), so I would love to know if the interpreter file I have is authentic (the date/time stamp is 8/31/1988 12:38PM). Would you mind sharing your file?

The date/time stamp of my one is 7/25/1988 12:43PM, as it currently appears on my harddisk. The AGI file is 40866 bytes. I can't recall now where I found this one, but I've had it for a while I think. I can confirm that the game definitely handles the loop changing even though the VIEW has 5 loops.

Also, I don't have access to my original disks (I'm on temporary assignment for work away from home for several months). Could I ask for a copy of the game files for your KQ4 version as well?  Thx!

PM sent.
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 13, 2019, 03:00:34 AM
I've checked this now, and flag 20 is not set. So I guess that agrees with the fact that only 3.002.102 and above use flag 20 for controlling this.

Version 3.002.086 must be doing something similar but not controlled by flag 20.
I found it! It's a bug that's unique to version 3.002.086. Here's the code in v2.917 (which is identical in all versions up to 3.002.086:
Code: [Select]
   mov     [bp+newloop], 4                    ; set newloop value = 4(means no loop change)
   test    word ptr [si+flags], FIXED_LOOP    ; fixed loop?
   jnz     short setTheLoop                   ; if fixed, don't change loop
   cmp     byte ptr [si+loop_count], 2        ; loop count
   jz      short autoloop_sm                  ; if 2 loops, jump to small-loop selector
   cmp     byte ptr [si+loop_count], 3
   jnz     short autoloop_lg                  ; if three, also use small loop selector
                                              ; but if NOT three, go on to further check
autoloop_sm:
   mov     al, [si+obj_dir]                   ; use object direction
   sub     ah, ah
   mov     bx, ax                             ; to
   mov     al, loop_small[bx]                 ; convert direction to correct loop
   jmp     short changeLoopVal

autoloop_lg:
   cmp     byte ptr [si+loop_count], 4        ; if loop count != 4
   jnz     short setTheLoop                   ; don't change the loop
   mov     al, [si+obj_dir]                   ; if loop count == 4
   sub     ah, ah                             ; use object direction
   mov     bx, ax                             ; to
   mov     al, loop_large[bx]                 ; convert direction to correct loop
changeLoopVal:
   mov     [bp+newloop], al
setTheLoop:
...   ; change loop if newloop value has changed (!= 4)

In 3.002.0086, they made one very small change (marked with ***):
Code: [Select]
   mov     [bp+newloop], 4                    ; set newloop value = 4(means no loop change)
   test    word ptr [si+flags], FIXED_LOOP    ; fixed loop?
   jnz     short setTheLoop                   ; if fixed, don't change loop
   cmp     byte ptr [si+loop_count], 2        ; loop count
   jz      short autoloop_sm                  ; if 2 loops, jump to small-loop selector
   cmp     byte ptr [si+loop_count], 3
   jnz     short autoloop_lg                  ; if three, also use small loop selector
                                              ; but if NOT three, go on to further check
autoloop_sm:
   mov     al, [si+obj_dir]                   ; use object direction
   sub     ah, ah
   mov     bx, ax                             ; to
   mov     al, loop_small[bx]                 ; convert direction to correct loop
   jmp     short changeLoopVal

autoloop_lg:
   cmp     byte ptr [si+loop_count], 4        ; if loop count ***<*** 4
   ***jb***      short setTheLoop             ; don't change the loop
   mov     al, [si+obj_dir]                   ; if loop count ***>*** 4
   sub     ah, ah                             ; use object direction
   mov     bx, ax                             ; to
   mov     al, loop_large[bx]                 ; convert direction to correct loop
changeLoopVal:
   mov     [bp+newloop], al
setTheLoop:
...   ; change loop if newloop value has changed (!=4)

The change in that test from 'jnz' to 'jb' has the effect of always changing the loop no matter how many loops are in the view. In version 3.002.098 (not 3.002.102, as I previously thought), the check for flag f20 was added which fixed the problem.

The upshot of this is that for all versions up to 3.002.086, objects assigned a view with more than 4 loops will not update loop based on direction; in version 3.002.086 objects will update loop based on direction no matter how many loops there are; and in versions beyond 3.002.086 objects with more than 4 loops will only update loop if flag f20 is TRUE.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 13, 2019, 04:53:57 AM
Thanks for solving that mystery. That definitely explains it then. Looks like I will have to add specific version checks in there to make it truly compatible.
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 13, 2019, 12:28:50 PM
Thanks for solving that mystery. That definitely explains it then. Looks like I will have to add specific version checks in there to make it truly compatible.


The one thing I've learned after all this time studying AGI is that Sierra was terrible at error checking and that a lot of their code (in the interpreter) was a bit sloppy in handling non-standard data. When opening Sierra games you gotta check for a whole bunch of things that can cause problems.

For example, when I loaded KQ4 yesterday, I found there are four resources (picture 150, 151 and view 198, 199) that point to non-existent VOL files. And there are five pictures (2, 10, 15, 102, 115) that have bad data for pixels (using a Y value of 168 occurs a bunch of times in all those pictures). Sierra's interpreter will happily overflow the picture buffer without complaining (the stuff that gets overwritten is whatever is beyond the picture buffer in memory, which is not even allocated to the program!). So they display with no visible effects, but if you want to draw them in a new interpreter or editor, you have to account for that.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 13, 2019, 01:40:07 PM
For example, when I loaded KQ4 yesterday, I found there are four resources (picture 150, 151 and view 198, 199) that point to non-existent VOL files. And there are five pictures (2, 10, 15, 102, 115) that have bad data for pixels (using a Y value of 168 occurs a bunch of times in all those pictures). Sierra's interpreter will happily overflow the picture buffer without complaining (the stuff that gets overwritten is whatever is beyond the picture buffer in memory, which is not even allocated to the program!). So they display with no visible effects, but if you want to draw them in a new interpreter or editor, you have to account for that.

Yes, I've come across both of those issues in KQ4 already. For resources that point to non-existent VOL files, I think we can just ignore that. Clearly they're unused resources, and logic scripts must not be referring to them. Some of the other games have that issue as well.

For those pictures where the 0xF7 relative draw tool is drawing down to Y = 168, I'm not 100% sure what to do. If I remove the validation check, then they load fine and render fine in my Interpreter, but then that means we're allowing an original Sierra game to violate rules that our own editors enforce. Sierra's original PICTURE editor must have allowed them to draw down to 168, perhaps as a result of a bug maybe? Given this issue is in KQ4, maybe it was one of their later PICTURE editors. This issue doesn't seem to be in any other game that I've noticed so far.

I guess what I need to do is put some sort of option for disabling that validation into the Picture part of the AGI Library. Maybe original games can use the AGI Library with validation turned off, whereas the picture tool in Visual AGI would have to run with it turned on, although it probably doesn't need to, because if it has been written properly, it won't let such data be created.
Title: Re: C# AGI Interpreter
Post by: OmerMor on January 13, 2019, 04:09:45 PM
Sierra's original PICTURE editor must have allowed them to draw down to 168, perhaps as a result of a bug maybe? Given this issue is in KQ4, maybe it was one of their later PICTURE editors. This issue doesn't seem to be in any other game that I've noticed so far.

I have 2 versions of Sierra's PICTURE editor: PIC.EXE (1987-02-23) and PE.EXE (1987-05-28).
I did not check them out yet, but you are welcome to do so: I just released my stash of AGI tools in this thread: http://sciprogramming.com/community/index.php?topic=1814.0 (http://sciprogramming.com/community/index.php?topic=1814.0)
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 13, 2019, 05:03:35 PM
For those pictures where the 0xF7 relative draw tool is drawing down to Y = 168, I'm not 100% sure what to do. If I remove the validation check, then they load fine and render fine in my Interpreter, but then that means we're allowing an original Sierra game to violate rules that our own editors enforce. Sierra's original PICTURE editor must have allowed them to draw down to 168, perhaps as a result of a bug maybe? Given this issue is in KQ4, maybe it was one of their later PICTURE editors. This issue doesn't seem to be in any other game that I've noticed so far.
As I mentioned earlier, in the interpreter, the position is converted to a linear offset in the picture memory buffer (pos = 160*y+x). The interpreter doesn't bother checking either X or Y; so when the offset exceeds the actual buffer size, AGI just sets that byte in memory even thought it's a buffer overflow. But since the picture buffer is the very last hunk of memory that is allocated, it doesn't overwrite any AGI data. So it looks just fine in AGI.

In WinAGI, I just do a check every time a pixel is plotted. If the linear offset exceeds the buffer size, I just ignore it. That way the picture will be rendered the same as in AGI.
Title: Re: C# AGI Interpreter
Post by: AGKorson on January 13, 2019, 05:13:13 PM
Sierra's original PICTURE editor must have allowed them to draw down to 168, perhaps as a result of a bug maybe? Given this issue is in KQ4, maybe it was one of their later PICTURE editors. This issue doesn't seem to be in any other game that I've noticed so far.

I have 2 versions of Sierra's PICTURE editor: PIC.EXE (1987-02-23) and PE.EXE (1987-05-28).
I did not check them out yet, but you are welcome to do so: I just released my stash of AGI tools in this thread: http://sciprogramming.com/community/index.php?topic=1814.0 (http://sciprogramming.com/community/index.php?topic=1814.0)

Thx Omer. Maybe I will try to decompile these to see it there is a bug that allowed a Y value of 168 to slip in.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 14, 2019, 05:12:01 PM
I was playing through SQ2 last night (nearly half way through I think) and spotted another four issues I need to take a look at.

Actually there were five issues I had spotted in SQ2 at the nearly halfway through mark. All now fixed, so I'll continue playing through to the end now.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 19, 2019, 05:18:54 AM
Actually there were five issues I had spotted in SQ2 at the nearly halfway through mark. All now fixed, so I'll continue playing through to the end now.

I've finished playing SQ2 through to the end. I did notice a couple of minor issues that I need to take a look at. The text in one of the text windows was wrapped differently. I thought I had that working properly, but I'll need to take another look and see why this particular one isn't right. And there was one sentence that it didn't understand that the original appears to understand, which has made me realise I have't implemented ignored words yet. Not sure how I've got so far without that.

AGKorson is helping me with an issue in KQ4 to do with effective Y calculation of fixed priority objects that I will hopefully have resolved later today. Thanks Andrew for identifying the issue.

Assuming I fix the above issues over the next day, then the main outstanding issue I have at the moment is in MH2. For some reason it hangs on the WAIT icon after leaving the passage way that leads to Tad Timov's office to return outside. The scripts seem to be stuck in some kind of loop, which means whatever it is waiting for the Interpreter to do, it obviously isn't doing.

Next game I want to try playing through to the end is KQ4.
Title: Re: C# AGI Interpreter
Post by: Collector on January 19, 2019, 11:07:01 AM
Looks like it is the AGI3 games where you are finding issues.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 19, 2019, 12:32:55 PM
It's a mixture actually. All the early issues I found in KQ1, because that was the first game I was trying to play through. Then above I've recently mentioned seven issues I found while playing SQ2, which is AGI v2. But I have found a lot of issues in the AGI v3 games as well. There are a lot of AGI v2 games I haven't played through much of, so there could be other issues in there. I did spend quite a bit of time looking at DDP and MUMG because they do some unusual things, unusual when compared with the core set of AGI v2 games. There were quite a few issues I worked through with those two games.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 19, 2019, 12:41:15 PM
And there was one sentence that it didn't understand that the original appears to understand, which has made me realise I have't implemented ignored words yet. Not sure how I've got so far without that.

So turns out I have implemented ignored words already. The bit that isn't working is recognising the longest "word" when it is trying to match the input line with recognised word numbers. This recognition is breaking down when an ignored word is involved. So in the sentence "cut glass with cutter", it is ignoring "with" as expected, but is then matching "glass cutter" as a whole word, because that has a word number, which is probably a synonym of "cutter", and having ignored "with", it now thinks "glass" and "cutter" are next to each other. I can think of a quick way of fixing this already. Will try that now.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 19, 2019, 01:25:13 PM
Okay, that parsing issue is now fixed.

The text in one of the text windows was wrapped differently. I thought I had that working properly, but I'll need to take another look and see why this particular one isn't right.

And I've found the cause of this text window issue as well. When Roger searches the dead body of Vohaul, the message that is displayed has two spaces between two of the words (as stored in the LOGIC resource), but when it renders it to screen, it removes one of the spaces so that only one space is rendered between these two words.
Title: Re: C# AGI Interpreter
Post by: Collector on January 19, 2019, 09:08:01 PM
I did spend quite a bit of time looking at DDP and MUMG because they do some unusual things, unusual when compared with the core set of AGI v2 games.
The booter version of DDP? The "DOS" version floating around on the internet is actually a hacked version of the booter that crashes at points when you press certain keys like escape.
Title: Re: C# AGI Interpreter
Post by: lskovlun on January 20, 2019, 01:52:02 AM
The booter version of DDP? The "DOS" version floating around on the internet is actually a hacked version of the booter that crashes at points when you press certain keys like escape.
There is a DOS version according to NewRisingSun here (https://www.vogons.org/viewtopic.php?f=33&t=50096).
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 20, 2019, 04:50:14 AM
The booter version of DDP? The "DOS" version floating around on the internet is actually a hacked version of the booter that crashes at points when you press certain keys like escape.
There is a DOS version according to NewRisingSun here (https://www.vogons.org/viewtopic.php?f=33&t=50096).

No, not the original booter version, although I know the booter version is available on the Wayback Machine. According to that information from NewRisingSun, it appears that I have the rare DOS 1.50 version of DDP, since that is the version that comes up in the game. I can't remember now where I got it from, but the file timestamps indicate that I've had it for at least 8 years, which isn't long after I joined sciprogramming.com. I just tried downloading the DDP version on Al Lower's web site and that doesn't currently work with my Interpreter straight off. It doesn't like the OBJECT file, so I replaced it with the OBJECT file from the version I have. That allowed the game to start up properly, but after I viewed the game version within the game, it fell over with an error that I haven't investigated yet. So still a bit of work to do then for that version of DDP. NewRisingSun says that one is the Amiga data files running on a DOS Interpreter. Is there also a hacked version of the booter available? It wouldn't surprise me.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 20, 2019, 07:47:26 AM
I just tried downloading the DDP version on Al Lower's web site and that doesn't currently work with my Interpreter straight off. It doesn't like the OBJECT file, so I replaced it with the OBJECT file from the version I have. That allowed the game to start up properly, but after I viewed the game version within the game, it fell over with an error that I haven't investigated yet.

The AGI Interpreter used by the version on Al's site is 2.272. I think my Interpreter is not compatible with early versions of the original Interpreter at this stage, so that is probably a good test case. I know that various things changed between the versions that I am not yet accounting for.
Title: Re: C# AGI Interpreter
Post by: Collector on January 20, 2019, 09:56:50 AM
Perhaps my memory is wrong and I was thinking of the hacked Amiga version, not booter. I would be interested in seeing the DOS 1.50 version.

This does bring up a question of how hard would it be to add support for AGI games from other platforms? Then there are the fan games that have memory or stack issues that will run in NAGI, but not the original interpreters.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 22, 2019, 02:17:56 AM
I doubt that I would extend the Interpreter to support other platforms. At the moment I see it as covering all of the DOS AGI V2 and V3 games. Not sure if I'll extend it to AGI V1. There seems little point. One of the main goals was to integrate it with Visual AGI, i.e. run a game created within Visual AGI. That requirement doesn't need to support many different versions of platforms I don't think.

Regarding fan made games and memory issues, in theory my Interpreter should be able to play all of the fan made games without any such issues, because it doesn't have the same memory limitations. So it is a bit like NAGI in that regard. It would certainly allow you to run a game that would not work in the original Interpreter. Not sure if that is a good or bad thing. It would probably be quite difficult to exactly match the original AGI Interpreter's memory management and restrictions.
Title: Re: C# AGI Interpreter
Post by: Collector on January 22, 2019, 09:41:42 AM
You have a point about AGI1 games. I am not sure if I even have any outside of what may be on some of my booter disks. I do have a copy of the 2nd IBM release of the original KQ, but wouldn't that be GAL, more than purely AGI? I wish I had a means of extracting the files from my booter disks.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 22, 2019, 02:36:40 PM
AGKorson is helping me with an issue in KQ4 to do with effective Y calculation of fixed priority objects that I will hopefully have resolved later today. Thanks Andrew for identifying the issue.
...
Next game I want to try playing through to the end is KQ4.

Andrew's suggestion on how to fix the effective Y issue did indeed fix the problem I was having with one of the dwarves in the mine.

I've started playing through KQ4 and I'm now up to about a third of the way through I'd say. I discovered another bug in the room with the pool and cupid. It wouldn't understand it when I said "get bow". Turns out that I hadn't properly implemented the "AND" part of "if" statements, in that I hadn't put in the shortcut break out when it encounters the first condition that is false. So what it was doing was evaluating everything in the AND. Normally that wouldn't have mattered, but some test conditions have side effects, "said" being the one in this case. There were multiple "get bow" said tests in the room script, and it believed it had already matched the input with an earlier test, but actually it should never have tested that particular "said". All I was missing was a "break" in one of my for loops. That fixed the issue and I was able to progress.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 25, 2019, 04:55:31 PM
I just noticed the attached priority screen from PICTURE 57 in KQ4AGI (the witches cave).
Title: Re: C# AGI Interpreter
Post by: lskovlun on January 25, 2019, 07:44:55 PM
Hmmm, how does that even work?
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 26, 2019, 04:35:41 AM
Not sure if this is what you mean, but since all of those blue control lines are drawn above the black control line, then there is no way that they can affect ego or the witch that chases her. Their base line would stop at the black control line and not go any higher. So one of the game developers (John Hamilton I assume) must have thought it would be fun to put some control line graffiti in that area of the priority screen. I've seen other not so obvious examples of this in other rooms. For example, in room 110, I can see CHRIS written in black control lines within one area, which would be Chris Hoyt. There is something else written in another area on that priority screen but I can't make it out. - Also in room 26, one of them has written "HI" in one area. - I wonder how many other Sierra games have examples of this.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on January 26, 2019, 04:46:10 AM
I've started playing through KQ4 and I'm now up to about a third of the way through I'd say.

I have now finished playing through to the end of KQ4. So that I could get through as quickly as possible, I've been following video walkthroughs. In this particular case, it was a walkthrough that got the maximum or 230 points.

I did find another blocking bug along the way. Actually it was blocking in more ways than one. It blocked my progress getting further in the game, and it was to do with the rendering of a blocking control line. I had a bug in my add.to.pic implementation to do with the rendering of the control box that prevented the ghost boy from climbing the ladder. So I fixed that then was able to play through to the end.

There is another bug I noticed, I think to do with the follow.ego implementation. Pretty much any character that tries to chase ego just seems to randomly move, more like a wander, but not even that. Every step seems to be randomly placed, like its doing some kind of weird dance. I hadn't noticed any follow.ego issues in other games so far. I plan to go back and confirm that its working in earlier games, and if so, then take a closer look at the KQ4 LOGIC scripts for the rooms where this happens to see what is going on.

p.s. I just tried out two more of the KQ4AGI easter eggs, the "beam me up" one and the "rap kq" one.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on February 10, 2019, 12:41:54 PM
There is another bug I noticed, I think to do with the follow.ego implementation. Pretty much any character that tries to chase ego just seems to randomly move, more like a wander, but not even that. Every step seems to be randomly placed, like its doing some kind of weird dance. I hadn't noticed any follow.ego issues in other games so far. I plan to go back and confirm that its working in earlier games, and if so, then take a closer look at the KQ4 LOGIC scripts for the rooms where this happens to see what is going on.

This bug is now fixed. It turns out that my Interpreter was updating the stopped flag too often. It should only update it when the step time counter hits zero. It was correctly updating the animated object's position on the counter hitting zero, but for updates to the stopped flag, it was doing this on all values of the step time counter. The result was that it thought the animated object was stopped (i.e. hadn't moved) in the current cycle (which is true, but is of no relevance if it hasn't tried to update the animated object's position in that cycle). This lead to the seemingly random movement, because the follow ego logic chooses a random direction if the animated object is stopped by something. So my Interpreter was mistakenly thinking it had been stopped when it hadn't and therefore making it select a random direction to avoid whatever had supposedly stopped it.

I also fixed two other bugs I found in KQ4: One of these was in any room with a zombie. The whole game slowed down to a complete halt for about 10 seconds whenever the first zombie came out of the ground. Turns out this was due to the Logic script trying to start the same sound 20 times in a row. I assume that in the original AGI interpreter, this didn't have any impact on performance, e.g. perhaps it just immediately started playing the sound again from the start with no noticeable effect, but in my Interpreter, it generates the WAVE file data on the fly before playing a sound and doing this 20 times in quick succession was slowly things down to a crawl. I considered caching generated sounds, which perhaps I might still do for the current room. For now I've got a check in there that if it tries to play a sound that is already playing, it just leaves it playing and does nothing. This fixes the issue in the zombie rooms.

The other issue was in the KQ4 Help screen. It was being displayed with a black background where the original game has a white background. This was being caused by a difference in how the background colour is used when switching to the text screen. The original Interpreter sets the text attribute value using the currently set background and foreground, and then uses the background part of the text attribute when clearing the screen. I changed my Interpreter to do the same and now this issue is also resolved.

There is only one other issue I know of now that affects KQ4, and also SQ2, in fact it is possible that it affects all games but so far I have only observed it in those two games. For some reason the "Save game" option in the menu stops responding after a while. All other menu options continue to work. I had a similar issue a while back that affected a number of menu options, but this new issue seems to affect only the "Save game" option. That will be the next issue I try to track down.
Title: Re: C# AGI Interpreter
Post by: lance.ewing on February 10, 2019, 05:13:05 PM
There is only one other issue I know of now that affects KQ4, and also SQ2, in fact it is possible that it affects all games but so far I have only observed it in those two games. For some reason the "Save game" option in the menu stops responding after a while. All other menu options continue to work. I had a similar issue a while back that affected a number of menu options, but this new issue seems to affect only the "Save game" option. That will be the next issue I try to track down.

And now this issue has also been resolved. The issue was that the Interpreter wasn't re-enabling all menu items when restoring or restarting after death. I also noticed that I wasn't setting the RESTART flag on a restart, which meant that the credits and story weren't being skipped for games that use that flag.