AGI extended character support

[ZvikaZ]

This topic was mentioned at https://sciprogramming.com/community/index.php?topic=2051.msg15257#msg15257, but it's off topic for that thread, and it already got many interesting topics, so I'm replying at a new thread...

AGKorson discussed what's possible with the original Sierra AGI interpreter, running on Dos/Dosbox.

However, with ScummVM we have more flexibility. Actually, I have translated PQ1 to Hebrew - https://sciprogramming.com/community/index.php?topic=2001.0.

For instructions, please see https://github.com/adventurebrew/HebrewAdventure/tree/master/tools/agi.
I admit that these instructions were phrased mainly as a reference for myself in the future, but I will gladly help if you want.

Also, maybe some minor modification need to be done to ScummVM to support AGI French, but I can do this as well.

[AGKorson]

Good observation ZvikaZ. ScummVM might be the best way to go.

The original NAGI (version 2.06 for windows/2.07 for Linux IIRC) does not support extended characters.  If a message has an extended character, NAGI prints garbled data.

Nick made the source code for NAGI publicly available, so back in 2006, I ported it to VC7, and modified it to support extended characters and changing color palettes. It was actually pretty easy to get support for extended characters (changing the font file loader to load 256 characters instead of 128) and then modifying the font files (*.nbf) that are included with the executable. I'm attaching NAGI v2.09 here if anyone is interested. (I also tried to recompile it in VS 2019, but I can't get it to work; if anyone wants to try that, I'd be happy to share my source to see if you can figure it out.)

If you don't like the glyphs that I include in NAGI v2.09, you can edit them. I created an app called NFEdit that you can use to easily modify the font files. I explain it in another thread (https://sciprogramming.com/community/index.php?topic=1766.msg11533#msg11533) if you are interested.

[Collector]

One thing that always annoyed me about NAGI was that the window close button did not work. You didn't perhaps fix that, did you?

[AGKorson]

One thing that always annoyed me about NAGI was that the window close button did not work. You didn't perhaps fix that, did you?

LOL, no I didn't. But I wish I had. I was doing some checks last night to make sure it does what it's supposed to, and I know I silently cursed that 'feature' at least ten times...

[doomlazer]

For instructions, please see https://github.com/adventurebrew/HebrewAdventure/tree/master/tools/agi.

This information was extremely helpful. Thank you.

Edit: Need to do more testing before I ask questions - some of my assumptions were incorrect.

[doomlazer]

AGKorson, your string hack for extended character support in DOSBox from this post is awesome. Still in the testing phase, but this will surely be used in Threepwang's SQ1 French Translation.

This, along with the ScummVM font information from ZvikaZ's Hebrew translation, have been invaluable. Thank you both! I'm truly standing on the shoulders of giants.

[AGKorson]

Glad to hear you found it helpful!

Since I posted that, I've continued working on the code for that hack, and can now add extended character support (and a ton of other features) to any MSDOS version of AGI by just running a special logic at the start of the game.  I plan on releasing a full set of logic scripts (I'm calling it the AGI Power Pack) very soon that anyone can just drop into a game to open up a whole new world of capabilities for AGI games.

In the meantime, if you need any help with the extended character script, feel free to PM me and I'd be happy to help.

[doomlazer]

For the SQ1 French translation we've been struggling with the extended characters in DOSBox. I've come up with a hack to create and use a custom codepage which allows us to modify the extended characters, but because AGI handles standard and extended chars differently, it's still using a mix of system glyphs and our custom codepage in-game.

The details of what we are trying to accomplish and where we are currently at can be found in this Vogons thread: https://www.vogons.org/viewtopic.php?p=1061173#p1061173

Looking at AGK's post here. I'm wondering if it would be possible to trick the AGI interpreter to use our custom codepage for the "7 bit ASCII" pointer INT 43h?

If it's possible to get AGI to reference all glyphs from the custom codepage I believe it would allow AGI fan games to use a completely custom 8x8 font in DOSBox.

[AGKorson]

The reason your characters aren't showing correctly when printing black-on-white is because of the approach Sierra used to deal with how EGA handles character drawing when in graphics mode - it's not the same way characters are drawn in text mode.

In text mode, the character attributes allow you to independently set background and foreground colors. When a character is drawn to the screen the EGA card will replace pixels at the character location based on the glyph bitmap.

But in graphics mode, the 'background' attribute doesn't work the same. Basically, only the high bit of the 'background color' value matters. If it's clear, you get a black background, and the character draws the same as on the text screen. If that bit is set, the character is XORed onto the screen with the existing pixels. So, in a nutshell, you can't (easily) draw black characters on a white background in graphics mode. (This applies to ALL graphics modes on the EGA card, and it's a 'feature' of the EGA card, and not an MSDOS thing.)

Sierra decided to solve the problem by inverting the glyph bitmap, and then drawing it as white-on-black; since the glyph is inverted, it gets drawn as if it were a 'black-on-white' character. To invert the glyph, they copy the glyph bitmap from the standard character interrupt location. And since this is hard-coded into the AGI character drawing function, extended characters don't work.

The hack that I wrote to address this checks the value of the character, and if it's extended, it grabs the glyph from the extended character interrupt instead of the standard interrupt.

I don't know much about how MSDOS handles fonts and code pages. I think it's updating the interrupt values to point to the correct glyphs (which is why the extended characters are displaying correctly). But since AGI is hard coded to the location of the default glyphs, and not the current glyphs, it's not working for standard characters.

I think this might be an easy fix. The hack just needs to modify the location used for standard character glyphs from the default (F000:FA6E) to whatever value is actually located at INT 43h. But I'd have to do some testing.

[doomlazer]

For instructions, please see https://github.com/adventurebrew/HebrewAdventure/tree/master/tools/agi.
I admit that these instructions were phrased mainly as a reference for myself in the future, but I will gladly help if you want.

Also, maybe some minor modification need to be done to ScummVM to support AGI French, but I can do this as well.

ZvikaZ, I'm having a bit of trouble matching player commands containing extended characters in ScummVM. I've compiled a test version of SVM that allows FR_FRA language games to input chars >= 0x80 and uses loadExtendedDictionary().

I tried creating the WORDS.TOK.EXTENDED with your words_import.py script and, while it seems to output a complete word list, I'm getting an assertion error, which might be the problem.

I'm not sure how to covert the Hebrew characters to their French equivalent in the following section of the python script:

Code: [Select]

if not word.isascii():
            assert 'א' <= word[0] <= 'ת'
            for prefix in ['ה', 'ב']:

The other place I think the issue might be is in keyboard.cpp. I see both the RU_RUS and HE_ISR convert for their codepage. I can remap the extended characters to the French codepage 858 and the letters appear correctly in-game, but still don't match commands. Also, there doesn't seem to be a consistent offset, as the French 858 ext chars are in a different order compared to 435. I'm probably missing something critical about how to properly do the codepage conversions.

Any advice you can provide would be appreciated. This appears to be the last big technical hurdle to get French AGI translations 100% working in both DOSBox and ScummVM.

[ZvikaZ]

When I wrote it, I assumed that this script will be used only for Hebrew.
Therefore that assertion.
You can locally delete from line 40 to line 50.
All that block tries to solve a problem that I think is irrelevant for French.

Later, I should add this behavior as optional, depending on user's argument, but don't wait for me...

Let's see if that improves the situation.

Also, regarding the codepage conersion, well, that should be addressed. I'm not sure what codepage 435 and 858 are.
However, note that you should decide what codepage you want to use. Use whatever is easier for you.
As you can see from the comment in keyboard.cpp, I decided to use win1255 for Hebrew. As its name implies, it didn't even exist at the Dos era, it was invented later.

Anyway, the conersion there is from Unicode to the desired codepage. I'd recommend adding breakpoints and verify the behavior there with the debugger.

[doomlazer]

Thank you. I was able to get things working by hex editing WORDS.TOK.EXTENDED. Now that I understand the problem, I should be able to adjust words_import.py to export what I need!