Community

SCI Programming => SCI Development Tools => Topic started by: ZvikaZ on May 04, 2020, 11:34:41 AM

Title: SCI Editors - beside SCICompanion/SCIStudio?
Post by: ZvikaZ on May 04, 2020, 11:34:41 AM
Hi.

What tools can be used to edit SCI games, besides SCICompanion and SCIStudio?

I'm trying to change `vocab.000` of SQ3 to support Hebrew, but both tools refuse to save it with the Hebrew inside.
(but it *is* possible to change messages texts to Hebrew)

(Maybe it's impossible by design of SCI??  :'( )
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 04, 2020, 01:55:14 PM
Depends on the interpreter. SCI0 vocabs use a 7-bit character set, SCI01 supports an 8-bit set.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 04, 2020, 02:31:12 PM
SCI0 vocabs use a 7-bit character set
Which, as I said on the GitHub issue, has to do with the way the dictionary compression works.

As I also said there, you have the whole "Hebrew is the other way around" thing to consider.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: OmerMor on May 04, 2020, 02:59:57 PM
As I also said there, you have the whole "Hebrew is the other way around" thing to consider.

I thought English is the other way around?  ::)
מה, לא ככה?
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 04, 2020, 03:09:47 PM
I thought English is the other way around?  ::)
מה, לא ככה?
Other way around compared to the languages SCI games were released in, smartass X3
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 04, 2020, 03:28:54 PM
As I also said there, you have the whole "Hebrew is the other way around" thing to consider.

I thought English is the other way around?  ::)
מה, לא ככה?
Let me just add a third take on it here: SCI needs bidi support :o
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: ZvikaZ on May 04, 2020, 03:53:33 PM
Well, it has BiDI support, but currently only in my local Git repo ;-)
See https://github.com/scummvm/scummvm/pull/2215 (and small demonstration at https://cdn.discordapp.com/attachments/669609616794976287/705710346870194266/unknown.png)

Regarding SQ3 - see attachments.
I haven't bothered translating the messages, it's the easiest part.
But I have succeeded with BiDi input from right to left!

The only obstacle is this vocab thing. There must be a way to circumstance this.
As you can see in the pictures - my Hebrew verb *was* recognized. It only failed with the second word.

I did it with taking vocab.900 from QfG2, exporting it as patch, and saving it as vocab.000 in SQ3.
But it's still has problems, and it's quite cumbersome.

Any ideas how to overcome the vocab problem are welcomed.

EDIT:
===
I've seen http://sciprogramming.com/community/index.php?topic=1781.30 , but didn't really understand how did he overcome the problem.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: EricOakford on May 04, 2020, 04:25:26 PM
To quote the ScummVM wiki (https://wiki.scummvm.org/index.php?title=Sierra_Game_Versions#SCI_Games):

Quote
There are no real SCI0 translations. This is because this interpreter does not cover special characters like accent grave, accent egu, umlaut, etc. The SCI0 games that are translated are converted to the hybrid SCI01.

Strangely, I did notice references to foreign language support in the newest version of SQ3 (1.018), as well as the SCI0 version of Mixed-Up Mother Goose, and the Iceman and HQ1 demos. They all use interpreter version 0.000.685.

On the subject of using vocab.900 from QFG2: that's an SCI01 game, while SQ3 is SCI0. So compatibility issues would be expected in that case.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 04, 2020, 04:44:15 PM
International versions of SQ3 were in SCI01.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: MusicallyInspired on May 04, 2020, 07:21:02 PM
I've heard people call QFG2 an SCI1 game. Is there any truth to that?
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 04, 2020, 08:28:04 PM
ScummVM's wiki has "SCI1 EGA", which lists only various QfG2 releases, and "SCI1 Early, Middle, and Late", all in both EGA and VGA.

So it boils down to your definition of SCI1. QfG2 has a text parser instead of an icon bar, but also one or two bitmap screens and improved memory usage.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: ZvikaZ on May 05, 2020, 05:50:42 AM
To quote the ScummVM wiki (https://wiki.scummvm.org/index.php?title=Sierra_Game_Versions#SCI_Games):

Quote
There are no real SCI0 translations. This is because this interpreter does not cover special characters like accent grave, accent egu, umlaut, etc. The SCI0 games that are translated are converted to the hybrid SCI01.

Well, they say that SCI0 translation don't exist, for Sierra's original interpreter; but it doesn't mean that it's not possible to create SCI0 translation, with modified interpreter (i.e. ScummVM with little changes).

On the subject of using vocab.900 from QFG2: that's an SCI01 game, while SQ3 is SCI0. So compatibility issues would be expected in that case.

Why is so?
Looking in ScummVM code (https://github.com/scummvm/scummvm/blob/master/engines/sci/parser/vocabulary.cpp) the only difference between SCI0 and SCI01 is in the loading from file phase (lines 143-164) - the difference between using bit 8 to signal string end (SCI0) and using regular C-style '\0' char to signal string end (SCI01).
Once the internal words dictionary has been populated, there is no further difference.

Indeed, the problem with QfG2 is that the words are belonging to different groups, and it's a lot of work to change them to match SQ3 groups.
(I have also checked Sierra's German SQ3, but its vocab groups look more like QFG2's, so, no help here)

Therefore, I think that it makes perfect sense to just convert original SQ3 vocab to SCI01 format, and use it with ScummVM.
I can write such a tool, but if something already exists - it will save me re-inventing the wheel...

If there isn't such a tool, I need a little help here ;-)
SCI0 vocab is documented at http://sciwiki.sierrahelp.com//index.php?title=SCI_Specifications:_Chapter_6_-_SCI_in_action#Vocabulary_file_formats
I can see the difference between it and SCI01 at ScummVM code.
But ScummVM ignores the initial pointer section.
Is there some documentation for that? (I want to make sure that my vocab will be loadable by SCICompanion/SCIStudio)

EDIT
===
It's working  :)
Pasting here the Python code, it might help someone in the future:

Code: [Select]
# Sierra's SCI vocab format has "old" version (used by vocab.0) - with 7 bits ascii
# and 8th bit used for string end
# while "new" version (used by vocab.900) has 8 bits ascii
#
# for Hebrew translation, we need the vocab to be in the newer version

import pathlib

INPUT_FILE = r"C:\Zvika\Games\sq3ega.hebrew\sq3ega\vocab.000.orig"
OUTPUT_FILE1 = r"D:\ZVIKA\Sierra\Quest for Glory 2\VOCAB.900"
OUTPUT_FILE2 = r"C:\Zvika\Games\sq3ega.hebrew\sq3ega\VOCAB.000"

in_vocab = list(pathlib.Path(INPUT_FILE).read_bytes())
out_vocab = in_vocab[0:2]   # vocab signature

# vocab.900 starts with 255 16-bit pointers
# they aren't interesting...
out_vocab.extend([0] * (256*2))

bytes_until_word_text = 0
for idx, val in enumerate(in_vocab[(26*2):]):
    if bytes_until_word_text == 0:
        if val < 0x80:
            out_vocab.append(val)
        else:
            out_vocab.append(val - 0x80)
            out_vocab.append(0)
            bytes_until_word_text = 3
    else:
        out_vocab.append(val)
        bytes_until_word_text -= 1

with open(OUTPUT_FILE1, "wb") as out_file:
    out_file.write(bytes(out_vocab))

with open(OUTPUT_FILE2, "wb") as out_file:
    out_file.write(bytes(out_vocab))

Just to be detailed, you need to modify that vocab.900 in QFG2 dir with SCICompanion, export patch, rename it to vocab.000, and save in SQ3 dir.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 05, 2020, 08:32:37 PM
So you gave a 01-format dictionary to a 0 interpreter, and it doesn't completely lose its way over it?

Impressive if true.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Collector on May 05, 2020, 09:35:21 PM
I've heard people call QFG2 an SCI1 game. Is there any truth to that?

Here is what Cory Cole told me:

Quote
Quest for Glory II definitely did *not* use the SCI1 interpreter that Sierra used for King's Quest V and the other 256-color EGA games. That interpreter did not have a parser, so could not handle typed input at all. If they internally called QfG2's interpreter 'SCI1', it was just a PR thing - It was not the same interpreter.

As to *why* QfG2 used the parser and EGA graphics, it's because all of Sierra's other games were running late. Sierra needed a major release for Christmas 1990, and we were the sacrificial lamb. (I think it was partially because I did not fight as hard as Mark Crowe - The choice was between QfG2 and SQ3 as to which game would ship early, but with 16-color graphics.)
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 06, 2020, 02:43:31 AM
Here is what Cory Cole told me:
But he's wrong, isn't he? The interp grew to 1.5 times the size, and among other things the memory management was overhauled. You couldn't even start QfG2 under SCI0; it would run out of heap memory before getting anywhere (this happened in FreeSCI before the VM rewrite). Polygons were used in QfG2 - they don't exist in SCI0. And, most tellingly, the interp version number: 1.000.072.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: ZvikaZ on May 06, 2020, 04:19:21 AM
So you gave a 01-format dictionary to a 0 interpreter, and it doesn't completely lose its way over it?

Impressive if true.

I didn't try giving it to the *original* Sierra's SCI0 interpreter. I assume it will indeed crash, or go crazy...
I gave it to ScummVM, which already had code to support that mixed situation, because of Japanese PQ2 (it only issues a warning).

BTW,
I wander how did they made Japanese PQ2, as increasing the characters coding from 7 bits to 8 bits gives only 256 values (actually, less than that...)
Is it enough for Japanese?
Or did they made further hacks to use 16 bit encoding?
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: ZvikaZ on May 06, 2020, 04:21:10 AM
Here is what Cory Cole told me:
But he's wrong, isn't he? The interp grew to 1.5 times the size, and among other things the memory management was overhauled. You couldn't even start QfG2 under SCI0; it would run out of heap memory before getting anywhere (this happened in FreeSCI before the VM rewrite). Polygons were used in QfG2 - they don't exist in SCI0. And, most tellingly, the interp version number: 1.000.072.

I think he also confused SQ3 with SQ4, as SQ3 was released in 1989, before QFG2.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 06, 2020, 04:43:06 AM
I wander how did they made Japanese PQ2, as increasing the characters coding from 7 bits to 8 bits gives only 256 values (actually, less than that...)
Is it enough for Japanese?
Or did they made further hacks to use 16 bit encoding?
There were national Japanese standards for the home computers in that market. All they had to do was implement that.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 06, 2020, 07:57:36 AM
Hmmhmm. https://en.wikipedia.org/wiki/Shift_JIS (https://en.wikipedia.org/wiki/Shift_JIS) for example is a particularly likely method to encode Japanese. Characters are either one or two bytes.

SQ4 Japanese didn't do anything like that though, it just did its own 8-bit thing. And my SCI11+ has UTF-8.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: MusicallyInspired on May 06, 2020, 09:41:44 PM
That note about Crowe fighting for SQ4 to be done in VGA first is interesting because Scott said that they were given a choice to use parser or not and they said parser. But then they forced them to do point & click, to which Scott said he and Mark weren't happy about. I wonder if the choice they were ACTUALLY given were "EGA with a parser" and "VGA without a parser."
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Charles on May 06, 2020, 10:46:52 PM
I suspect they started making SQ4 in VGA with a parser. We know Sierra had a VGA Interpreter with a parser at one point, because the 1990 VGA Xmas demo has it ( http://sciprogramming.com/community/index.php?topic=27.0 (http://sciprogramming.com/community/index.php?topic=27.0)).

And SQ4 was first released in March 1991. Seems reasonable they started development on the same parser the Xmas demo used 3 months earlier.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 07, 2020, 05:48:21 AM
I'd say that the choice to compile the parser in or out is largely a business decision. It's not like the choice affects the architecture of the rest of SCI much.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: MusicallyInspired on May 07, 2020, 11:41:09 AM
No, that's what I'm saying. Maybe Ken/management decided they just didn't want a parser in VGA. And that that was the real decision they were giving Scott and Mark.
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: Kawa on May 07, 2020, 01:18:00 PM
I'd say that the choice to compile the parser in or out is largely a business decision. It's not like the choice affects the architecture of the rest of SCI much.
Besides the existence or inexistence of Said spec blocks in script resources. Though I think that was only in SCI11...
Title: Re: SCI Editors - beside SCICompanion/SCIStudio?
Post by: lskovlun on May 07, 2020, 03:30:50 PM
I'd say that the choice to compile the parser in or out is largely a business decision. It's not like the choice affects the architecture of the rest of SCI much.
Besides the existence or inexistence of Said spec blocks in script resources. Though I think that was only in SCI11...
True enough. I was thinking of SCI1 and the immediate effect of that decision.