Community

General and Everything Else => The Games and other Sierra Adventure stuff => Topic started by: lance.ewing on December 27, 2016, 12:20:33 PM

Title: Mega Tokyo AGI forums backup
Post by: lance.ewing on December 27, 2016, 12:20:33 PM
That is old enough for the old MegaTokyo forums. If only they were still accessible.

Stuart George has a backup from 2004 that he says he'll check the contents of before handing over. Apparently it is a dump of the whole DB.

Stuart George has sent through a backup of the old Meka Tokyo forums data, i.e. the backup referred to in the quote above. I haven't yet had a chance to look at the data, but I thought I'd get the discussion started about what we think would be the best way to make this available again on the Internet in read only form. It would be ideal if it could be included as part of this sciprogramming site, but I'm not familiar with hosting forums and forum software, so not sure what would be involved and whether that would technically work out or not. Does sciprogramming.com use the same forum software?

I'll take a look at the file now and see what the names of the DB tables are.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 27, 2016, 12:47:09 PM
According to the Wayback Machine, the mega-tokyo forums were hosted on YaBB (Yet Another Bulletin Board system), which a quick search tells me is a Perl based BB system. That doesn't sound right though because the URLs have php in them.

Edit: Aha! It's not YaBB but YaBB SE.

Edit 2: And YaBB SE went on to become SMF, which is what sciprogramming.com is using, right?

Edit 3: This is list of the tables for those who are familiar with forum software:

tblbanned
tblboards
tblcalendar
tblcalendar_holiday
tblcategories
tblcensor
tblinstant_messages
tbllog_activity
tbllog_banned
tbllog_boards
tbllog_clicks
tbllog_errors
tbllog_floodcontrol
tbllog_karma
tbllog_mark_read
tbllog_mark_read_copy
tbllog_online
tbllog_topics
tblmembergroups
tblmembers
tblmessages
tblpolls
tblreserved_names
tblsettings
tbltopics
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 27, 2016, 02:05:15 PM
This is SMF, yes. I wonder if there's any chance of finding my old crap? I never did find my old QfG and MMBN battle systems...
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 27, 2016, 02:21:13 PM
Quote
+--------+----------+-------------------------------+-------------------------------------------------------------------------------------------------+--------------------------------+------------+-----------+----------+----------------+---------------------+-------+
| ID_CAT | ID_BOARD | name                          | description                                                                                     | moderators                     | boardOrder | numTopics | numPosts | isAnnouncement | notifyAnnouncements | count |
+--------+----------+-------------------------------+-------------------------------------------------------------------------------------------------+--------------------------------+------------+-----------+----------+----------------+---------------------+-------+
|      3 |        4 | AGI Games, Fan and Commercial | Stuck in a Sierra game? Writing a fan based game? Just want to talk about AGI? Discuss it here  | df,sonneveld,AGI1122           |          1 |       966 |     8848 |              0 |                   0 |     0 |
|      3 |        5 | SCI Games, Fan and Commercial | Got questions about Sierra SCI Games? or how about creating your own SCI games with SCI Studio? | sonneveld,Brian_Provinciano,df |          2 |       737 |     4977 |              0 |                   0 |     0 |
+--------+----------+-------------------------------+-------------------------------------------------------------------------------------------------+--------------------------------+------------+-----------+----------+----------------+---------------------+-------+

Under the AGI board, there are 966 topics and 8848 posts. Under SCI there are 737 topics and 4977 posts. The sciprogramming AGI board pales in insignificance when compared with these numbers.
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 27, 2016, 02:26:57 PM
All things considered, that does not surprise me.
Title: Re: Mega Toyko AGI forums backup
Post by: MusicallyInspired on December 27, 2016, 05:27:11 PM
Could it be integrated into these forums in locked/read only subcategory? That's best case scenario, I'd say. Otherwise, maybe a separate link from the SCI Programming main site.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 27, 2016, 05:52:28 PM
I haven't asked Stuart yet what version of the software he was using, but several things suggest version 1.4.1 of YaBB SE.

The first of those is the following capture on the Wayback Machine:

http://web.archive.org/web/20030203174103/http://www.mega-tokyo.com/forum/index.php

If you hover over the link at the bottom to the YaBB SE web site, it includes the version of YaBB, which it gives as 1.4.1.

I've also been through the 1.4.1 code base and the sql script exactly matches the table structures of what Stuart sent me. So I think that asking him that question would just be a formality. It seems highly likely he'd confirm 1.4.1 is the version. I did notice that some of the earlier captures on the Wayback Machine were using 1.3.0, so it looks like he upgraded at least once.

Although the SMF database table structure is quite similar to YaBB SE, there are quite a number of differences. I did notice that there were some conversion scripts to convert from YaBB SE to SMF, so perhaps those would be useful. Maybe YaBB SE 1.4.1 could be installed somewhere, the MT forums data backup script loaded, the YaBB SE to SMF upgrade script run on top of that to convert to SMF, confirm it is all working, and then export that again for inclusion in to the sciprogramming.com forums somehow?
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 27, 2016, 06:22:33 PM
Or one could generate statics of the relevant threads. Might be the neater solution, considering things like user accounts and such. Hell, pass me the dump and I'll do it in a day or two, including hosting.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 27, 2016, 06:48:30 PM
Yeah, that'd be great actually. Grab it from here (and anyone else who wants to attempt to do something with it):

https://sites.google.com/site/agifans/files/mt_forums.zip (https://sites.google.com/site/agifans/files/mt_forums.zip)

It seems YaBB SE 1.4.1 won't run on the latest version of PHP and mysql anyway, which isn't surprising I guess. The SQL script above works fine with the latest mysql, but the script that comes with YaBB SE 1.4.1 needs some minor tweaks to change the TYPE=MyISAM to be ENGINE=MyISAM.

When I tried to install and run YaBB SE 1.4.1, I got this error: 

8192: Function split() is deprecated
(/var/www/html/forum/Sources/Subs.php ln 650)

The readme recommends this version of PHP:

PHP 4.0.4  -  Unix Platforms
PHP 4.1.x  -  Win32 Platforms

So if someone was going to try getting it running as it was at the time the export was done (in 2004), then they might have to use an older PHP. Either that or tweak the php scripts until they work.
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 27, 2016, 07:11:51 PM
I've nothing to do tomorrow so good night and thanks for the link.
Title: Re: Mega Toyko AGI forums backup
Post by: Collector on December 27, 2016, 10:31:28 PM
There seems to be a YaBB SE to SMF database3 converter here: http://www.simplemachines.org/community/index.php?P=8d2c58b6e9b4aed24d8af22cd436f11d&topic=140776.0

I would say to convert it to import into a clean SMF install and then the threads could be imported here.
Title: Re: Mega Toyko AGI forums backup
Post by: Collector on December 27, 2016, 10:33:26 PM
Too bad that there is not a backup of the images and attachments, too.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 28, 2016, 02:16:11 AM
There seems to be a YaBB SE to SMF database3 converter here: http://www.simplemachines.org/community/index.php?P=8d2c58b6e9b4aed24d8af22cd436f11d&topic=140776.0

Worth a shot. Reading through that page, I see that Chris Cromer was involved in the discussion.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 28, 2016, 02:34:07 AM
Actually, using the converter was my initial plan. I'd seen the converter already in this location:

http://download.simplemachines.org/?converters;software=yabbse

I've just tried running it and the first page assumes that you have both SMF and YaBB SE installations that it can look at. It has a field for each where you must specify the path to the install. So I'm assuming that getting the exported data working under YaBB SE is probably the first step. The converter is for version 1.5.x.  So perhaps given the data is from 1.4.1, we'd need to get it working on 1.4.1 first and then upgrade that to 1.5.x and then run the convert script.
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 28, 2016, 04:30:00 AM
http://helmet.kafuka.org/megatokyo/ (http://helmet.kafuka.org/megatokyo/)

First pass, need a better BBCode parser.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 28, 2016, 06:42:29 PM
Quite good as a quick effort. I was using it to have a little read of a few topics earlier today.

I've managed to get YaBB 1.4.1 running against a restore of the MT DB backup, and I've also tweaked the styles a bit to make it look like MT. Things like logging in aren't working, but from a read only perspective, it seems to be working well.

So I've just started a web crawler against my local install to see if I can suck out all the content as a static site. Not sure how big it will end up being, or how long it is going to take, but is worth giving it a go I guess. I'm using HTTrack. I didn't tweak any of the settings, so it is possible that if it doesn't give me what I want on the first pass, I can dig a little deeper and see if it can be made to work better. Seems to be fetching images from external sites at the moment. Probably avatars. I should be able to turn that off.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 29, 2016, 01:23:48 PM
I decided to configure both the topics per page and messages per page to 1000 so that it would reduce the number of pages that are crawled, and I think as a static site, that might end up being more useful as well. It would mean, for example, that the AGI board would have all of the topics in one big long list (i.e. no pagination), and then when you click through to a particular topic, all messages for that topic would appear in one page (i.e. no pagination). The end result will basically be what Kawa has already done, but with the original YaBB SE look and feel (adjusted to look somewhat like the MT styles), and it will of course take up far more disk space.

It's been running for 42 minutes so far and has saved over 400 Meg of data. I'm expecting it to take hours to finish; may even have to leave it running overnight. I'm kind of hoping it isn't sucking down the whole Internet with it.
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 29, 2016, 01:45:39 PM
over 400 Meg
Mine's 5.61 MB. Quite a difference.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 29, 2016, 02:07:45 PM
Yeah.  :D  And it's a long way off completion as well. I'm not certain what it's doing really. I suspect it is crawling links in YaBB SE that I wouldn't really want it to. Either I'm going to have to add in some exclusions (if the tool supports that), or I'll have to remove a whole lot of files at the end of the process.
Title: Re: Mega Toyko AGI forums backup
Post by: lance.ewing on December 29, 2016, 02:36:41 PM
I highly suspect that that previous crawl was going to take forever. So I've explored the options a bit and have tried to restrict what it is crawling. It may end up being too restricted now, but I can build up from here I guess.
Title: Re: Mega Toyko AGI forums backup
Post by: Kawa on December 29, 2016, 06:29:34 PM
Fun fact: Mega Toyko. The thread title is borken. I like it.
Title: Re: Mega Tokyo AGI forums backup
Post by: lance.ewing on December 30, 2016, 02:58:20 AM
Fixed.  :)
Title: Re: Mega Toyko AGI forums backup
Post by: troflip on December 30, 2016, 07:46:35 PM
http://helmet.kafuka.org/megatokyo/ (http://helmet.kafuka.org/megatokyo/)

First pass, need a better BBCode parser.

Thanks, lots of interesting history there :P
Title: Re: Mega Tokyo AGI forums backup
Post by: lskovlun on December 30, 2016, 08:39:56 PM
Interesting. There are some things here that I don't even remember attempting, such as the multiple egos thing. ???
Title: Re: Mega Tokyo AGI forums backup
Post by: MusicallyInspired on December 31, 2016, 12:19:27 PM
This is so great. I thought this stuff was lost to time.
Title: Re: Mega Tokyo AGI forums backup
Post by: lance.ewing on December 31, 2016, 12:44:00 PM
Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.
Title: Re: Mega Tokyo AGI forums backup
Post by: Kawa on December 31, 2016, 01:27:11 PM
Yeah, look at the AGI forum now.
Title: Re: Mega Tokyo AGI forums backup
Post by: Collector on December 31, 2016, 02:32:26 PM
My posts would be in the SCI forum.
Title: Re: Mega Tokyo AGI forums backup
Post by: MusicallyInspired on December 31, 2016, 08:15:21 PM
Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.

Interesting. I came after that "golden age" of AGI game development. I started getting involved right around 2000 or so, if memory serves.

Now it's just too bad that Chris's AGI forums weren't backed up. Such a shame. Some good stuff there as well. I was never very active in the AGI community, though.
Title: Re: Mega Tokyo AGI forums backup
Post by: Collector on December 31, 2016, 10:32:43 PM
Yeah, Chris told me that there was no backup at all. At least I was able to reconstruct all of the stuff from his Wiki via the Wayback.
Title: Re: Mega Tokyo AGI forums backup
Post by: Nicktatorship on April 19, 2017, 02:51:56 AM
So, this is kind of an old thread, but today while searching around (trying to determine if someone I found was the same person as from AGI past), I found an old comment on a post on the osdev forums. Then realised I was talking about Nat Budin, and *then* realised I recognised Nick Sonneveld's username too. I did another search and just, did one of the old forums get consumed by osdev, or were we just... spread out there too?

Examples:
http://forum.osdev.org/viewtopic.php?f=13&t=3236 (http://forum.osdev.org/viewtopic.php?f=13&t=3236)

http://forum.osdev.org/viewtopic.php?f=11&t=4259 (http://forum.osdev.org/viewtopic.php?f=11&t=4259)

http://forum.osdev.org/viewtopic.php?f=11&t=4157 (http://forum.osdev.org/viewtopic.php?f=11&t=4157)

Maybe this was already known, but it's striking me as bizarre.

EDIT: Okay, seems a lot of mega-tokyo may have ended up in forum 11 also? Still some vestiges of what once was. http://forum.osdev.org/viewforum.php?f=11&start=2650 (http://forum.osdev.org/viewforum.php?f=11&start=2650)

EDIT 2: Cross-referenced it with some of the posts in the snapshot Lance linked to earlier in the thread and yeah, looks like the Off-Topic and OS Dev bits are still intact on the main osdev site.
Title: Re: Mega Tokyo AGI forums backup
Post by: lance.ewing on April 19, 2017, 03:52:41 AM
I'd have to check to be certain but I suspect some of the non-AGI/SCI specific forums were consumed as you suggest may have happened. I have a db backup of the whole of the MT forums, so could search for those posts later on; unless someone else beats me to it.
Title: Re: Mega Tokyo AGI forums backup
Post by: Collector on April 19, 2017, 10:54:17 AM
As I remember it there were three main forums on Mega Tokyo; AGI, SCI and OS Dev. I never paid much attention to the OS Dev.
Title: Re: Mega Tokyo AGI forums backup
Post by: MusicallyInspired on April 19, 2017, 03:28:42 PM
Ah yes I recall that as well...never went in there myself either.
Title: Re: Mega Tokyo AGI forums backup
Post by: lance.ewing on April 19, 2017, 03:51:00 PM
I've now checked the MT DB backup and can confirm that all three of those threads are in that backup.

We have confirmation of the "consumption" in the following osdev.org thread from last year:

http://forum.osdev.org/viewtopic.php?f=11&t=30895

First reply says this:

Quote
Briefly: one of the two (or three?) fora that were eventually merged into OS-Dev was a forum on a message board called Mega-Tokyo, originally hosted on Geocities IIRC. It was primarily a site for supporting an FOSS reimplementation of the AGI and SGI game interpreters originally used by Sierra Online in the mid-1980s. I am not sure how it came to host an OS forum, but I assume that one of the mods had an interest in the subject.

Reading on it would seem that it was a merger of the osdev forums on MT and the osdev.org forums:

http://wiki.osdev.org/History#Timeline_of_the_OSDev_Community

Quote
Oct 18 2006: The two largest os development forums, Mega-Tokyo.com and OSDev.org, are merged in to one single forum creating the single largest community of operating system developers working on different OSes. The original OS development newsgroups are eclipsed by several orders of magnitude in traffic.