Mega Tokyo AGI forums backup

[lance.ewing]

Quite good as a quick effort. I was using it to have a little read of a few topics earlier today.

I've managed to get YaBB 1.4.1 running against a restore of the MT DB backup, and I've also tweaked the styles a bit to make it look like MT. Things like logging in aren't working, but from a read only perspective, it seems to be working well.

So I've just started a web crawler against my local install to see if I can suck out all the content as a static site. Not sure how big it will end up being, or how long it is going to take, but is worth giving it a go I guess. I'm using HTTrack. I didn't tweak any of the settings, so it is possible that if it doesn't give me what I want on the first pass, I can dig a little deeper and see if it can be made to work better. Seems to be fetching images from external sites at the moment. Probably avatars. I should be able to turn that off.

[lance.ewing]

I decided to configure both the topics per page and messages per page to 1000 so that it would reduce the number of pages that are crawled, and I think as a static site, that might end up being more useful as well. It would mean, for example, that the AGI board would have all of the topics in one big long list (i.e. no pagination), and then when you click through to a particular topic, all messages for that topic would appear in one page (i.e. no pagination). The end result will basically be what Kawa has already done, but with the original YaBB SE look and feel (adjusted to look somewhat like the MT styles), and it will of course take up far more disk space.

It's been running for 42 minutes so far and has saved over 400 Meg of data. I'm expecting it to take hours to finish; may even have to leave it running overnight. I'm kind of hoping it isn't sucking down the whole Internet with it.

[Kawa]

over 400 Meg

Mine's 5.61 MB. Quite a difference.

[lance.ewing]

Yeah.    And it's a long way off completion as well. I'm not certain what it's doing really. I suspect it is crawling links in YaBB SE that I wouldn't really want it to. Either I'm going to have to add in some exclusions (if the tool supports that), or I'll have to remove a whole lot of files at the end of the process.

[lance.ewing]

I highly suspect that that previous crawl was going to take forever. So I've explored the options a bit and have tried to restrict what it is crawling. It may end up being too restricted now, but I can build up from here I guess.

[Kawa]

Fun fact: Mega Toyko. The thread title is borken. I like it.

[lance.ewing]

Fixed. 

[troflip]

http://helmet.kafuka.org/megatokyo/

First pass, need a better BBCode parser.

Thanks, lots of interesting history there

[lskovlun]

Interesting. There are some things here that I don't even remember attempting, such as the multiple egos thing.

[MusicallyInspired]

This is so great. I thought this stuff was lost to time.

[lance.ewing]

Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.

[Kawa]

Yeah, look at the AGI forum now.

[Collector]

My posts would be in the SCI forum.

[MusicallyInspired]

Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.

Interesting. I came after that "golden age" of AGI game development. I started getting involved right around 2000 or so, if memory serves.

Now it's just too bad that Chris's AGI forums weren't backed up. Such a shame. Some good stuff there as well. I was never very active in the AGI community, though.

[Collector]

Yeah, Chris told me that there was no backup at all. At least I was able to reconstruct all of the stuff from his Wiki via the Wayback.