Author Topic: Mega Tokyo AGI forums backup  (Read 694 times)

0 Members and 1 Guest are viewing this topic.

Offline lance.ewing

Re: Mega Toyko AGI forums backup
« Reply #15 on: December 28, 2016, 06:42:29 PM »
Quite good as a quick effort. I was using it to have a little read of a few topics earlier today.

I've managed to get YaBB 1.4.1 running against a restore of the MT DB backup, and I've also tweaked the styles a bit to make it look like MT. Things like logging in aren't working, but from a read only perspective, it seems to be working well.

So I've just started a web crawler against my local install to see if I can suck out all the content as a static site. Not sure how big it will end up being, or how long it is going to take, but is worth giving it a go I guess. I'm using HTTrack. I didn't tweak any of the settings, so it is possible that if it doesn't give me what I want on the first pass, I can dig a little deeper and see if it can be made to work better. Seems to be fetching images from external sites at the moment. Probably avatars. I should be able to turn that off.

Offline lance.ewing

Re: Mega Toyko AGI forums backup
« Reply #16 on: December 29, 2016, 01:23:48 PM »
I decided to configure both the topics per page and messages per page to 1000 so that it would reduce the number of pages that are crawled, and I think as a static site, that might end up being more useful as well. It would mean, for example, that the AGI board would have all of the topics in one big long list (i.e. no pagination), and then when you click through to a particular topic, all messages for that topic would appear in one page (i.e. no pagination). The end result will basically be what Kawa has already done, but with the original YaBB SE look and feel (adjusted to look somewhat like the MT styles), and it will of course take up far more disk space.

It's been running for 42 minutes so far and has saved over 400 Meg of data. I'm expecting it to take hours to finish; may even have to leave it running overnight. I'm kind of hoping it isn't sucking down the whole Internet with it.

Offline Kawa

Re: Mega Toyko AGI forums backup
« Reply #17 on: December 29, 2016, 01:45:39 PM »
over 400 Meg
Mine's 5.61 MB. Quite a difference.

Offline lance.ewing

Re: Mega Toyko AGI forums backup
« Reply #18 on: December 29, 2016, 02:07:45 PM »
Yeah.  :D  And it's a long way off completion as well. I'm not certain what it's doing really. I suspect it is crawling links in YaBB SE that I wouldn't really want it to. Either I'm going to have to add in some exclusions (if the tool supports that), or I'll have to remove a whole lot of files at the end of the process.

Offline lance.ewing

Re: Mega Toyko AGI forums backup
« Reply #19 on: December 29, 2016, 02:36:41 PM »
I highly suspect that that previous crawl was going to take forever. So I've explored the options a bit and have tried to restrict what it is crawling. It may end up being too restricted now, but I can build up from here I guess.

Offline Kawa

Re: Mega Toyko AGI forums backup
« Reply #20 on: December 29, 2016, 06:29:34 PM »
Fun fact: Mega Toyko. The thread title is borken. I like it.

Offline lance.ewing

Re: Mega Tokyo AGI forums backup
« Reply #21 on: December 30, 2016, 02:58:20 AM »
Fixed.  :)

Offline troflip

Re: Mega Toyko AGI forums backup
« Reply #22 on: December 30, 2016, 07:46:35 PM »
http://helmet.kafuka.org/megatokyo/

First pass, need a better BBCode parser.

Thanks, lots of interesting history there :P
Check out my website: http://icefallgames.com
Groundhog Day Competition

Offline lskovlun

Re: Mega Tokyo AGI forums backup
« Reply #23 on: December 30, 2016, 08:39:56 PM »
Interesting. There are some things here that I don't even remember attempting, such as the multiple egos thing. ???

Offline MusicallyInspired

Re: Mega Tokyo AGI forums backup
« Reply #24 on: December 31, 2016, 12:19:27 PM »
This is so great. I thought this stuff was lost to time.
Brass Lantern Prop Competition

Offline lance.ewing

Re: Mega Tokyo AGI forums backup
« Reply #25 on: December 31, 2016, 12:44:00 PM »
Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.

Offline Kawa

Re: Mega Tokyo AGI forums backup
« Reply #26 on: December 31, 2016, 01:27:11 PM »
Yeah, look at the AGI forum now.

Offline Collector

Re: Mega Tokyo AGI forums backup
« Reply #27 on: December 31, 2016, 02:32:26 PM »
My posts would be in the SCI forum.
KQII Remake Pic

Offline MusicallyInspired

Re: Mega Tokyo AGI forums backup
« Reply #28 on: December 31, 2016, 08:15:21 PM »
Yeah, it's really great. I've been reading through the AGI board on the bus and train this week. I missed all of the period of the AGI community between 2000-2010. I think I posted a few messages in the earliest incarnation of an AGI board in the 90s, but between 2000-2010, I was out of the picture altogether. You won't find a single post in there from me, which is a shame really, looking back. - Oh well. It's nice to see how active it was once upon a time.

Interesting. I came after that "golden age" of AGI game development. I started getting involved right around 2000 or so, if memory serves.

Now it's just too bad that Chris's AGI forums weren't backed up. Such a shame. Some good stuff there as well. I was never very active in the AGI community, though.
Brass Lantern Prop Competition

Offline Collector

Re: Mega Tokyo AGI forums backup
« Reply #29 on: December 31, 2016, 10:32:43 PM »
Yeah, Chris told me that there was no backup at all. At least I was able to reconstruct all of the stuff from his Wiki via the Wayback.
KQII Remake Pic


SMF 2.0.11 | SMF © 2015, Simple Machines
Simple Audio Video Embedder

Page created in 0.108 seconds with 21 queries.