Websites don't just magically appear on the Wayback Machine, someone has to actually archive them. Sometimes that's Internet Archive themselves. Sometimes that's the volunteers at the ArchiveTeam community rushing to archive entire websites before they shut down.
That requires their own servers, and they're *expensive*. You can financially contribute on their OpenCollective site :)
https://opencollective.com/archiveteam/updates/changes-to-our-infrastructure-and-use-of-donations
@cbruegg Internet Archive does have their own crawler which seems to randomly wander around the web. They also have "Save Page Now" to manually submit individual pages.
The ArchiveTeam community runs archivebot to archive lists of URLs or recursively crawl specific websites, controlled via an IRC bot. They also run several distributed projects where users can contribute their bandwidth (eg. blogger which is doing 3GiB/min right now) and which usually involve website-specific crawling code.
They actively track upcoming website shutdowns on their wiki and try to get them saved before they close.
@jackyan I wasn't home when I got your message but I forwarded it to other archiveteam people. It seems they have now paused the job and they plan to resume it later with reduced speed.
Sorry for the inconvenience! There was really no need to make requests so fast if the site isn't shutting down soon, it's just a proactive archival.
@jackyan Some bits of information:
- ArchiveTeam ArchiveBot - that's how it identifies unless told otherwise - is not to be confused with some bots operated by Internet Archive. You can see my message above for a short summary of *different* IA-related bots out there.
- ArchiveBot is operated manually, For each website it crawls, some actual human has pasted the address, decided on parameters and hit <Enter>. Sometimes humans make mistakes and grab too much or too fast. IRC is the best way to reach the human in question ASAP.
- ArchiveBot reads robots.txt, but doesn't abide by it. It's a job of the human operator to decide on crawl speed, forbidden URLs and so on. Returning 429 or 503 doesn't have any immediate effect on the bot, but will hint the human (if one is paying attention) to dial the speed down. Please do not return 403 or 404 - they are considered permanent and won't be retried. Blackholing the IP for a day is suboptimal, but acceptable in cases of serious overload.
- ArchiveBot was originally designed for grabbing dying websites ASAP, so its default parameters are a bit skewed for emergency use. However, its main use today is grabbing websites that look valuable and would be a shame to lose - because even if it's active today, nobody knows what will happen tomorrow. Yours probably falls to this category. So, if you don't want to be hit by AB, don't publish anything useful :)
- Other sites are probably getting hit due to being linked from your main site. ArchiveBot tries so preserve some "context" of the target website by making a shallow copy of everything it links to.
- Long-term observations are unlikely to be ArchiveBot-related, unless your website is very popular. It's probably some Internet Archive crawler walking your site slowly and carefully.
Does it make sense?
Nicolás Alvarez
Christian Brüggemann
Jack Yan (甄爵恩)