finally have the pieces in place for sciop to get hooked up to a collaborative scrape swarm, like archiveteam warrior except for generic web scrapes as well as more curated scrapes, and instead of uploading to archive.org it auto-creates torrents. first stage in closing the loop, to be continued by mutating the 'bounty' system from private trackers into a web of trust validation and prioritization system, anticipating when we get to federation, want to have local instance-scale preservation targets as well as global quorum sensing for what needs to be preserved. more info later, sleep now
prosocial botnet
sysadmins don't be mad at me i'm trying to make there be less scrape hits against your server by deduplicating and coordinating them and offloading them into torrents lol
also we'll have some cool news regarding web-accessibility of torrent-backed web archives soon. we are making the distributed archive dot org real.
so hopefully the picture is coming into focus of federated archive instances monitoring sets of pages, being able to distribute periodic and targeted snapshots of them as ongoing automatic scrape tasks across volunteer machines, autoseeding via torrent feeds, and then webtorrent-powered snapshot perusal. #sciop is just getting started baby.