MistyMar 25, 2025

I really can't overemphasize how destructive this is to the open web. The web is built just as much on a consensus on how to fairly do things as it is on technical measures keep things going, and AI scrapers have blown that up. https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html

I know Molly White wrote a passionate post recently asking people not to throw away open access just to stop AI scrapers, but many people *don't have a choice* anymore. If the old deal is broken, some people may be forced to take things private.

AI bots are destroying Open Access

There's a war going on on the Internet. AI companies with billions to burn are hard at work destroying the websites of libraries, archives, ...

Show threadMx Autumn 
@misty I have seen a number of websites start using "proof of work" holding pages that delay continuing for five seconds or so before giving access.

It holds up the bots, wasting compute cycles, but likely also acts as a method to detect and block them, e.g. a human will wait while a bot may have multiple sessions attempting to hit different urls.
Mar 26, 2025 at 11:14amWeb