I really can't overemphasize how destructive this is to the open web. The web is built just as much on a consensus on how to fairly do things as it is on technical measures keep things going, and AI scrapers have blown that up. https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html

I know Molly White wrote a passionate post recently asking people not to throw away open access just to stop AI scrapers, but many people *don't have a choice* anymore. If the old deal is broken, some people may be forced to take things private.

AI bots are destroying Open Access

There's a war going on on the Internet. AI companies with billions to burn are hard at work destroying the websites of libraries, archives, ...

@misty

I recall reading an article in Byte magazine back in the early 90s or thereabouts. Even at that time they were predicting that the internet will end up like a cable TV service or perhaps a library.
To get any kind of quality we will need to subscribe to a package of curated services/web sites which is walled off from other services and the wider internet jungle.

@sleepy62 @misty

That's what Gates et al were expecting too. The AOL/CompuServe/Prodigy model. Early/mid 90s was before commercial use of the internet really took off so nobody really knew what to expect, and so people assumed the existing models would hold.

@jonhendry @misty

Yes that is how much of it started out. Geocities also comes to mind. Maybe the open internet was always a dream that could never survive the on-slot of bad actors and AI bots.

We are going back to the future..