Stefan JudisJul 28, 2024

This is pretty wild — our friends at "Read The Docs" saw file download traffic drop by 75% after blocking AI bots.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

AI crawlers need to be more respectful

We talk a bit about the AI crawler abuse we are seeing at Read the Docs, and warn that this behavior is not sustainable.

Read the Docs
Show threadLea de Groot TZ+10

@stefan @paulshryock not sure about this modern technique of scaling up endlessly. Ive always been pretty happy with “when i get effectively DDOSed, I go down, im not paying for their abuse”.

I think the web would really be better overall if more people did that, possibly with “yeah the AI bots are trying to screw us again” announcements :(

Jul 28, 2024 at 9:33pmWeb