Stefan JudisJul 28, 2024

This is pretty wild — our friends at "Read The Docs" saw file download traffic drop by 75% after blocking AI bots.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

AI crawlers need to be more respectful

We talk a bit about the AI crawler abuse we are seeing at Read the Docs, and warn that this behavior is not sustainable.

Read the Docs
Show thread<baris>
Haha no surprises there, I had to block a lot of these bots as well since they literally DDoS you.
Jul 29, 2024 at 2:00pmWeb