Stefan JudisJul 28, 2024

This is pretty wild — our friends at "Read The Docs" saw file download traffic drop by 75% after blocking AI bots.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

AI crawlers need to be more respectful

We talk a bit about the AI crawler abuse we are seeing at Read the Docs, and warn that this behavior is not sustainable.

Read the Docs
Show threadHarsh Shandilya
@stefan I've been seeing a very similar pattern on my much smaller site: https://androiddev.social/@msfjarvis/112848867904557404
Harsh Shandilya (@[email protected])

Attached: 1 image I started redirecting #AI crawlers away based on their user agents and the 307s are now the bulk of my traffic. I sure do love the AI revolution

Android Dev Social
Jul 29, 2024 at 12:24pmWeb