Stefan JudisJul 28, 2024

This is pretty wild — our friends at "Read The Docs" saw file download traffic drop by 75% after blocking AI bots.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

AI crawlers need to be more respectful

We talk a bit about the AI crawler abuse we are seeing at Read the Docs, and warn that this behavior is not sustainable.

Read the Docs
Show threadshom 🐧📷🤿🏔️🪚✊🏽
@stefan they should have known it was AI bots, humans don't read documentation /s
Jul 29, 2024 at 12:06pmWeb
Show threadPaul ShryockJul 29, 2024
@shom @stefan you're not wrong.
Show threadMark GardnerJul 29, 2024
@shom @stefan 👏🏻