Stefan JudisJul 28, 2024

This is pretty wild — our friends at "Read The Docs" saw file download traffic drop by 75% after blocking AI bots.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

AI crawlers need to be more respectful

We talk a bit about the AI crawler abuse we are seeing at Read the Docs, and warn that this behavior is not sustainable.

Read the Docs
Show threadcanteenJul 29, 2024
@stefan while this is important and good and I'm happy for them, the listed traffic costs are ridiculous
Show threadLatte macchiato  Jul 29, 2024
@canteen @stefan I see you don't use AWS. Good.

They charge 0.09$/GB of egress.
Show threadcanteen
@privateger @stefan The point I was making is that nobody should use AWS or any related exploitative cloud platforms
Jul 29, 2024 at 3:35pmWeb