"we have noticed an increase in abusive site crawling, mainly from AI products and services. These products are recklessly crawling many sites across the web, and we've already had to block several sources of abusive traffic."

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day."

"By blocking these crawlers, bandwidth for our downloaded files has decreased by 75%"

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/

via @stefan