List of AI bots to add to robots.txt (although they may not obey -- may need to throw them in the bitbucket and 404 or 444 them). In addition to these, you may have to block specific random browser versions for the most aggressive bots who ignore robots.txt.
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt
@ai6yr whacking Scrapy seems a bit heavy handed, that's a whole-ass html handling framework and not a specific bot.
@ai6yr well, not all use is scraping, despite the name. Think of it like blocking webkit (if you had a means to know about it?)
If all you need/care about is users that come in via traditional browsers though, it's probably a fair move.
(Also scrapy users can change the user agent, I believe, as could any of these abusive LLM scrapers. Most ignore robots.txt anyway, did you know? Meaning putting this into your robots.txt may be doing less than you hope)
since robots.txt is ignored some new approaches are in need:
https://anubis.techaro.lol/
Cymphoni Fantastique
AI6YR Ben
draeath
jcccb