ClarkesworldIf you've been blocking AI from scraping your website, there's another one to add. This time it's Google.
I've updated my post on the subject.
https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
I've modified this post to reflect that Google-Extended ISN'T a bot. It's a way of letting Google know you don't want your site used in this manner. It MUST be in your robots.txt to work at all.
Eli the Bearded@clarkesworld When did they add this? I swear I don't remember it from last time I went through https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers looking for "bots" to block
