sastian
ClarkesworldIf you've been blocking AI from scraping your website, there's another one to add. This time it's Google.
I've updated my post on the subject.
https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
I've modified this post to reflect that Google-Extended ISN'T a bot. It's a way of letting Google know you don't want your site used in this manner. It MUST be in your robots.txt to work at all.
Eli the Bearded@clarkesworld When did they add this? I swear I don't remember it from last time I went through https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers looking for "bots" to block
John Morahan@clarkesworld Note that according to the linked Google article, "Google-Extended" isn't an actual crawler with its own user agent string - rather, it's a name that Google's existing crawlers interpret as a request to opt out of AI when they see it in robots.txt. So blocking it at the firewall etc would have no effect.
@jmorahan Hmmm. Ok. Updating document.
Robertus CognoscensThank you for this. I'm not entirely sure if it's necessary for me if I use GitHub Pages, but I added the file nonetheless.
eklem@robertoqs @clarkesworld Also, does it work when the robots.txt-file is in a sub-folder and not root?
Hugo van Kemenade@eklem @robertoqs @clarkesworld
For GitHub Pages, add robots.txt to a repo called <username>.github.io and then it will appear at <username>.github.io/robots.txt
For example:
https://github.com/hugovk/hugovk.github.io/commit/79a14a01d37d574e2a76127722cdaf25cc1b9293
https://github.com/hugovk/hugovk.github.io
https://hugovk.github.io/robots.txt
More info:
https://stackoverflow.com/a/47652485/724176
https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages
#GitHub #GitHubPages #robotstxt
I think it has to be in the root directory.
@robertoqs @clarkesworld Thanks. I created a repo with the name [username].github.io as suggested by @hugovk and put a robots.txt in it, so I should be covered now.
Well, if you were already using GitHub Pages, you already had a repository named [username].github.io, if I'm not mistaken. The point about robots.txt is to add it to the root directory, otherwise known as the main branch in GitHub.
@robertoqs @eklem @clarkesworld
I didn't have a [username].github.io repo until I created it this morning. But I did have other repos using GitHub Pages, and they are served like [username].github.io/other-repo
But until I created [username].github.io with robots.txt, there was nothing at [username].github.io/robots.txt for the others. As you say, it must be at the root.
Some links to docs: https://mastodon.social/@hugovk/111146631566244808
Ah, I see. Then [username].github.io is only required when not using a custom domain, like in my website's case. That's what I was thinking about.
So what I did was simply to put robots.txt next to my HTML files. Also, thanks for the documentation. I love Stack Overflow, by the way.
Danny 👾@clarkesworld Google joining the 'bot party' is hardly surprising.
Adam Johnson 
Great post, thanks for all the reserach. Maybe add FacebookBot to block Meta’s efforts?
“FacebookBot crawls public web pages to improve language models for our speech recognition technology.”
@clarkesworld Also I added a link from my somewhat popular #Django robots.txt post:
Ann@clarkesworld Thank you.
Irina@clarkesworld Thanks! Three were already there and I added the other two.
EVHaste@clarkesworld Thanks fam. Glad to see this mag is still running btw.
Eystein@clarkesworld Thanks! That got me around to add my robots.txt
Doug Baker - Elbows Up! 🇨🇦@clarkesworld I'm pretty much a #Firefox guy now; hoping there's a difference.