Mastodawn

Patrick Samphire Sep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show thread

Juan Luis

@patricksamphire Why it's on website admins to keep adding bot names and having to be on top of the latest "AI" news to block them all? Can't we just say "block all the AI crawlers, past, present, and future"?

(Not blaming *you* of course, just venting 😭)

Show thread

Patrick Samphire Sep 29, 2023

@astrojuanlu You're right. In fact, we shouldn't have to block at all. They should only be able to crawl if we give express permission. But that's up to our governments to make the rules, and they haven't.

Show thread

Miha Markič Sep 30, 2023

@patricksamphire @astrojuanlu Not only governments, just a consensus to add "block: all" to robots.txt. Or/and block by default if that file doesn't exist.
If they are able to "obey" robots.txt, they would be able to obey this as well.