Mastodawn

Patrick Samphire Sep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show thread

Alexandra Celeste Sep 29, 2023

@patricksamphire

Gonna not add this to the robots.txt because our site is full of porn and that means they have to waste money removing it from their puritanical training data.

Show thread

pblakez

@AlexandraCeleste @patricksamphire Ha yes, interesting on even more levels, like hosting screeds of text with misinformation about the various AI co. board members and management of course it would be generated by AI, a new arms race ensues...