Mastodawn

Patrick Samphire Sep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show thread

Zach Bennoui Sep 29, 2023

@patricksamphire Genuine question, why are people so concerned about this? I personally really like the browsing stuff with GPT, but that doesn't necessarily make me want to visit a website less. It's about convenience, and intentionally breaking this doesn't seem like the right thing to do in my opinion.

Show thread

Patrick Samphire

@ZBennoui For one, it's not just about search. It's being used to put lots of people out of jobs, by taking their writing and generating copy from it without permission. Secondly, it hallucinates. That information you're getting is just as likely to be made up as accurate. Copy writers, translators, artists have all lost jobs and income already from this. I'm not going to let my writing contribute to that.