Patrick SamphireSep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show threadPatrick Samphire

Add this one to the list:

User-agent: Amazonbot
Disallow: /

Thanks go to @fasterandworse

Sep 29, 2023 at 2:28pmWeb
Show threadኢራ עירא Ira عيرا 🍓🎗️Sep 29, 2023
@patricksamphire @fasterandworse why would I want that? I want the models to be better and I believe I write excellent content in my blog...
Show threadStephen FarrugiaSep 29, 2023
@I @patricksamphire go ahead
Show threadPatrick SamphireSep 29, 2023
@I @fasterandworse Cool. No one said you had to.
Show threadኢራ עירא Ira عيرا 🍓🎗️Sep 29, 2023
@patricksamphire @fasterandworse what I failed to see is why I would want to. My blog is public so it can be accessed and read, man or machine. What is my interest in blocking it? Even if I was not in favor of shared knowledge, open source and open data, I still don't see why I should want to block progress. It's not like I am losing money or reputation or any other measurable damage by leaving this door open.
Show threadPatrick SamphireSep 29, 2023
@I @fasterandworse Cool. A lot of people are losing jobs to "AI" products trained on scraped data. I'm not in favour of that, so I'm choosing not to let AI train on my work. You can choose otherwise, entirely freely.