Patrick SamphireSep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show threadWandering Thinker
@patricksamphire And the bots don't will ignore this?
Sep 29, 2023 at 2:13pmWeb
Show threadPatrick SamphireSep 29, 2023
@wthinker Well-behaved bots will obey it. That includes the ones listed (GPT, Google, Facebook and some others). Other bots won't, but there's nothing you can do about that other than put in restrictions on how many pages can be crawled in how much time, and that won't be reliable anyway.