Patrick SamphireSep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show threadPatrick SamphireSep 29, 2023
It won't stop everything, but it does cover a lot of the major ones.
Show thread_NecSep 29, 2023
@patricksamphire any of these even read and parse the robots.txt at all?
Show threadJohannes  Sep 29, 2023
@_Nec @patricksamphire well if they don’t and it ever comes out, you auto-win the resulting lawsuit and can retire on your own island 😂
Show thread_Nec
@jkbecker @patricksamphire Is that regulated by FCC? Lawsuit on what base?
(always found "robots.txt" a hilarious joke, is it actually legally binding?)
Sep 29, 2023 at 12:42pmElk
Show threadPatrick SamphireSep 29, 2023
@_Nec @jkbecker No, it's not legally binding. However, all the bots listed actually do obey it.