Patrick SamphireSep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show threadsometimes ian

@patricksamphire Thanks for sharing this! I've created this to try and collect any future changes:

https://github.com/ecnepsnai/Robots.txt-Block-AI

Do you have a link for Neil Clarke that I could add to the attributes?

GitHub - ecnepsnai/Robots.txt-Block-AI: A robots.txt to ask Ai from scraping your content

A robots.txt to ask Ai from scraping your content. Contribute to ecnepsnai/Robots.txt-Block-AI development by creating an account on GitHub.

GitHub
Sep 29, 2023 at 6:28pmWeb
Show threadPatrick SamphireSep 29, 2023
@ecn This is the source I used: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
Block the Bots that Feed “AI” Models by Scraping Your Website – Neil Clarke