Mastodawn

Patrick Samphire Sep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show thread

🌈 Dana Oct 1, 2023

@patricksamphire I have a question, would the Google bot affect SERP? I know they test like AI answers pulled from several sources, so I wonder if that would hurt SEO.

Show thread

Patrick Samphire Oct 1, 2023

@dana_cz Shouldn't do. Google uses Googlebot for search engine results. This is blocking Google-extended, which is a bot used to train the Google generative AI.

Show thread

🌈 Dana

@patricksamphire Thank you! I'll give that a go and see what comes up. To be fair I haven't seen any AI SERP yet.