Mastodawn

Patrick Samphire Sep 29, 2023

To stop AIs scraping your website for content, add this to your robots.txt file on your website.

Thanks to Neil Clarke for most of these.

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-Agent: FacebookBot
Disallow: /

Show thread

Ale

@patricksamphire i read "to stop Ale scrapping your website" :c . to make me stop scrapping your website you just have yo disallow request made by non browsers. i do not scrape pages with that rule bc is literally saying "we do not want you to scrape here" (like nitter)