important info if you have a blog/content website etc which you would not want to be used by OpenAI for their data vacuuming
➡️ add this to robots.txt
User-agent: GPTBot
Disallow: /
➡️ or block these IP ranges
20.15.240.64/28
20.15.240.80/28
20.15.240.96/28
20.15.240.176/28
20.15.241.0/28
20.15.242.128/28
20.15.242.144/28
20.15.242.192/28
40.83.2.64/28
@filipw Added to robots.txt, thanks for sharing that!
(Would add IP blocks, too, but I don't think either GitHub Pages or Netlify support that except on very expensive plans.)
@filipw it would be nice to have the same for google and others invasive ones.
Or if we could sue them for that kind of behaviour ...
to be clear: robots.txt is processed from top to bottom, so that line needs to go before any User-agent: * blocks.
Do I have that right?
@filipw Well, this part is also very interesting: https://platform.openai.com/docs/gptbot/customize-gptbot-access
What would happen if a lot of people would just dump a lot of random meaningless stuff in a specific file for chatGPT (probably something that "looks like" meaningful text) xD
@filipw since the bots are actively scraping, I think both (as @genehack suggested) plus redirections such as…
Filip W
Cassandra is only carbon now
Hyde 📷 🖋 
Martin Vassor
David Krooshof
Vasilis
Alessio 
Roger Lipscombe
Ursidinoj/The Bjornsdottirs
Mal 甄/kalessin/Peri
The Doctor
Mark 
Roel
WTL
Quenby
Richard W. Woodley ELBOWS UP 🇨🇦🌹🚴♂️📷 🗺️
genehack
mirabilosRewriteCond %{HTTP_USER_AGENT} " +https://openai."
RewriteRule . - [R=451]
RewriteCond %{HTTP_USER_AGENT} "GPTBot/"
RewriteRule . - [R=451]
… to get them if they acquire new IPs. (Perhaps exclude robots.txt from that rule, who knows.)
JesseTong
ShadSterling
Peter Hoseyrobots.txt isn’t rechecked…