SeirdyAug 6, 2023

If you blocked ChatGPT-User in your robots.txt, you need to update it. OpenAI now uses the “GPTBot” user-agent.

User-agent: GPTBot Disallow: /

Got 100 hits yesterday. Ugh.

Edit: this won’t stop them in their tracks but it will at least make things slightly less convenient for OpenAI.

Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

Seirdy’s Home
Show threaddon Elías (como los buses) 🥨Aug 7, 2023
@Seirdy do you know if their IPs are coming from a specific AS? It would be useful to block them at firewall level
Show threadSeirdyAug 7, 2023

@donelias Here are the docs: https://platform.openai.com/docs/gptbot

The page is just text ironically gated by an aggressive CAPTCHA. Links to IPs used are at the end.

OpenAI Platform

Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.

Show threaddon Elías (como los buses) 🥨
@Seirdy awesome, thanks 🙏
Aug 7, 2023 at 4:03amWeb