SeirdyAug 6, 2023

If you blocked ChatGPT-User in your robots.txt, you need to update it. OpenAI now uses the “GPTBot” user-agent.

User-agent: GPTBot Disallow: /

Got 100 hits yesterday. Ugh.

Edit: this won’t stop them in their tracks but it will at least make things slightly less convenient for OpenAI.

Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

Seirdy’s Home
Show threadSeirdyAug 6, 2023
‘k, updated the seirdy.one robots.txt. Changes will go live in a sec.
Show threadSeirdyAug 7, 2023
Does anybody know if they respect a GPTBot-specific noindex tag? if so I might allow them to crawl just so they discover the tag and exclude all syndicated versions of my site from their index. Since the issue isn’t crawling, it’s indexing.
Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

Seirdy’s Home
Show threadThe Doctor
@Seirdy Doubtful.
Aug 7, 2023 at 12:39amWeb