SeirdyAug 6, 2023

If you blocked ChatGPT-User in your robots.txt, you need to update it. OpenAI now uses the “GPTBot” user-agent.

User-agent: GPTBot Disallow: /

Got 100 hits yesterday. Ugh.

Edit: this won’t stop them in their tracks but it will at least make things slightly less convenient for OpenAI.

Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

Seirdy’s Home
Show threadAnotherGrumpyDayInHell 💙🇺🇲🇺🇦
@Seirdy I'm kind of a newb when it comes to all of this, how do I use robots.txt in my preferences so that I don't get scraped by AI bots?
Aug 7, 2023 at 2:12amWeb
Show threadSeirdyAug 7, 2023
@AnotherDayInHell See https://seirdy.one/robots.txt to disallow crawling. Disallowing indexing is a little more complicated, though.
Show threadAnotherGrumpyDayInHell 💙🇺🇲🇺🇦Aug 7, 2023
@Seirdy thank you