SeirdyAug 6, 2023

If you blocked ChatGPT-User in your robots.txt, you need to update it. OpenAI now uses the “GPTBot” user-agent.

User-agent: GPTBot Disallow: /

Got 100 hits yesterday. Ugh.

Edit: this won’t stop them in their tracks but it will at least make things slightly less convenient for OpenAI.

Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

Seirdy’s Home
Show threadStacey CampbellAug 8, 2023
@Seirdy
$ fgrep GPTBot access.log.1 | wc -l
2592
Show threadSeirdyAug 8, 2023
@stacey_campbell Jesus christ, what was the peak RPS?
Show threadStacey CampbellAug 8, 2023
@Seirdy A hit every 10 seconds. Very regular.
Show threadSeirdy
@stacey_campbell Oh, well that’s not too bad for a crawler. Much less intensive than being the first link in a post on Fedi and triggering the generation of thousands of link-previews.
Aug 8, 2023 at 4:01pmWeb