@mgorny They may use Claude Code.

I‘d not be surprised if we see a decline in software and service quality over the next few years in general. Once all seniors are retired or laid off this may be the new normal.

I am guessing they load robots.txt before each intended fetch to verify that the URL they intend to fetch is permitted. If they primarily want resources that are not permitted, it would explain why they fetch robots.txt more often than anything else.

Of course caching robots.txt would be better. The only problem with that is that you may end up fetching a URL which is no longer permitted because you used an outdated version of robots.txt.

If you want a crawler to be extra well behaved you could take this approach:

• If your cached robots.txt is older than 24 hours or you haven’t cached it at all. Then retrieve robots.txt.
• If your cached robots.txt is less than 24 hours old and doesn’t permit the desired URL. Then you don’t retrieve anything.
• If your cached robots.txt is between 1 minute and 24 hours old and does permit the URL you intend to fetch, then you fetch robots.txt again to ensure the desired URL is still permitted.
• If your cached robots.txt is less than 1 minute old and does permit the desired URL you trust the cache.

But I think that’s probably a bit too advanced for an AI company to work out.