I try to be friendly to "AI" search web crawlers but I'm on (pretty good, but nevertheless) shared hosting and "Claude-SearchBot" has been hammering my site with ~12000 requests now, in intervals between 1-4 seconds. Inevitably, it ended up on the blocklist because of the excessive resource consumption.
And that's exactly ONE day after "Amzn-SearchBot" ended on the blocklist, for the exact same reason. Fucking idiots.
GoogleBot and BingBot have such a light footprint by comparison, IDK how it can be so hard for the companies that claim to have SUCH "intelligence" at their hands to not make that better. Instead, they just show their glaring ignorance and incompetence.
(not that I'd expect that their "search" results would bring much traffic to my site anyway... so I guess blocking them isn't a big loss in the end. đ€·đ»ââïž)
#Webmastering #Website #AI #Claude #Amazon #Hosting #Firewall #WAF