terry matulaKind of wild to see all the Mastodon/Fediverse instances that immediately ping my site after posting a link or even replying to a post with a link.
And then right after that, all the "exploit" pings.
I learned about Team Cymru, and their IP services. Pretty cool, and "free". Using PHP's dns_get_record, I'm able to make a super quick query for an ASN record, and block the ones I don't want.
I'm no security expert, but this seems to be working well.
This whole project is just pleasant to work on... no frameworks or database or external dependencies. Just pure PHP, with a smattering of CSS and JS.
bob.php 
@terry not enough context for me to latch on to what you're doing with it. for example... can i automate blocking the ai bots?
@bobmagicii it's a bit janky, but I pass the IP to the Team Cymru service to get the ASN. And I'm storing the asn to a blocklist.json file.
My biggest issue was with Tencent Chinese servers, with multiple IPs but they all had the same ASN. This way I can block ALL of them with a single value instead of hundreds/thousands of different IPs.
For AI bots that have "good" user agents, I have a separate system that parses the ua and blocks based on that. Most big AI companies are good about it.
@terry ok that is about what i was thinking was going on. i found user agent on the bots not trustworthy because after i made fun of them for sending clear user agents, a month later, i found their traffic sending normal browser agents from the same data centers.
like i don't see Anthropic Claude anymore but i still see traffic from their datacenter using a 2 year old iphone ua.
may be automated responses to my blocking.
but if i can get an authourity to just tell me "yeah, that's them"
@bobmagicii I could just be lucky, and looking at logs I don't see any "Claude" ua. I do see "amazonbot" and "GPTBot" among others. If Anthropic is hitting my site, they're not slamming it at least.
Looking at search referrers, it's interesting I don't see Google. Just DDG and Bing. Not sure if that's just because of the type of people who might search for obscure Houston bands, or Gemini screwing up search.