IBBoardIt's the weekend. So you know what that means…
Dealing with the gods damned fucking bots that are abusing my site because they're either vibe-coded or trying to power GenAI 😒
I'm currently at the "Can I block the entire 34.4.5.0 - 34.63.255.255 range, or does Google use it for legitimate purposes and not just hosting spammers and script kiddies on the googleusercontent.com domain" stage of blocking IPs 😐
The WhoIs info says "The IP addresses under this Org-ID are in use by Google Cloud customers", so I'm probably safe.
It's awesome how, in additional to the social, societal, environmental and capability damage that GenAI is causing, it is ALSO managing to fuck up the Internet 🙄
Current state of Apache:
WRRWRRWRCRRWRCRRRWRRRRRRCRRRRRRRRRRRCCRWWRRRRRRRRWRCRRCWRCRRRRRC
R_RCRCRWWWR_RCRRW_RR_WR_RW_RCRR_RR_RRRRRRRRRW_R_CRCWW_CRRCRCCRRR
And if I'm reading the server status page correctly (increasing "Seconds Since" and still showing the proxy IP with no verb and path) then a lot of these requests can be 30+ seconds in and STILL not processing properly.
Meanwhile, PHP FPM has 50-70 processes open and no upper cap, and doesn't seem to be struggling 😐
(R=Reading Request, W=Sending Reply, C=Closing Connection, _=Open Slot)
Added a third MPM event server. All of its threads were almost immediately consumed.
I'm getting a LOT of hits from a vast swathe of different African IPs. But they look like residential and mobile IPs rather than data centres. So blocking them also blocks legitimate users 😒
On the plus side, I've still got headroom on this little Pi. RAM is below 50% used and even with 256 threads for Apache then I'm still only at 75% CPU usage!
One thing that helps reduce bot traffic?
Putting the rewrite rules that BLOCK the bot traffic BEFORE the rules that say "rewrite this pretty URL to this PHP file and stop processing further rules" 🤦
Apparently some of my extension and refactoring a week or so back actually broke it 😐
Back to 128 threads for Apache and I've still got spares open! Much better.
Still getting quite a high load. But Apache's server status dashboard seems happier. So I think it's just lots of requests that are quickly being given a 402 or 429 status by my bot-and-hack-attempt blocking.
Good news: Load is WAY down today!
Turns out Fail2Ban and IP blocking is a HUGELY helpful way to keep bot traffic down, once you pick out a few patterns (like "dot files that could contain credentials")
You just need to have your server configured so that it is ACTUALLY doing the blocking, and not processing the request before it gets to the block 😐