"we caught one" god. Luxurious.
Once again, I strongly advise you to set up fail2ban so that anyone you serve a 404 catches at least a full day ban, and if you don't care about talking to other people's services, do your best to fully block the IP ranges associated with all the major hosting companies.
@mhoye a single 404 is very aggressive, all it takes is one fatfingered link in a page pointing to your site and you're banning visitors en masse.
after 5-10 404s in a row for different URLs, different story.
@mhoye yes i'm all for having specific poison URLs that trigger an immediate ban.
But "immediate ban on any 404 whatsoever" seems heavy handed.
@mhoye @azonenberg .... we do personally do that
we're prepared to accept that we do not exist, in a statistical sense. that is true in SO many ways
but we hesitate to put into place a rule which would lock ourselves out. we'd at least apply a threshold of a few 404s over a window of time, not just a single one
@ireneista @mhoye exactly, i've copy pasted a link and truncated it or something so many times while making another blog page or sending an email etc.
URLs get truncated in emails by 80 column format limits all the time too.
If you ban on a single 404, or repeated hits to a single truncated/corrupted URL, you'll cut off a lot of legitimate users. Specific bogus URLs, e.g. .asp and .php URLs when your site is written in python, make a lot more sense as poison ban URLs.
As someone who is currently unable to order delivery from my local grocery store due to excessively paranoid WAFs, I'm strongly against this kind of hair trigger defense mechanism.
I've stopped buying components from Mouser because they lock me out constantly for reasons unknown, doing a single search for a component part number is often enough to trigger it.
@azonenberg @mhoye we have noticed that mouser is a particular offender, indeed, right up there with the less-regulated financial institutions (the various money transfer services that do the things banks do but avoid being treated as banks by the law) in regard to its hair-trigger nature
mouser does seem to take a much narrower list of signals into account than the money transfer services do, which makes it easier to defeat. so that's nice?
@azonenberg @mhoye and, like, for all that we care about the service they provide and rely on it, we would like to live in a world made of more, smaller power structures rather than a few large ones, because we believe smaller structures are easier to hold accountable to human purpose
so when we see a major corporation making a mistake, we aren't going to, like, point it out to them or complain that they're making it. it's a good thing that it's hurting them, heh, sigh
@azonenberg @mhoye but like
yeah, the rest of us should try to avoid making that same mistake. a thing we've been chewing on lately is what forms of log analysis we could do that would let us calibrate whether our existing fail2ban thresholds are getting false positives.... it's not an easy question, and we're not good at the deep part of statistics (understanding what it can and can't tell us)
mhoye
Andrew Zonenberg
Irenes (many)