political campaign enjoyerMar 26

Today started out well: I got a bug report against iocaine that enabling the firewall results in send queues filling up. This is something I observed too, and was about to tackle today.

But the reporter made a suggestion: accept established & related connections, only drop new ones.

This allows requests in flight to be served normally, but new connections are dropped. This way, send queues do not fill up.

The downside is that if a crawler makes multiple requests within the same connection, they'll all get served (poison). That's not a big problem, because the costiest part of the whole deal is the TLS handshake, and we're past that at this point. Crawlers typically don't make many requests over the same connection anyway.

I'll be deploying this on Eru shortly, see how it behaves. I have high hopes. If it works out, I'll scale Eru back.

Show threadpolitical campaign enjoyerMar 26

New iocaine deployed, lets see how it performs!

Caddy's memory usage seems okayish so far, it's not using much more than prior to enabling iocaine's firewalling. That's a good sign, because prior  to the update, the memory use shot up pretty rapidly.

Show threadpolitical campaign enjoyerMar 26
It's.. hm. Something's not quite right.
Show threadpolitical campaign enjoyerMar 26
Meanwhile, I figured I can enable auto-merge, but ideally, I'd like to preserve my existing sets. Trying to export the current firewall so I can edit it to add auto-merge has been running for 15+ minutes.
Show threadpolitical campaign enjoyerMar 26
At this point, I might just... flush it and start anew while the server is scaled up.
Show threadpolitical campaign enjoyerMar 26
Yeah, ok, 30 minutes to export the sets is stupid. Lets flush this and re-create.
Show threadpolitical campaign enjoyerMar 26

There we go! Request rate down from ~1k req/sec to ~150 req/sec within ~45 minutes. Caddy memory use is fine.

iocaine's CPU use is rather high, though. It's currently the dominating process on Eru. I suppose the firewalling isn't that cheap, and I'm still banning IPs at a high-ish rate.

Will have to profile this at some point. But for now, I'm glad it's working out well so far.

If all goes well, I will release #iocaine 3.4.0 on April 1st. A departure from my usual Friday releases, but... I can't not release on April 1st if I have the opportunity.

Show threadpolitical campaign enjoyerMar 26
I think I need a graph for firewall blocks, see if there's any correlation with CPU use. Because iocaine keeps hogging an entire core, even though request rate is down to ~150 req/sec - yet, it's still using as much CPU as when request rate was around 800 req/sec.
Show threadVysogotaMar 26
@algernon err... I don't follow.. why would firewalling taking CPU lead to high CPU use of the iocaine process?
Show threadpolitical campaign enjoyer

@petko That's the thing. It shouldn't. But when I turn iocaine's firewalling code on, CPU use goes up to 100%, seemingly irregardless of the rate of incoming requests. It keeps spinning at high CPU even when request rate is low.

Heck, turning Caddy off, iocaine still uses ~10% CPU, even though it's supposed to do nothing at that point. I even kept it off for 15 minutes, so it can empty any firewall block request queues, and it was still at it.

If I turn firewalling off, it's a lot quieter.

Mar 26 at 7:42pmWeb