p4ybackDec 17, 2022
Jerry 🦙💝🦙
Now at 1700 new accounts on the day @infosec.exchange. This is what it looks like on the load balancer:
Dec 16, 2022 at 11:28pmWeb
Show threadJerry 🦙💝🦙Dec 16, 2022
And this is what's happening at the CDN:
Show threadJerry 🦙💝🦙Dec 16, 2022
And this is what the aggregate CPU on the PUMA servers look like (note, almost no change)
Show threadJerry 🦙💝🦙Dec 16, 2022
And this is the aggregate load on the sidekiq servers (also, nearly no change)
Show threadJerry 🦙💝🦙Dec 16, 2022
Transaction growth on the postgresql database:
Show threadJerry 🦙💝🦙Dec 16, 2022
CPU on the redis/elasticsearch server. I can barely find it:
Show threadJerry 🦙💝🦙Dec 16, 2022
final graph (I don't have much else that would be interesting) is the sidekiq graph:
Show threadJerry 🦙💝🦙Dec 16, 2022
I lied. One more. I feel bad that I am just now realizing I didn't tell Prometheus to monitor the load balancer, so here is a snapshot of nmon from the load balancer:
Show threadJerry 🦙💝🦙Dec 16, 2022
ok.... one more for the sidekiq nerds:
Show threadMichael StancliftDec 17, 2022
@jerry that failure count 🥺
Show threadJerry 🦙💝🦙Dec 17, 2022
@vmstan there are an amazing number of other instances that are down. some for good
Show threadceolnDec 17, 2022
@jerry @vmstan
😢
Show threadpostmodernDec 17, 2022
@jerry that's a lot of failed jobs. What's the main cause of those? Failed requests to other servers?
Show threadJerry 🦙💝🦙Dec 17, 2022
@postmodern yes - 98% (give or take) are other instances that are dead/offline/not responding/error 500/etc
Show threadnicdex 🇨🇦 🇺🇦Dec 17, 2022
@jerry wow how many users do you have to require this much sidekiq threads ?
Show threadJerry 🦙💝🦙Dec 17, 2022
@nicdex 37k. I went through some rough times with Sidekiq a month ago. I never want my queues higher than 0 😂
Show threadVissDec 17, 2022

@jerry kinda surprised you dont have a giant linux terminal with:
- htop
- nload
- iftop
- atop
- ioping

and prolly asciiquarium and cmatrix for good measure :D

Show threadzeno's paradox speedrunnerDec 17, 2022

@Viss @jerry https://nicolargo.github.io/glances/

tho it can be a little heavy

Glances - An Eye on your system

Glances is a cross-platform curses-based system monitoring tool written in Python.

Show threadJerry 🦙💝🦙Dec 17, 2022
@pixelnull @Viss oh that looks amazing
Show threadzeno's paradox speedrunnerDec 17, 2022

@jerry @Viss it has a package avail for most distros.

i wouldn't do a github or pip install for it, they can get a little experimental with releases in my exp

Show threadVissDec 17, 2022
@pixelnull @jerry oh yes! glances! i always forget about glances. You need a big terminal for it tho
Show threadJerry 🦙💝🦙Dec 17, 2022
@Viss @pixelnull I have two 43" monitors 🎉​
Show threadVissDec 17, 2022
@jerry @pixelnull heh, i excitedly await the screenshots :D
Show threadzeno's paradox speedrunnerDec 17, 2022
@Viss @jerry nono take a picture of the monitors with your phone... make sure to brush your teeth in front of them tho. they need a bunch of gunk on them
Show threadVissDec 17, 2022

@pixelnull @jerry

brew install glances

Show threadaburka 🫣Dec 17, 2022
@Viss @pixelnull @jerry brew is always like "ah, if you wish to install an apple pie you must first install the universe"
Show threadVissDec 17, 2022
@aburka @pixelnull @jerry hah, yeah
Show threadCompuguy 🐱Dec 17, 2022
@aburka @Viss @pixelnull @jerry Yep, though I would 1-up you and say that trying to deploy software that still uses Python 3.6 & Pipenv is much worse....trust me!
Show threadzeno's paradox speedrunnerDec 17, 2022

@compuguy @aburka @Viss @jerry

python packaging is a dumpster fire, nestled in a tire fire, that is then placed in a high-rise apartment building fire.

a fire-turducken as it were

Show threadCompuguy 🐱Dec 17, 2022
@pixelnull @aburka @Viss @jerry Yep, but that's what the developers I work with use. That and Node/NPM. I call it "Dependency Hell"....
Show threadBagheeraAltered ​ Dec 16, 2022
@jerry as always, amazing work
Show threadVissDec 16, 2022
@jerry whoa those are some really interesting spikes. can you tell what they are?
Show threadScott Dec 16, 2022
@Viss @jerry lots and lots of INSERT INTO?
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss Not really. I have neglected to instrument the basic linux stats on the load balancer. That is simply queries that nginx is responding to, so I assume that it is probably related to something that causes lots of other instances to hit the server all at once, like a large account mentioning something here.
Show threadVissDec 16, 2022
@jerry interesting. yeah cuz the 'average load' looks to be considerably less than those spikes, so something is causing.. i guess a shitload of queries to happen all at once, in bursts?
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss exactly. If someone on an instance with followers on thousands of different instances posts a link that references this site, all those instances reach out. This is what the attack of last week was all about - creating many accounts that caused instances to attempt to DDOS some other sites.
Show threadVissDec 16, 2022
@jerry ah ha - illumnation of burst-style DDOS scenarios. too many signups spike psql, psql chokes, front end throws 500s
Show threadthroAUDec 17, 2022
@Viss @jerry probably every time Elon posts on twitter. Lol.
Show threadVissDec 17, 2022
@throAU @jerry well, banning journalists and letting nazis on the site were two big spikes, for sure
Show threadVissDec 16, 2022
@jerry i wonder how soon there's gonna be a patch to replace sidekiq with something else.
Show threadnullagentDec 16, 2022
@Viss @jerry Whenever that is, it wouldn't be too soon for my taste.
Show threadVissDec 16, 2022
@nullagent @jerry right? the biggest improvement to twitter back in the day was when they got the fuck off of ruby on rails
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss perhaps. Now that I have experience with it, it's not the worst thing in the world.
Show threadVissDec 16, 2022

@jerry so it looks like other folks have figured out how to migrate from sidekiq to rabbitmq https://stanko.io/rabbitmq-is-more-than-a-sidekiq-replacement-b730d8176fb

(we run rabbitmq. its way more burly. we dumped redis to go to this)

RabbitMQ is more than a Sidekiq replacement | Stanko K.R.

A rabbit dressed up as the karate kid - a cross of RabbitMQ and Sidekiq logos I've had gripes with Sidekiq because of which I switched to RabbitMQ. Here a...

Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss interesting - will have to look at it
Show threadVissDec 16, 2022
@jerry I imagine that its not gonna be super easy or straightforward to migrate from sidekiq to rabbitmq, but I would imagine that rabbit will treat your cpus and ram way better than sidekiq does
Show threadDan HonDec 16, 2022
@jerry @Viss oooh nice.
Show threadVissDec 16, 2022
@danhon @jerry this post doesnt have anything to do with mastodon specifically tho - but, you gotta start somewhere :D
Show threadCompuguy 🐱Dec 16, 2022
@Viss @jerry I doubt it. I've followed a guide and spit off the scheduler and mailer queues to a separate container. The rest can be scaled up to meet demand.
Show threadXavier Knol Dec 16, 2022
@jerry wow, that is quite an increase in traffic.
Show threadVissDec 16, 2022
@jerry when you moved to object storage, was that a massive i/o improvement?
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss I had the media on their own cluster, which were pushing ~500Mbps each at times
Show threadVissDec 16, 2022
@jerry whoof, thats nutty. ~17kish active users, was it? or was it 12k? i remember your older post saying you had like 27k users or something but a lot of those signups were like 'landgrabs to secure their handle' then they never post
Show threadzeno's paradox speedrunnerDec 16, 2022
@jerry dat cache hit percentage
Show threadzeno's paradox speedrunnerDec 16, 2022
@jerry fuck i'm glad you're a good admin
Show threadXavier Knol Dec 16, 2022
@jerry that looks like quite the traffic, the sine wave looks beautiful though.
Show threadLesley Carhart Dec 16, 2022
@jerry I am having a hard time getting a decent list of who’s really new on the directory page for some reason. I want to greet folks…
Show threadDJ SundogDec 16, 2022

@hacks4pancakes @jerry

try https://infosec.exchange/api/v1/directory?order=new&local=true (via https://infosec.exchange/api/v1/directory?order=new&local=true) (and congrats on your well deserved award!)

Show threadLesley Carhart Dec 16, 2022
@jerry @djsundog thank you!