Jerry 🦙💝🦙Dec 16, 2022
Now at 1700 new accounts on the day @infosec.exchange. This is what it looks like on the load balancer:
Show threadJerry 🦙💝🦙Dec 16, 2022
And this is what's happening at the CDN:
Show threadJerry 🦙💝🦙Dec 16, 2022
And this is what the aggregate CPU on the PUMA servers look like (note, almost no change)
Show threadJerry 🦙💝🦙
And this is the aggregate load on the sidekiq servers (also, nearly no change)
Dec 16, 2022 at 11:36pmWeb
Show threadJerry 🦙💝🦙Dec 16, 2022
Transaction growth on the postgresql database:
Show threadJerry 🦙💝🦙Dec 16, 2022
CPU on the redis/elasticsearch server. I can barely find it:
Show threadJerry 🦙💝🦙Dec 16, 2022
final graph (I don't have much else that would be interesting) is the sidekiq graph:
Show threadJerry 🦙💝🦙Dec 16, 2022
I lied. One more. I feel bad that I am just now realizing I didn't tell Prometheus to monitor the load balancer, so here is a snapshot of nmon from the load balancer:
Show threadJerry 🦙💝🦙Dec 16, 2022
ok.... one more for the sidekiq nerds:
Show threadMichael StancliftDec 17, 2022
@jerry that failure count 🥺
Show threadJerry 🦙💝🦙Dec 17, 2022
@vmstan there are an amazing number of other instances that are down. some for good
Show threadceolnDec 17, 2022
@jerry @vmstan
😢
Show threadpostmodernDec 17, 2022
@jerry that's a lot of failed jobs. What's the main cause of those? Failed requests to other servers?
Show threadJerry 🦙💝🦙Dec 17, 2022
@postmodern yes - 98% (give or take) are other instances that are dead/offline/not responding/error 500/etc
Show threadnicdex 🇨🇦 🇺🇦Dec 17, 2022
@jerry wow how many users do you have to require this much sidekiq threads ?
Show threadJerry 🦙💝🦙Dec 17, 2022
@nicdex 37k. I went through some rough times with Sidekiq a month ago. I never want my queues higher than 0 😂
Show threadVissDec 17, 2022

@jerry kinda surprised you dont have a giant linux terminal with:
- htop
- nload
- iftop
- atop
- ioping

and prolly asciiquarium and cmatrix for good measure :D

Show threadzeno's paradox speedrunnerDec 17, 2022

@Viss @jerry https://nicolargo.github.io/glances/

tho it can be a little heavy

Glances - An Eye on your system

Glances is a cross-platform curses-based system monitoring tool written in Python.

Show threadJerry 🦙💝🦙Dec 17, 2022
@pixelnull @Viss oh that looks amazing
Show threadzeno's paradox speedrunnerDec 17, 2022

@jerry @Viss it has a package avail for most distros.

i wouldn't do a github or pip install for it, they can get a little experimental with releases in my exp

Show threadVissDec 17, 2022
@pixelnull @jerry oh yes! glances! i always forget about glances. You need a big terminal for it tho
Show threadJerry 🦙💝🦙Dec 17, 2022
@Viss @pixelnull I have two 43" monitors 🎉​
Show threadVissDec 17, 2022
@jerry @pixelnull heh, i excitedly await the screenshots :D
Show threadzeno's paradox speedrunnerDec 17, 2022
@Viss @jerry nono take a picture of the monitors with your phone... make sure to brush your teeth in front of them tho. they need a bunch of gunk on them
Show threadVissDec 17, 2022

@pixelnull @jerry

brew install glances

Show threadaburka 🫣Dec 17, 2022
@Viss @pixelnull @jerry brew is always like "ah, if you wish to install an apple pie you must first install the universe"
Show threadVissDec 17, 2022
@aburka @pixelnull @jerry hah, yeah
Show threadCompuguy 🐱Dec 17, 2022
@aburka @Viss @pixelnull @jerry Yep, though I would 1-up you and say that trying to deploy software that still uses Python 3.6 & Pipenv is much worse....trust me!
Show threadzeno's paradox speedrunnerDec 17, 2022

@compuguy @aburka @Viss @jerry

python packaging is a dumpster fire, nestled in a tire fire, that is then placed in a high-rise apartment building fire.

a fire-turducken as it were

Show threadCompuguy 🐱Dec 17, 2022
@pixelnull @aburka @Viss @jerry Yep, but that's what the developers I work with use. That and Node/NPM. I call it "Dependency Hell"....
Show threadzeno's paradox speedrunnerDec 17, 2022

@compuguy @aburka @Viss @jerry normally, i'm all like "fuck it throw in the whole [python current version] and all the libs with it"

60Megs for a small script? sure why not

Show threadCompuguy 🐱Dec 17, 2022
@pixelnull @aburka @Viss @jerry That actually exists. Its called PyInstaller. Works great unless you have to use a *really* old OS like I don't know.....CentOS 7. Curse you GLIBC 2.17!
Show threadzeno's paradox speedrunnerDec 17, 2022

@compuguy @aburka @Viss according to my org, CentOS 7 is cutting edge

*cries*

Show threadJustin KnoxDec 17, 2022
@pixelnull @compuguy @aburka @Viss @jerry fire turducken. And that’s a follow. Cheers!
Show threadCompuguy 🐱Dec 17, 2022
@jerry @Viss @pixelnull NICE! You got a mini-NOC there 😎
Show threadJerry 🦙💝🦙Dec 17, 2022
@pixelnull @Viss yeah, I will play with it on other systems before curl $randosite |bash on the load balancer
Show threadzeno's paradox speedrunnerDec 17, 2022
@jerry @Viss it's literally in my first install on my systems, i mean in a long chain of other things
Show threadWaLLy3K ​Dec 17, 2022

@pixelnull @jerry @Viss Piping to bash is convenient, but not a great option if you're wanting to do it on a prod box.

https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/

Detecting the use of "curl | bash" server side | Application Security

Another reason not to pipe from curl to bash. Detecting curl | bash serverside.

Show threadVissDec 17, 2022

@wally3k @pixelnull @jerry

curl hax.lol

you can also do it with apache rewrite rules by detecting the user-agent.

Show threadRallias  Dec 17, 2022
@Viss @wally3k @pixelnull @jerry I disclaim any relationship with this website.
Show threadVissDec 17, 2022
@rallias @wally3k @pixelnull @jerry good, cuz i own it :D
Show threadRallias  Dec 17, 2022
@Viss @wally3k @pixelnull @jerry I approve, I just disclaim relationship.
Show threadVissDec 17, 2022
@rallias @wally3k @pixelnull @jerry if you hit it in a browser its a bit more colorful. if your browser is old enough itll play the goat simulator theme :D
Show threadBagheeraAltered ​ Dec 16, 2022
@jerry as always, amazing work
Show threadVissDec 16, 2022
@jerry whoa those are some really interesting spikes. can you tell what they are?
Show threadScott Dec 16, 2022
@Viss @jerry lots and lots of INSERT INTO?
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss Not really. I have neglected to instrument the basic linux stats on the load balancer. That is simply queries that nginx is responding to, so I assume that it is probably related to something that causes lots of other instances to hit the server all at once, like a large account mentioning something here.
Show threadVissDec 16, 2022
@jerry interesting. yeah cuz the 'average load' looks to be considerably less than those spikes, so something is causing.. i guess a shitload of queries to happen all at once, in bursts?
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss exactly. If someone on an instance with followers on thousands of different instances posts a link that references this site, all those instances reach out. This is what the attack of last week was all about - creating many accounts that caused instances to attempt to DDOS some other sites.
Show threadVissDec 16, 2022
@jerry ah ha - illumnation of burst-style DDOS scenarios. too many signups spike psql, psql chokes, front end throws 500s
Show threadthroAUDec 17, 2022
@Viss @jerry probably every time Elon posts on twitter. Lol.
Show threadVissDec 17, 2022
@throAU @jerry well, banning journalists and letting nazis on the site were two big spikes, for sure
Show threadVissDec 16, 2022
@jerry i wonder how soon there's gonna be a patch to replace sidekiq with something else.
Show threadnullagentDec 16, 2022
@Viss @jerry Whenever that is, it wouldn't be too soon for my taste.
Show threadVissDec 16, 2022
@nullagent @jerry right? the biggest improvement to twitter back in the day was when they got the fuck off of ruby on rails
Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss perhaps. Now that I have experience with it, it's not the worst thing in the world.
Show threadVissDec 16, 2022

@jerry so it looks like other folks have figured out how to migrate from sidekiq to rabbitmq https://stanko.io/rabbitmq-is-more-than-a-sidekiq-replacement-b730d8176fb

(we run rabbitmq. its way more burly. we dumped redis to go to this)

RabbitMQ is more than a Sidekiq replacement | Stanko K.R.

A rabbit dressed up as the karate kid - a cross of RabbitMQ and Sidekiq logos I've had gripes with Sidekiq because of which I switched to RabbitMQ. Here a...

Show threadJerry 🦙💝🦙Dec 16, 2022
@Viss interesting - will have to look at it
Show threadVissDec 16, 2022
@jerry I imagine that its not gonna be super easy or straightforward to migrate from sidekiq to rabbitmq, but I would imagine that rabbit will treat your cpus and ram way better than sidekiq does
Show threadDan HonDec 16, 2022
@jerry @Viss oooh nice.
Show threadVissDec 16, 2022
@danhon @jerry this post doesnt have anything to do with mastodon specifically tho - but, you gotta start somewhere :D
Show threadCompuguy 🐱Dec 16, 2022
@Viss @jerry I doubt it. I've followed a guide and spit off the scheduler and mailer queues to a separate container. The rest can be scaled up to meet demand.