jwzNov 26, 2022
Mastodon stampede. "Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops...
https://jwz.org/b/yj6w
Mastodon stampede

"Federation" now apparently means "DDoS yourself." Every time I do a new blog post, within a second I have over a thousand simultaneous hits of that URL on my web server from unique IPs. Load goes over 100, and mariadb stops responding. The server is basically unusable for 30 to 60 seconds until the stampede of Mastodons slows down. Presumably each of those IPs is an instance, none of which ...

Show threadLucyNov 26, 2022
@jwz MariaDB/MySQL cope so bad under high load it's insane. Maybe having some sort of "staticizing" mechanism to snapshot the dynamic content and then serve it through nginx with some fine tuning would help? (compression? connection reuse? cache-instructing headers?)
Show threadjwzNov 26, 2022
@lucent Again, I don't really need you leaning over my desk and saying "You know what you OUGHTA do", thanks.
Show threadAlexander Hamizdat 🌹Nov 26, 2022
@jwz @lucent we're only doing this because it's so shocking. Like, you're a bit of a legend and we did not expect not only for these issues to knock your blog over, but for you to be so salty over it.
Show threadLucyNov 26, 2022
nah, the stress is understandable. @jwz copypasted the same message to all people that replied because everyone was pushing the same solution in different flavours.
the issue of fediverse instances pinging as soon as the post gets forwarded is legitimate, but hard to tackle (i.e. would you trust a pre-crawled preview coming from another server?).
Show threadAlexander Hamizdat 🌹Nov 26, 2022

@lucent right, so if he has a lot of followers, he's on a lot of home feeds so these are at least mostly legitimate. I know I clicked as soon as I saw it.

My problem isn't that he's surprised by a traffic spike, but that he's trying to make it Mastodon's problem. He should own his own setup and stop pretending like 1000 hits is some sort of DDoS. It's ok to say "I got knocked over, I need to consider caching" instead of "I got knocked over, fuck you for visiting" which is how this comes off.

Show threadLucyNov 26, 2022
I get your point of view, but as I said I get also being pissed at stuff crashing because of Fediverse software pinging you back as soon as your post lands on an instance. Fortunately nothing bad happened, I'm not hurt, I apologized for being the "yet another guy that posted the same solution".
Needless to say though, complaining to the bubble without bringing the issue up to either the mentioned software devs or to the W3C pushing for a standard to deal with this situation, obviously makes the whole rant void. Bigger websites or more aggressive setups would easily cope with the average Fedi requests.
Show threadAlexander Hamizdat 🌹Nov 26, 2022

@lucent yeah I mean who among us is above getting pissed off and blaming that stupid hyperactive microservice for our problems.

But as an Sr SRE, I don't have time for recriminations. I assess the situation and find ways *I* can move forward rather than shouting at the clouds. It's up to *me* to answer to why there's no backpressure strategy.

Anyway if you're not blocked, maybe mention the database thing. It's not caching, so maybe it'll be helpful. That's abnormally weak for MariaDB.

Show threadLucyNov 26, 2022
I mean, "mysqltuner.pl" is a search result away from any search engine and for sure points out solutions good enough to counter performance issues *that bad*, I think and hope that it wasn't ignored as a point of failure.
Show threadAlexander Hamizdat 🌹Nov 26, 2022
@lucent hope against hope, amiright? do they still do mysqlbouncer?
Show threadLucyNov 26, 2022
@alexhammy209 probably not. Anyways mysqltuner is still actively maintained (last commit 25 days ago) and supports MariaDB and Percona too, including their specific DB engines. Always worth a shot when fine tuning on a lazy day.
Show threadAlexander Hamizdat 🌹
@lucent oh shoot a bit of googling confirms, mysqlproxy is ⚰️
Nov 26, 2022 at 8:05pmWeb