Ottawa.place AdminFeb 26

Good afternoon! We've identified the cause of the instance's slowness: we're facing a bunch of nasty bots 🤖🙅

There's a bot ring that I've identified that's making a lot of unauthenticated requests to the /media_proxy endpoint, causing a lot of load as the server goes out and fetches, resizes, etc. the media that's being requested. This endpoint does not require authentication, so anyone on the internet can do this.

Unfortunately, the bot ring is rotating through cloud IPs quickly; each individual IP only makes a few requests, bypassing the rate limit built in to Mastodon (30 requests per 10 minutes per IP).

I'm going to work on a fix here (keeping the exact specifics quiet to prevent the bots knowing what we're looking for), but it'll probably take a few days. I'll follow up with another post once the mitigation has been deployed.

-- @andrew

#mastoadmin

Show threadWestFox35
@OttawaPlaceAdmin @andrew would you mind sharing more of your media proxy setup? Is this a low end hardware deployment or something more substantial?
Feb 26 at 8:03pmWeb
Show threadAndrew DunhamFeb 26

@WestFox35 @OttawaPlaceAdmin This whole instance runs on a decently beefy machine (8 core Xeon E5-1620 v2, 64GB RAM). The issue with this particular bot swarm is actually that the `/media_proxy` endpoint causes the Puma web worker to perform the fetch and imagemagick work, inline in the worker, instead of delegating to Sidekiq. So, you end up tying up all of the processes/threads shelling out to Imagemagick / ffmpeg / etc. instead of serving web requests.

This is compounded by the fact that Imagemagic is (by default) multi-threaded so it ends up contending on CPU to transcode all the media files. I fixed that with the Imagemagick policy.xml file limiting each to 1 thread, but the general problem of request starvation can still happen.

Show threadWestFox35Feb 26

@andrew @OttawaPlaceAdmin oh. This is a different situation than what I was thinking of.

Can you not block their ASN/IP range?

Show threadAndrew DunhamFeb 26
@WestFox35 @OttawaPlaceAdmin Unfortunately not, since that'd also block everyone that runs a Mastodon instance on e.g. DigitalOcean (one of the ASNs doing this), since they also need to be able to `POST /inbox` for federation to work 😒
Show threadWestFox35Feb 26
@andrew @OttawaPlaceAdmin good luck!