Good afternoon! We've identified the cause of the instance's slowness: we're facing a bunch of nasty bots 🤖🙅

There's a bot ring that I've identified that's making a lot of unauthenticated requests to the /media_proxy endpoint, causing a lot of load as the server goes out and fetches, resizes, etc. the media that's being requested. This endpoint does not require authentication, so anyone on the internet can do this.

Unfortunately, the bot ring is rotating through cloud IPs quickly; each individual IP only makes a few requests, bypassing the rate limit built in to Mastodon (30 requests per 10 minutes per IP).

I'm going to work on a fix here (keeping the exact specifics quiet to prevent the bots knowing what we're looking for), but it'll probably take a few days. I'll follow up with another post once the mitigation has been deployed.

-- @andrew

#mastoadmin

Good morning all! We just deployed a small update to the server to hopefully help address some of the latency that people have been seeing. I also fixed some of the monitoring, so if it happens again I’ll have more data to see what’s going on. Please do continue to report issues you see; it’s never a bother 😃

— @andrew