Good afternoon! We've identified the cause of the instance's slowness: we're facing a bunch of nasty bots 🤖🙅

There's a bot ring that I've identified that's making a lot of unauthenticated requests to the /media_proxy endpoint, causing a lot of load as the server goes out and fetches, resizes, etc. the media that's being requested. This endpoint does not require authentication, so anyone on the internet can do this.

Unfortunately, the bot ring is rotating through cloud IPs quickly; each individual IP only makes a few requests, bypassing the rate limit built in to Mastodon (30 requests per 10 minutes per IP).

I'm going to work on a fix here (keeping the exact specifics quiet to prevent the bots knowing what we're looking for), but it'll probably take a few days. I'll follow up with another post once the mitigation has been deployed.

-- @andrew

#mastoadmin

Good morning all! We just deployed a small update to the server to hopefully help address some of the latency that people have been seeing. I also fixed some of the monitoring, so if it happens again I’ll have more data to see what’s going on. Please do continue to report issues you see; it’s never a bother 😃

— @andrew

Hello instance members! A question from your friend local admin team here (@andrew and @leigh) about updates to this instance.

As some of you have probably noticed, we're still on version 4.2 of Mastodon, because that's the most recently-released version of the "Hometown" Mastodon fork that we're currently running. Unfortunately, the fork hasn't seen updates recently, and I don't see that changing soon.

As such, I wanted to check with folks here on the instance about what y'all would prefer as the path forward! I think we have a few options:

1. Keep running Hometown, and wait for an update. Upside: very little change in your day-to-day, downside: no new features (like quote posts), security updates require manual work from the admin team.

2. Switch back to the mainline Mastodon software. Upside: reasonably easy to do, we track upstream so security and feature updates are easy. Downside: no local-only post option, a few other smaller features / changes from Hometown also go away.

3. Switch to a different Mastodon fork (e.g. the glitch fork). Upside: keep local posts, along with newer Mastodon features. Downside: the most technically difficult migration (which will also almost certainly require extended downtime), we'd still be on a fork of Mastodon with no guarantee of ongoing maintenance.

4. Something else I haven't thought of!

(also if we do option 2, we'd need to decide what to do with local-only posts; I could delete them, change the visibility on them, or something else entirely)

I'm going to open this up to everyone here as a poll, open for a week. I'll probably re-run the same poll later to make sure we don't miss anyone, too. Please feel free to leave comments, suggestions, and more in the replies here.

We had a small blip with the instance's database just now; it recovered by itself, but I've made a change that should prevent it from recurring. Please let us know if you see any issues!

@andrew