Chad Feb 24

So - I started work on moving our big database off of the cloud at around 11PM MST. It took some time to pull down that 450GB database, and it took a little longer than expected to restore that backup into our new database. As a result, our maintenance window was extended by three hours.

Once restored, we worked through some performance issues. This was expected as while we can test lots of things in a vacuum, sometimes without the real world load it's tough to know what to expect.

Once those performance tweaks were made, I saw some immediate improvement and we're now back and even better than before.

We're still working through the backlog of activity from the fediverse that we missed while we were down. It will probably take a few hours to catch up, so you'll see statuses from an hour or so ago popping up late. We're moving in the positive direction on this!

I've been looking forward to writing this for a long time: WE ARE FINALLY COMPLETELY OFF THE CLOUD. 🥳

#mstdnca #MastoAdmin

Show threadChad Feb 24

Also - congrats to our high scorers on our down page. 😉

I hope the mini-games helped! We'll keep and use this down page in the future.

#mstdnca

Show threadZazzoo 🇨🇦Feb 24

@chad Awww damn, I missed the downtime!

When's the last time a user's said that to you, Chad? 😂

Show threadChad Feb 24
@zazzoo You may be the first :)
Show threadZazzoo 🇨🇦

@chad Ha, no doubt! And congratulations on the successful transition!

But while i have you, notifications seem to be a little wonky. I'm not getting them when I'm tagged in a convo. I do get 'like/boost' notifications though. Maybe this is something that will work out over time? When the sever was having issues last time it seemed to take a day or two to 'settle in' - maybe playing catch-up on federation tasks or something? Okay, I'll stop pretending to understand how Mastodon works on the back end now.

Feb 24 at 4:52pmWeb
Show threadChad Feb 24

@zazzoo still doing catch up. I don’t expect we’ll be 100% caught up until around 4pm MST. Then after we’ve had a full day on the new DB we’ll assess performance again.

We’ve already made changes to some backend processes we noticed were a little too noisy.

Show threadZazzoo 🇨🇦Feb 24
@chad Well looks pretty damn good now. You are a wizard, sir! A cloud-free wizard! 😂