Mastodawn

So - I started work on moving our big database off of the cloud at around 11PM MST. It took some time to pull down that 450GB database, and it took a little longer than expected to restore that backup into our new database. As a result, our maintenance window was extended by three hours.

Once restored, we worked through some performance issues. This was expected as while we can test lots of things in a vacuum, sometimes without the real world load it's tough to know what to expect.

Once those performance tweaks were made, I saw some immediate improvement and we're now back and even better than before.

We're still working through the backlog of activity from the fediverse that we missed while we were down. It will probably take a few hours to catch up, so you'll see statuses from an hour or so ago popping up late. We're moving in the positive direction on this!

I've been looking forward to writing this for a long time: WE ARE FINALLY COMPLETELY OFF THE CLOUD. 🥳

#mstdnca #MastoAdmin

Also - congrats to our high scorers on our down page. 😉

I hope the mini-games helped! We'll keep and use this down page in the future.

Zazzoo 🇨🇦Feb 24

@chad Awww damn, I missed the downtime!

When's the last time a user's said that to you, Chad? 😂

@zazzoo You may be the first :)

Zazzoo 🇨🇦Feb 24

@chad Ha, no doubt! And congratulations on the successful transition!

But while i have you, notifications seem to be a little wonky. I'm not getting them when I'm tagged in a convo. I do get 'like/boost' notifications though. Maybe this is something that will work out over time? When the sever was having issues last time it seemed to take a day or two to 'settle in' - maybe playing catch-up on federation tasks or something? Okay, I'll stop pretending to understand how Mastodon works on the back end now.

@zazzoo still doing catch up. I don’t expect we’ll be 100% caught up until around 4pm MST. Then after we’ve had a full day on the new DB we’ll assess performance again.

We’ve already made changes to some backend processes we noticed were a little too noisy.

Zazzoo 🇨🇦Feb 24

@chad Well looks pretty damn good now. You are a wizard, sir! A cloud-free wizard! 😂

Justin M Feb 25

@chad seems pretty responsive so far

javensbukan Feb 24

@chad Wooo! Amazing! 😍 🎉

Great work!

Jorijn Schrijvershof Feb 24

@chad nice Chad!

@jorijn thanks for all your help - really couldn't have done it without your advice.

Jorijn Schrijvershof Feb 24

@chad no problem at all Chad. Happy someone found my tech ramblings useful!

Dave Sanderson 🇨🇦🍁Feb 24

A toast a successful transfer! 🍾🥂

David Haigh Feb 24

@chad Congratulations! No small feat!

I keep thinking there's a deeper lesson here for all of us when it comes to #cloud vs on premise hosting...

Alex Block 🇨🇦🍁🇨🇦Feb 24

@chad Well done! You must be much relieved... An epic adventure happily now in the past. Looking forward to a more reliable and responsive experience going forward.

@alexblock So. Relieved.

nigelsezyarr Feb 24

@chad Hip hip huzzah! 🎉🎉

@chad congrats! nice accomplishment! 👌

Noah Bailey Feb 24

@chad thanks for your hard work skipper!!

Francis Pell Feb 24

@chad Good job !! Thank you for your hard work

@chad Congratulations. Not an easy task for sure. I appreciate this as a fellow instance admin

Travis 'Nep' Smith Feb 24

@chad you’re my hero

James Baker 🍁Feb 24

@chad beauty, eh?

Dustin Hudyma Feb 24

@chad Feels good to be off the corporate cloud, Chad and the mstdn.ca team! 🎉

Mags at Large Feb 24

@chad Thank you for all you do!

Lori M Olson Feb 24

@chad great work!

Mark Connolly 🍻 🚴🏼‍♀️ (he, him, his)Feb 24

@chad Congratulations to you, and all the folks who worked with you to get to this fantastic milestone! 😀❤️

FWIW, I’m seeing a huge boost in performance right now as I look at my timeline. 😀❤️

@chad congrats on this achievement!

Adam on MSTDN Canada Feb 24

@chad Wow! it is fast! Hope you're able to enjoy that celebratory beer.

Gull (Thomas) 🇨🇦Feb 25

@chad Congrats, and thanks for your hard work! The performance boost is magnificent.