Masto.hostNov 6, 2022
Currently there is an issue on the Masto.host network. I'm investigating and will provide more information as soon as possible.
Dave TroyNov 6, 2022
Show threadMasto.host

So, what happened was that one of the web servers (host 2) was struggling and I decided to upgrade it like I did yesterday for (host 1): https://mastodon.social/@mastohost/109292313550726464

Yesterday it was really smooth and the upgrade to the cloud instance only caused 90 seconds of downtime.

Today, everything went wrong:
- the network interface for the private IP changed
- when changing the configuration to the new network interface and doing a network restart, it just stopped responding

Nov 6, 2022 at 7:22pmWeb
Show threadMasto.hostNov 6, 2022

So, I was on the phone debugging the problem with OVH and we were able to bring the instance back online.

This caused the service to be partially down (because host 1 was still running for anyone that was using that DNS configuration) for a little over an hour.

Really sorry about the trouble this might have caused.

Traffic grew 10x in one week and it was impossible for me to predict (or afford) to scale for this before it happened.

Hope you understand.

Show threadWesleyNov 6, 2022
@mastohost no worries, you folks are doing amazing in the circumstances. Keep up the good work and get a break when you can too!
Show threadTanaki (pumpkin spice loaf)Nov 6, 2022
@mastohost thanks for the explanation!
Show threadNasher Nov 6, 2022
@mastohost With the #twittermigration, things are getting ouf of hand for anyone implied in administrating servers. Thanks for the heads up !
Show threadSchmutzie 🦉Nov 6, 2022
@mastohost I’m on your waiting list and completely understand. Who could have forecast such an influx? You’re being as responsive and transparent as possible, and that’s impressive :)
Show threadSalarRahmanian  Nov 6, 2022
@mastohost fyi mentioned you in my latest newsletter https://fosstodon.org/@softinio/109297853373790610
Salar Rahmanian (@[email protected])

Just published my latest newsletter which includes a quick write up on #mastodon and great content by @alexelcu @[email protected] @kev and others! Enjoy and I appreciate you all subscribing to my newsletter! Thanks all. https://notes.softinio.com/p/softinios-notes-on-software-engineering?sd=pf

Fosstodon
Show threadoortvilleNov 6, 2022
@mastohost Thank you for making it so easy and affordable to start up a small instance. Thanks for your efforts to keep up with the #TwitterMigration.
Show threadpantosNov 6, 2022
@mastohost I have rarely experienced a hoster who works so honestly and transparently. Thank you for your great work!
Show threadEricNov 6, 2022
@mastohost Amazed how you still manage to keep everything running with this huge traffic increase, bravo.
Show threadFribygdas Natteravn Nov 6, 2022
@mastohost Wooow, 10x, that's amazing!
Show threadPaul Goldsmith-PinkhamNov 6, 2022
@mastohost thanks so much for being so fast on this!
Show threadjosefinNov 6, 2022

@mastohost Totally understand. The level of downtime we've had is still almost none over the several years we've been on masto.host.

With the week we've had on the network I'd say you're doing great!

Keep it up!
(pun intended 😉 )

Show threadhybrid havocNov 6, 2022
@mastohost appreciate all you do! I like to think that most people are very understanding given how and how quickly we have gotten here.
Show threadPaul Goldsmith-PinkhamNov 6, 2022
@mastohost Thanks a million!
Show threadMieke RothNov 6, 2022
@mastohost thank you for all the work! It’s enormous, the growth!
Show threadBrad G.Nov 6, 2022
@mastohost we totally understand... thanks for all the hard work.
Show threadRebecca NordquistNov 6, 2022
@mastohost
I can't even imagine the week it has been for you all. Thank you so much for the amazing work you are doing!
@miekeroth
Show threadCalligrafaeNov 6, 2022
@mastohost You are awesome and amazing and dealing with a huge stressful time right now and I appreciate you immensely.
Show threadkeeps vis.social runningNov 6, 2022
@welshpixie @mastohost I'd like to second that! Really appreciating you these days.
Show threadJohn ConwayNov 6, 2022

@mastohost Hey, with the explosion you're seeing, I'm amazed things have gone as smoothly as they have. Don't stress it too much.

Heck, early Twitter was down way more than this.

Show threadManu MNov 6, 2022
@mastohost Thank you for your transparency and honesty. Please keep the hard work, the result is great!
Show threadPaquitaNov 6, 2022
@mastohost thank you for your commitment, you're doing a great job 💖
Show threadFacts CentralNov 6, 2022
@mastohost thanks for the information.
No worry sh…t happens
Show threadPhoton Empress 🌸Nov 6, 2022
@mastohost outages are never fun, but explosive growth is a fantastic reason to be struggling with stability. Thanks for everything you’re doing, excited to be hosting with ya!
Show threadBill DayNov 6, 2022
@mastohost I am not one of your users, but I think that it is extraordinary that large instances like yours are managing the explosive growth of the system. I think it would be nice if some of the users with the means to do so would help fund expansion of the network. In the meantime, the people benefiting from your work should chill out a little bit. There are going to be rough spots.
Show threadKeith KursonNov 7, 2022
@mastohost 👏 Of course, great job handling the massive influx of users leaving twitter.
Show threadHal Nordlin Nov 7, 2022
@mastohost If anyone is looking for a temporary home I started https://purpleforest.social but I am doing manual invites to control volume and performance. Please DM if interested.
purpleforest.social

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!

Mastodon hosted on purpleforest.social
Show threadBoutelouaGracilisNov 7, 2022
@mastohost great work 👏
Show thread🇺🇸AJ🇺🇸Nov 8, 2022
@mastohost thanks for fixing it. #twitterrefugee
Show threadsatan Nov 6, 2022
@mastohost

Things like that happends, dont beat yourself.
You are doing a great job
Show threadKarel P KerezmanNov 6, 2022
@mastohost Oh heck.
Show threadFedilab AppsNov 6, 2022
@mastohost
Thank you for being so much responsive.
Show threadChristian Michel Nov 6, 2022
@mastohost Thanks for the information and your Support
Show threadSimon WolfNov 6, 2022
@mastohost Thank you for the transparency and for sorting it all out swiftly.
Show threadAndreas H. Opsvik Nov 6, 2022
@mastohost Thanks!
Show threadEmperorNov 6, 2022
@mastohost Great news. Our https://mas-to-don.be is back online. Now please add our new server so we can grow 🙏
Mastodon

De Belgische Mastodon community. Wees welkom. Kom en vermenigvuldig u.

Mastodon hosted on mas-to-don.be
Show threadCheeky Wee DrewieNov 6, 2022
@mastohost Thanks for the update! Good work.
Show threadDon Goodman-WilsonNov 6, 2022
@mastohost you’re doing great. 🤗 sincerely appreciate the hard work
Show threadJarnoNov 6, 2022
@mastohost thanks for the update and your effort.
Show threadmaureensaysNov 6, 2022
@mastohost glad everything works fine now ❤️ thank you
Show threadAl SuttonNov 6, 2022
@mastohost Sounds rough, but I know you folk are helping onboard a lot of interesting folk via the instances you're hosting, so if there's any help you folk need just put a shout out.
Show threadHubert SablonnièreNov 6, 2022
@mastohost We're on your team Hugo 🤗
Show threadDavid CarrollNov 6, 2022
@mastohost thank you 👏
Show threadDr. "Skinny" Dave Miller, esq.Nov 6, 2022

@mastohost

<joke begin>

Ninety seconds??? That's a lifetime!!!!!

<joke end>

Show threadyuhline Nov 6, 2022
@mastohost champ! Thank you for your work! Sorry we are all suddenly here! ❤️
Show threadJensen HarrisNov 7, 2022
@mastohost Good problem to have though, right?