There is currently an incident, and database servers 11, 22 and 23 stopped responding. Instances using those servers are down. I apologize for the disruption. I'm trying to gather more information and will update as soon as I know more.
OK, the problem has been identified in the data center: https://bare-metal-servers.status-ovhcloud.com/incidents/dgksqhl6lswj
[GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification

Bare Metal Cloud's Status Page - [GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification.

Database server 23 is responding again.
Database servers 11 and 22 still down.
Database server 11 is now also back online. Only database 22 still down.

Latest update from OVH:
"All services in room GRA0404 are now available again.
We are actively working to restore the servers in room GRA0406." - https://bare-metal-servers.status-ovhcloud.com/incidents/dgksqhl6lswj

Hopefully, the database server 22 should be back online soon.

Really sorry for the long downtime.

[GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification

Bare Metal Cloud's Status Page - [GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification.

Another update "Less than five per cent of the services in rooms GRA0406 are still unavailable.
We are working hard to restore the remaining servers in both rooms."- https://bare-metal-servers.status-ovhcloud.com/incidents/dgksqhl6lswj
[GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification

Bare Metal Cloud's Status Page - [GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification.

The incident has been resolved and instances that were affected by the database server outage are back online.

I'm extremely sorry for the long downtime.

This was a data center issue completely outside of my control. You can see the full details at: https://bare-metal-servers.status-ovhcloud.com/incidents/dgksqhl6lswj

Timelines will take a while to catch up but all should be good now.

Please let me know if you experience any further issues. Thanks 🐘

[GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification

Bare Metal Cloud's Status Page - [GRA4][Dedicated Servers] - Rooms GRA0404 - GRA0406 incident notification.

@mastohost

Thank you for the updates and your work on this 🙏

It's clear this was outside your control, no worries 🙂

@mastohost no worries from me, these things happen. Thanks for the transparent updates and no vague waffle.

@mastohost no worries, it wasn't on you

thank you for the updates! 

@mastohost Thanks for the updates. SH*T happens.
@mastohost Makes one wonder what in the world made this big of a mess. Just glad they're working on it.
@greyduck All I know is that it was an electrical problem. No more details.
@mastohost Mobs of electrons causing trouble, going on strike, upending storefronts...
@mastohost Should the checkmarks on https://status.masto.host/ be updating? They all show green despite the outage
@noelle the servers are still sending out the Heartbeat Monitor that's why it's green. Ideally, they shouldn't be green, but https://360monitoring.com/ reports them as up as long as there is a heartbeat being sent.
Sites & Servers Monitoring - 360 Monitoring

Sites & servers monitoring solutions by 360 Monitoring. Keep track of your online business assets with confidence, supported by world-class internal and external system monitoring.

360 Monitoring
@mastohost Ah yes, sometimes I have built a health check which includes a call to database, redis, and anything else essential. No idea about the details of doing that with Mastodon and 360monitoring, though. @noelle
@mastohost I must be on database 23 👀 I'll keep an eye on when it comes back up. 😀
@leetaur That was a typo, it's database 22 that is still down. Sorry about that.

@mastohost Okay that makes sense!

I just checked, and my server is on APP 05, DB 22.

If needed, I can email you my instance's details, but since it is a known issue, I'll just keep watching this thread.

Thanks for keeping us updated.

@leetaur Yes, no need. Sorry for the long wait.

@mastohost thank you for your efforts!

Is there somewhere not on Mastodon you post a status page? I can see your posts from this alt account of mine, but not everyone has one of those.

@mastohost aha just found this:

https://status.masto.host/

Though it is a little confusing to see all green checks 😅

Masto.host - Status

@sara Yes, the servers are still sending out the Heartbeat Monitor that's why it's green. Sorry about that.
@mastohost @sara looks like we're back up, thank you!

@sarajw 🎉 Sorry about the trouble. Thanks.

@sara