Having some server issues tonight. I'm aware of it and looking into it!

Feels like a host networking issue to me so far.

I've found the issue. My hosting company, Akamai (Linode), decided to forcibly migrate my primary database to another server without notice.

I apologize for the downtime. Looking into mitigations.

Pulled the ripcord and promoted the replica to primary.

Downside: some work and a bunch of time to set up a new replica

Upside: everything should now be working again immediately

Still pretty upset at Akamai for doing this without warning. All other recent migrations have been performed with advance notice and maintenance windows with early-fire options. I don't know what happened here — with my MOST important server.

@overcastfm Seems to have fixed my issues! Thanks for the quick fix, sorry they boned you.
@overcastfm seems like a gross oversight on their part for not giving you a warning—but appreciate your quick action
@overcastfm PLEASE talk about it on atp
@overcastfm I hope they had a hardware issue otherwise that’s brutal.

@overcastfm seems odd as Linode are normally pretty good about stuff like that, we’ve just had a month plus notice that a box would be restarted on a specific date so we could schedule ourselves if we wanted to, they flag it in the control panel too.

Unless this was unplanned?

@overcastfm sounds like an oopsy by one of their ops folks? "We kinda need to do maintenance here but there are people's servers doing bits and bytes... mhh.. let's just roll and hope nobody notices it"
@overcastfm on the flip side, as an infrastructure engineer, it’s mind boggling to me how solid things usually were with just a single DB node. I miss those simpler days 😅
@overcastfm hug ops! Hope it’s not too difficult
@overcastfm Curious - were you referencing IPs or sub-domains?
@overcastfm I had one virtual server be forcibly migrated due to a host issue last week, but that’s a standard virtual server. Not sure what the db setup is for overcast.