Hello all, toot.garden is back online, I had a fun few hours of trying to figure out what the hell happened.

Essentially, for some bizzare reason, the I/O on the Hetzner dedi just *stopped* and because of this, it triggered all the remote monitors. I tried to log in, no success. I tried to ping the server, no success. I then tried to reboot it via Hetzner's Robot service. Still nothing.

Finally, after entering the rescue system, I went through the logs of the disk, and found that it had just died. No reason except for a non-descript I/O error. From there, I confirmed my suspicions and found that there were no successful reboots. Fast forward 30min, Hetzner support gives me KVM access. Sure enough, the disk had errors. They were fixable, but it was still rather odd. It was expecting more, and also less free space, and had random other errors...

I have contacted Hetzner again to ask about possible disk failure, and for now I will monitor it closely...

#mastodon #mastoadmin #downtime #hetzner #linux #networking

@linkeddev I typically understand about one word in ten of your tech posts, but appreciate so much the labor of love you’re continuing to engage in with this server. Thank you! 💙