I’m on a vacation, away from my homelab for almost 13 days now. On Feb 16, I suddenly lost all connection. No SSH, no monitoring, nothing. Most likely a power outage. My UPS can hold for ~40 minutes. After that, it’s game over.

(1/5)

#mastodon #fediverse #homelab #selfhosting #horror #mastoadmin

I live alone. The room was locked. There was literally no one who could check the server physically. I called my ISP thinking maybe a fiber cut. They didn't do anything despite I'm mentioning that last time one of their employee unplugged my cable and left it that way while working for another customer. So all I could do was wait… and wait.

(2/5)

Those 10 days were frustrating. You know something is wrong, you know it’s probably simple, but you’re powerless. This is the downside of running everything yourself from home.

(3/5)

What saved me is backups. I had the database and files safely stored in S3. Without that, this would’ve been a full wipe. Today I finally got back, restored everything to a Contabo VPS, and brought the instance back online.

(4/5)

Lesson: Backups and Redundancy aren't optional

#matodon #backups #linux #homelab #selfhosting

@ml

Everything can be a datacenter

@lasagne I didn't have enough resources to make the setup more redundant 😔