QST: I'll be shutting down mastodon.hams.social in the next hour or so (around 2023-06-28 11:00 UTC-7) to increase it's available disk space. It should only be down for a few minutes. Please follow here (@smitty, on a different host so not affefcted by the shutdown) for updates.

Please Boost for visibility.

The process I thought was going to be done around 11am had another step after that process. So it'll be later than originally planned.
Ugh. Another hour. Sorry folks..
Ok, NOW I'm shutting down mastodon.hams.social. Sorry for the delay.
Still working on this. Running an fsck before extending the FS now. Stand by.

QST: mastodon.hams.social is down right now for emergency maintenance. It ran out of diskspace. I'm presenting upgrading the VM to increase available disk space, and the e2fsck before running the resize2fs is taking .. a long time.

Follow this thread for status updates.

We are truly spoiled by SSDs, aren't we?

FS resize happening now.

In other news, I am not a fan of LVM... Never have been.

QST! Mastodon.hams.social is back up and running! Thank you for your patience! Check back on this thread for a full timeline of what happened.

This should address the "500 Server Error" problems we've been seeing for the last week or two.

Please boost for visibility. Thanks!

@smitty I have renewed my subscription for another month! 😉 Really appreciate your making this available, keeping the environment pleasant and dealing with LVM. Thank you!

@smitty
We're pretty spoiled by such reliable filesystems, too. At our shared previous employer, I discovered that at least one of our platforms didn't have enough memory to fsck it's filesystem.

If the fs ever bought it -- and it did consistently ALSO because the disk was too big -- you were pretty much doomed.

It was a fun discovery.

@mdwyer Lovely. I'm impressed that I never ran into that sort of problem. I was dealing with a system that ingested .5PB/day (you read that right). Granted, not to a single FS, to a cluster of a few hundred servers. But still... I continue to boggle at that scale...

@smitty
Actually, now that I think about it, I think the problem was in the LVM layer...? Nearly all the other customers didn't use them that way. It was only the one who stitched all the drives together into one giant volume and were still using the old OS who had problems. They were storing security video.

I'd be lying if I said I didn't miss the excitement of that scale.