Improbable IslandFolks we have a dead hard drive, we're gonna be slow or down throughout today.
There's two hard drives in our server (well I guess they're not hard drives anymore, they're those new solid-state thingies the size of a stick of RAM that go super fast) and everything goes on both, so when one breaks I get an email saying yo one drive broke and then I take backups of everything and swap out the dead drive for a new one and copy everything back over to the new drive. It takes a while
Disk swap request sent, now we wait.
Disk swapped out and the system didn't boot, so I guess I'm not going anywhere tonight lol, if you know any Islanders kindly spread the word to them
Gonna try and have things back up and running this evening, I'll keep you all informed as best I can
Resilvering the new drive now, this'll take a hot minute. I get to watch a terminal making a progress bar with [======>...] like a balaclava hacker boy
K now I gotta mess with the bootloader, but I'm doing it in an unfamiliar environment and in kinda a fucky way since the machine couldn't boot, so I gotta familiarize myself with how to do this
Yeah nope we're down for the night, there's something else screwy going on and I know better than to try to fuck around with bootloaders this close to midnight
Good morning, coffee's on, kid's dropped off at school, I'm diving back in
fucking efi partitions and bootloaders and did I even do this right the last time we had a bad hard drive ffs, moving to managed hosting is looking more and more attractive every day
We've reached the stage where I'm recruiting my IRL techy friends to come and gawp at this shit
Like when you go to the doctor for what you think is just a little mole or something and they go "I see, yes, I'm gonna get a second opinion" and then they bring in ten more guys who drop their coffees and shout FUCK ME LOOK AT THAT
There was a thing at the pinball place we called the Tech Black Hole, where you see a guy next to an open machine scratching his head so you go over and see if he needs any help and half an hour later there's four guys around the open machine scratching their heads
Still working on it, breaking for dinner
Yeah it's 11:30 at night and I rrrrreally don't wanna go with Plan B, emergency server migration and restore from backup, but it's looking increasingly likely
I've been smoking WAY too much these past few days
Current theory is that this happened:
* we had two drives, both mirrored /, /boot and /swap and had not-mirrored efi partitions
* at some point, probably while I was doing hard power-cycle resets during the AI slopbot attack 'cause nothing else would get through, drive A got a fucked up efi partition
* the machine kept booting fine off drive B
* drive B died and was replaced with drive B2
* I copied fucked table from drive A to drive B2
ya this partition is fucking hosed
both of them, both efi partitions are superduperturbofucked, one drive has logical sectors not the same as physical sectors and the other drive's all "what's a sector," what a week
I was gonna make May's monthly memento something May Day related but I might have to make it a superduperturboenfuckulator
It's just a flesh wound
02d00h│Admin CavemanJoe says "Down we go folks, see you on the other side!"
01m00s│Admin CavemanJoe says "WELL THAT FUCKING SUCKED"
01m00s│Admin CavemanJoe says "WELL THAT FUCKING SUCKED"
Alaric Snell-Pym@ImprobableIsland I live in fear of things like that going wrong on my servers - I don't know if I could recreate partition tables and boot metadata under pressure - so I back that up as well as the filesystem contents.
I'm sure this is a super helpful thing to hear right now 😇
@kitten_tech oh I did, believe me I did, I've got several backups of partition tables, but nonetheless here we are
Jon Gilbert
Lantha333@ImprobableIsland what does that mean? Is the island going to have losses?
@Lantha333 Won't mean owt to you beyond a bit more of a wait, but it'll be a fucking pain in the arse for me
DianeOh no! that sucks :(
kestrel_ii@ImprobableIsland massive sympathies & it's very, very much appreciated!
Liena@ImprobableIsland May your coffee be strong this morning, CMJ. Thank you for all that you do.