Post your IT redundancy tales here

https://awful.systems/post/2392943

Post your IT redundancy tales here - awful.systems

I’m in our daily standup and it’s turned into exchanging fucked up sysadmin redundancy tales. One place I worked lost a machine room. They’d fired people so fast that nobody remembered where the boxes were any more. I knew, but they didn’t ask me. Oh well! The cycle of IT binge and purge is eternal. Post your tales here.

I have two stories.

Company X: Our testbed server room was supported by redundant rooftop AC units, many yards apart. During a storm, a lightning bolt forked (split) One tip.of the bolt hit AC unit one and the other hit AC unit two, killing both cooling units. To make things worse, the server manufacturer did not add a temperature safety shutdown to the units and instead configured them to fan faster the hotter they got. By the time I got there the cable management was warping and melting due to heat.

Company Y: The main datacenter was on tower 2 and the backup datacenter was on tower 1. Most IT staff was present when the planes hit.

Company Y: The main datacenter was on tower 2 and the backup datacenter was on tower 1. Most IT staff was present when the planes hit.

Well, that one dark “redundancy” story…

I don’t understand why they had redundancy so physically close.

Whatever affects one has a high risk of affecting the other.

Different regions is a thing for a reason.

It’s probably good to situate in time when thinking about these things. The twin towers were how a lot of companies became examples of what location redundancy really means. These days people are keeping that lesson well in mind, but back then, not so much.