Holiday memories

We used to have a lot of physical devices on our network*. Servers, firewalls, file-shares, staging servers, development machines… all sitting on the network with their hard drives endlessly spinning, spinning, spinning!

System administrators are fond of referring to platter-based hard drives as “spinning rust,” partly as a reference to the ferro-magnetic iron crystals that store the actual data, but also to remind us that it’s always decaying and corroding. Over time, drives start generating errors or becoming unreliable. When we had physical devices that exhibited issues, we’d yank the hard drive and replace it. Over the years, we’d accumulated a pile of a dozen or more drives that were unreliable or bad but still contained data.

The data is not especially sensitive, but there could be stuff that could be abused or belongs to other parties. There may well be meeting notes, source code, sample data files, or there could be cached passwords or other credentials. It’s not worth just hoping it’d be OK to release to the world. So it’s a chore to render this data unreadable.

Pulling apart spinning platter hard disks is humbling. These are incredible little devices, with incredibly precise machining and elegant engineering. Going through a pile that spans a decade, you can actually see the improvements in technology: new vibration damping systems, different head-parking strategies, traps for dust, and more. I see these parts, and am inspired by the craftsmanship that goes into them.

So in the spirit of admiration, I offer these (hopefully unreadable) holiday memories.

* Now, of course, we have few physical devices but all those same services are implemented on “the cloud.” This means that someone else has physical devices somewhere, with their hard drives (or SSDs) endlessly spinning, spinning, spinning (or trimming, trimming, trimming).

Solving a VPN Mystery

The Department of Water and Power is doing work near the office, and over the weekend, there was a sustained power outage. I came in Monday to shrieking UPSes and had to power up the firewall and a few other machines. It was the normal stupid kind of stuff.

We have a few virtual servers out in “the cloud,” and we use point-to-point VPNs to make them seem local to our network. Those VPNs also needed restarting.

Through the course of the day, however, one VPN connection kept unceremoniously disconnecting. Looking at logs on the various servers was unenlightening. Everything was running normally, other than the surprise disconnects.

In the evenings, I’ve been watching the old Grenada TV/Jeremy Brett Sherlock Holmes series, so I had to apply Holmes’ deductive process. The virtual servers had experienced no changes except being disconnected, so I needed to focus on the firewall. The firewall had experienced no change, except being restarted. What could have happened?

I finally found a configuration that was incorrect (it was a netmask that was insufficiently restrictive, allowing devices not on the VPN to collide with VPN IP addresses). I fixed the netmask, and the VPN has been up and stable ever since.

But how could this be? It had been running properly literally for years. It had to be something to do with the power outage. But if that had corrupted the configuration, it wouldn’t have been a single IP netmask changing. “[W]hen you have eliminated the impossible, whatever remains, however improbable, must be the truth.” The bad configuration file could not have been in use.

The best theory is that the configuration file had been (accidentally?) modified at some point in the past, but never loaded. When the firewall was restarted, it loaded this modified configuration for the first time.