VegalypFeb 2, 2023

As I look fondly back on some time spent at my previous job, it is time for the fourth round of Discoveries. Relevant tags include #Networking #Infosec #sysadmins

Discovery 19: When I first started, the first solo task I was told to do was configure alerts on our servers and network hardware. SNMP traps were not enabled anywhere in the domain and hardware alerts from hardware monitoring (iLO, iDRAC, etc.) were not turned on.

I turned on hardware alerts for the domain and eventually configured SNMP traps to a shitty SNMP-to-email script I wrote since I was not authorized to build a server yet.

Later on, I built LibreNMS and it was an absolute pleasure to use. And it was way better than the terrible script I wrote.

Show threadVegalypFeb 2, 2023

Discovery 20: I don't know how I forgot about this, but we had redundant internet connections. One day our primary internet connection went down and the backup never kicked in.

My boss worked on that for a few hours and then got busy with something else once the primary connection was restored. I later realized VRRP was not actually set up. The agency had bought the connection but never actually did the work to enable the protocols for the redundancy.

I quietly fixed that one on a weekend because I couldn't believe that for 5 years (Long before I joined), the redundant connection was being paid for, but that it wouldn't actually work in the event of the primary connection dying.

Show threadVegalypFeb 2, 2023

Discovery 21: Rogue DHCP servers. The first month I was on the job, we kept having internet loss at our primary outpatient facility. As a total noob, I couldn't figure it out and finally called my boss to take a look.

An employee had been denied the WiFi password for her phone, so she brought her own router and cables from home to connect her device.

You can see the issue. She was let go the next day after a talking to from my boss and HR.

That day will be forever burned into my brain.

Show threadVegalypFeb 2, 2023

Discovery 22: Local government and Windows 7. My agency finally started upgrading to Windows 10 in 2020. I was in charge of building the golden image. I actually really enjoyed building it.

What I didn't enjoy was then being in charge of doing that to 600 devices.

We started the biggest project in agency history during a pandemic and I was the primary on it. My secondary, J, was an absolute rockstar and we got through it by the beginning of 2021, although we were both absolutely burnt out.

I still remember pulling 12 hours with her and us stumbling out of the building, ready to go home. As stressful as those days were, we were a rockstar team and we still talk fondly of being the backbone of the department for that whole year.

Back to Windows 7 though, we got off of it in early 2021 and we were happy about being done but disappointed that our agency was behind the curve.

Turns out we were the first agency of our kind in the state to be on Windows 10.

When I left in November 2022, there were still 3 agencies like ours only halfway through their upgrade projects. Godspeed to my fellow overworked IT professionals in local government/government funded healthcare.

Show threadVegalyp

Discovery 23: Spanning tree protocol.

I was the manager/team leader now, but still a noob at this point. The network at admin went down and all our servers came to a screeching halt. A power loss had taken out the admin building during a pretty nasty storm.

The initial network loss was due to prolonged power loss taking equipment down. As power was returned to the building (We had exhausted the battery backups to the building already), the equipment began turning on again.

Confident, I told my team everything was working.

However, because the WiFi had not turned on yet, one of my techs who was struggling to learn core IT concepts decided there must be something wrong and grabbed the spare patch cables we kept in the demarc room and connected all the switches together in a loop.

Although the network had been built long before me, I was still the manager. For my part, I had not enabled STP and this was entirely my fault.

But it took me more time than I'd care to mention to realize what happened.

Feb 2, 2023 at 3:36amWeb
Show threadVegalypFeb 2, 2023

Discovery 24: Risking my life to make the voices stop.

(Not schizophrenic voices, just the voices of managers that complained about everything. I actually overheard them complain to the head of Purchasing that the blue pens they ordered via requisition were 'Too blue'.)

At our agency, the IT department/team was responsible for running lines and installing cameras. At one of our sites, we had a program manager who would ask for more cameras every quarter. In a 2200 square foot facility, she had 62 cameras.

But 'situations' kept happening that required cameras. Her claim, that management backed up, was that cameras would prevent the incidents from happening, despite my repeated objections that cameras are useful after the fact only.

Management always agreed with her though, so we ran lines for another 9 cameras. But since I was short staffed at the time, I was able to convince management to hire some contractors to help.

There was just one problem: The attic at this facility was filled with asbestos.

So here I am, with contractors who had not been informed by our CEO, who negotiated the contract, that the attic was filled with asbestos. And I had been ordered in no uncertain terms to finish it that day.

So I donned full respirator and disposable overalls and went into attic.

This was in Southwest Georgia during the summer. The thermometer mounted at the top of the ladder that lead to the attic read 130 degrees Fahrenheit or 54 degrees Centigrade.

The area I needed to get to to help run lines required navigating over duct work, crawling under pipes, avoiding exposed nail heads sticking up from nailed frame boards (Thank you work boots), and avoiding falling in that lovely asbestos.

I was up there 60 minutes sweating my ass off, unable to drink water because of the respirator and asbestos, while running lines from one end to the contractors below me who were frantically working to minimize the time I spent up there.

To their credit, they were awesome and were having to deal with the awful cabling situation below the attic, so they earned their pay.

Back to the attic, I had finished running the lines and was navigating out of the attic. I then got stuck under one of the pipes.

In the dark heat, I had a brief urge to fall asleep. But I realized I would die if I did that.

I had to wriggle free and in the process lost my lanyard with agency ID, and keycard. I also briefly had the respirator come off of my face just to get free.

By the time I had gotten out of there, I couldn't see out of my periphery and was not sweating anymore. One of the contractors literally carried me to one of the showers and turned it on and ran to get me something to drink.

There I sat for a full 15 minutes. I had just risked my life because this program manager was so annoying and insufferable that I was willing to do anything just for peace and quiet from her.

When I finally walked out from the shower in my soaked work clothes, the program manager walked up to me and asked why I was slacking off and that I had a deadline to meet.