Despite multiple systems going into thermal shutdown, it sounds like the only casualty from last night's "excitement" was a single fan.
| Blog | https://nointerrupts.com |
| Gitlab | https://gitlab.com/uid_zero |
| Github | https://github.com/jjbaumgartner |
Jeremy Baumgartner
- 82 Followers
- 36 Following
- 318 Posts
Jack of all trades sysadmin (primarily Unix) with interest in security.
Well that was a shit show, but it could have been so much worse.
A chiller failed, and the backup didn't kick in. It seems like we turned off enough non-production and redundant systems that we staved off significant hardware failure, and had zero impact to business services.
Not half bad for a bunch of sleep-deprived assholes.
Tomorrow is going to be interesting, though.
Currently on an emergency work call...we have servers reporting temps in the 140-150 degree F range...
And it took several hours to get in contact with facilities to be told the HVAC isn't working at the datacenter.
No shit, Sherlock.
Happy "NetScaler patching day" to those who celebrate...
Dear vendors,
Sending your "We're making changes to your account" emails at 4:50pm on Friday is universally seen as a dick move. It's like you want it overlooked.
Normally I understand the desire to be overlooked. But in this case, you're improving account security by asking people to move to SSH keys and whitelisted IP ranges. THAT'S A GOOD THING and you should NOT try to hide it.
Sincerely,
People with a fucking brain
For those of you keeping track at home, that's national championship number 9 for the Badger women.
And because context is important, that's 9 championships since the women's hockey tournament started in 2001. That's 36% of these tournaments.
Ivan Ožić BebekThe Slow Death of the Power User https://fireborn.mataroa.blog/blog/the-slow-death-of-the-power-user/
ON WISCONSIN!
Badger women's ice hockey beats Ohio State for the second year in a row!
Today I have:
- (finally) changed the address on my wife's life insurance policy
- registered the dog for the year
- paid for the yard waste site sticker
Is that enough adulting for one day?
I really hate when I forget to try the easy/quick fixes.
I just spent 45 minutes troubleshooting Ansible connection issues to servers.
First one - someone removed the user from the local admins group on the server. Why, I have no idea.
Second one - I just had to run 'winrm quickconfig' to resolve the access denied issue.
Literally took 5 seconds to fix each one, but I went down rabbit holes first.
Horses, not zebras...