Keith AlexanderJan 1, 2024
Bob Young

New Year’s Eve: Musings on Y2K
At 3pm PST on 31 December, 1999, I sat down at the computer in my home office in Yakima, Washington. I logged remotely into the network at HQ and started monitoring our systems. The most critical moment would come at 4pm local time. We were in Pacific Standard Time (PST), -0800 UTC. In other words, at 4pm in Yakima, it would be midnight in Greenwich, England, where the time zone aligns with Coordinated Universal Time. (Coordinated Universal Time is abbreviated as UTC, not CUT, because there are actually other languages in the world besides English, and… never mind. Look it up if that story interests you).

Anyway.

The GPS satellites run on UTC, and our entire multi-state operation depended on GPS timing. My first hint of system failure because of a Y2K bug would occur at midnight, UTC.

Beginning at 3:55pm I began testing the major system once a minute. At 4:05pm I sent out the notice to corporate management that all was well.

I tested hourly, then, but the next critical moment wasn’t until 9pm PST, which was when midnight occurred on the US East Coast. Our equipment was all in MST and PST, but some of our many telecom providers might have systems with local time coordination in some other US time zone. (They’d all be using GPS now, but – this was 1999, and US telecommunications had plenty of legacy systems with other clocking methods).

In the end, nothing failed. Our entire system worked.

This wasn’t because Y2K was overblown.

It was because we replaced our billing system, which wasn’t able to generate an invoice after the date flip.

It was because we did software updates on several proprietary systems that would have failed.

It was because we did firmware updates, too.

Equipment inventories.
Application inventories.
Operating system inventories.
Software version inventories.
Firmware version inventories.

The reason January 1, 2000 seemed like such an ordinary day is because of the MASSIVE amount of work and money spent to make it ordinary. There are unsung heroes around the world who put in the work to update or replace systems that would’ve failed otherwise.

If you’re one of those people, I would love to hear your story.

#newyear #y2k #informationsecurity #gps

Dec 31, 2023 at 8:53pmWeb
Show threadthe roamerJan 1, 2024

@fifonetworks

That's a great read, thank you. The Y2K meltdown didn't happen because institutions spotted the issue in good time and did the work.

It's so annoying to see people treat the issue as a "hype".

We observe developments that, if unchannelled, will culminate in a catastrophic event. We take action to alter these developments. If the event then does not happen, we have succeeded. There was no hype --- there was foresight paired with preventative action.

Show threadJody HughesJan 1, 2024
@the_roamer @fifonetworks Yes. The ‘over-hype’ reaction to Y2K is akin to the antivax one: because we don’t see lots of measles, mumps, rubella, etc, the vaccines we are given are unnecessary.
Show threadthe roamerJan 1, 2024

@Gaolaitch

Exactly!

Show threadMartin Vermeer (not Jesus)Jan 1, 2024
@the_roamer @Gaolaitch It is akin to lots and lots of denialisms. Like, there never was an ozone-hole problem, when in reality it was recognized as a serious threat, and addressed.
With 'luck', climate change will rudely remind us that recognizing and addressing such threats is not a luxury. In fact, it is already doing so.
@JohnMashey
Show threadNicole ParsonsJan 1, 2024

@martinvermeer @the_roamer @Gaolaitch @JohnMashey

A similar set of preparations needs to occur to shore up the election process for democracy.

The unsung heroes who cut off internet access in 2018 to the Russian Internet Agency, is just one example.

https://www.washingtonpost.com/world/national-security/us-cyber-command-operation-disrupted-internet-access-of-russian-troll-factory-on-day-of-2018-midterms/2019/02/26/1827fc9e-36d6-11e9-af5b-b51b7ff322e9_story.html

https://www.propublica.org/article/infamous-russian-troll-farm-appears-to-be-source-of-anti-ukraine-propaganda

Similar things need to happen to the AI troll farms being ramped up by insurrectionist-funder Larry Ellison's Saudi installations.
https://www.arabnews.com/node/2425296/business-economy

https://www.washingtonpost.com/politics/2022/05/20/larry-ellison-oracle-trump-election-challenges/

U.S. Cyber Command operation disrupted Internet access of Russian troll factory on day of 2018 midterms

Some senators credit more-aggressive U.S. tactics with averting Russian interference in the elections.

The Washington Post
Show threadNicole ParsonsJan 1, 2024

@martinvermeer @the_roamer @Gaolaitch @JohnMashey

I sometimes wonder at what point did Republican billionaire donors explicitly stop being on the side of democracy.

For #KochNetwork, I believe it was the Clinton-Era EPA fines.
https://www.epa.gov/archive/epapages/newsroom_archive/newsreleases/981d17e5ab07246f8525686500621079.html

I'm not including Tory billionaire donors because the British upper classes & their allied monarchic despots in the Middle East have always been opposed to democracy.

And especially taxation of their wealth in the post-WW2 period.

Show threadPatrick McLaughlinJan 1, 2024

@Npars01 @martinvermeer @the_roamer @Gaolaitch @JohnMashey

The error here is imagining that they weren’t always. I offer you Henry Ford and Prescott Bush — and the Business Plot against FDR — as points of evidence.

One could go back earlier.

Always.

Show threadJonathanJan 2, 2024

@Npars01

I dunno, grampa Koch was pretty nuts

Show threadJonathanJan 2, 2024
@Npars01 once you have that amount of money, it's probably quite flattering to yourself (at least to a large chunk of humans who amassed that much money) to start considering the rest of your human brethren as ants to be squashed and democracy as something you don't care about because these stupid ants don't know anything anyway
Show threadScott marsroverdriver MaxwellJan 1, 2024
@Gaolaitch @the_roamer I'm dry, so I guess I didn't need this umbrella to keep the pouring rain off.
Show threadOldFartPhilJan 1, 2024
@Gaolaitch @the_roamer Why do I need an autonomic nervous system when I've never forgotten to breathe?
Show threadMartha HowellJan 1, 2024
@the_roamer @fifonetworks
Hours and hours of meetings to inventory issues, discuss fixes, meet with vendors, discuss patches and uogrades, and planned outages to ensure the actual event was seamless.
Show threadMartha HowellJan 1, 2024
@the_roamer @fifonetworks
We were still plagued by a daylight savings bug for years. It was a relatively easy fix that never seemed to make the project lists, until it had taken our online store down once too often. Seems like we could never remember about it until it had broken things, twice a year.
Show threadMr Lawrence 🇪🇺🇷🇼Jan 1, 2024
@the_roamer @fifonetworks Agreed! In the company I was in, we lost two systems to Y2K (both spotted ahead of time). One was mission critical and ran on OS/2 Warp. It was determined that there was no fix available but a workaround was put in place to force it to work from 2000-01-01. The other was on an old laptop running Windows 3.1.1 which had other issues and wasn’t vital, so it was replaced once the problem was proven on the day. #Y2K was definitely real 👍🏻
Show threadHairyChrisJan 2, 2024

@the_roamer @fifonetworks Yup. I worked in corporate IT support at the time, so weren't even at the cutting edge.

We spent the late 90s replacing non-compliant kit, patched *everything*, had a backup generator hired, and (lastly) I shut down everything at my location on 31/12 and restarted it 1/1 to avoid the roll-over.

Nothing weird happened. Job done!

EDIT: On 2nd thoughts we may have had some DOS/terminal print servers unhappy but those were dependent on 3rd party.

Show threadNick's worldJan 1, 2024
@JessTheUnstill @fifonetworks So you're saying that Y2K could have happened but that we were saved by people we don't know who updated things? Cool!
Show threadRoyce WilliamsJan 1, 2024

@fifonetworks Part of my Y2K story:

https://infosec.exchange/@tychotithonus/111670219441506220

I didn't include some of the social context, though. When I told the vendor representative in a meeting that I wanted to test the ATM machines for the Y2K bug, he laughed at me and said I was worried about nothing. This was extra motivation for me to figure out how to roll the clock on them ... by booting from a 3.5 floppy disk(!).

But once I proved that the ATMs would lock up when the date rolled, he apologized - and paid to fly techs all over Alaska to swap the ROM.

Royce Williams (@[email protected])

@robertatcara As someone who personally discovered and fixed Y2K bugs that would have had significant real world impact, it is disturbing to hear someone propagate this myth [that it was a "big fuss about nothing"]. And it is a myth. This is what really happened: https://time.com/5752129/y2k-bug-history/ The testing methodology insured that these impacts were not hypothetical. At my company, the testing was performed by *actually rolling the clock forward* to test systems to see what would happen. For example, I discovered that every ATM in the state of Alaska operated by my company would have locked up until a PROM chip was swapped. Someone had to fly all over the state to proactively swap the chip beforehand, to avoid significant customer impact. And that was just one story. I personally oversaw investigation and fixes for other hardware and software at that company that would have failed. And that was just my company. I spoke with others in IT at that time with similar stories. And that was just the people I knew. So no, it wasn't "a big fuss about nothing" - and saying so is both dangerously revisionist, and disrespectful of the work it took to prevent real impacts. #Y2K

Infosec Exchange
Show threadJoe ❌👑Jan 1, 2024
@fifonetworks I was (and still am) a software developer then and we fixed many Y2K bugs in the 18 months or so leading up to the critical date, as did many others. So much was spent on this that it delayed the first dot-com crash by a year, because there was so much economic activity in software.
Show threadAvi Rappoport (avirr)Jan 1, 2024
@fifonetworks My local school district had a second order Y2K bug in the financial system that came to light a year later. It lost over a million USD. If not for unsung engineers, it could have been like that in more places. I appreciate the effort!
Show threadSarah E BourneJan 1, 2024
@fifonetworks I'll bet @librarianshipwreck knows a few stories.
Show threadWhitmadJan 1, 2024

@fifonetworks

For three years before Y2K I was a contract developer at an investment bank, paid on the Y2K budget. In those three years I did two hours of Y2K related work. I have heard similar stories at other institutions.

I can believe that a lot of work went into preparing for Y2K, but I can say with certainty that many department heads used it as a pretext to get the budget for unrelated projects.

They paid me to be on call at new year. No-one called.

#newyear
#y2k

Show threadJohn Faithfull 🌍🇪🇺🏴󠁧󠁢󠁳󠁣󠁴󠁿🧡✊🏻✊🏿Jan 1, 2024
@fifonetworks 💯 % this! 👍
Show threadMark Nichols (LGBTQIA+ Ally)Jan 1, 2024
@fifonetworks I worked at a public utility company when the 1989/1990 year change occurred. Our billing system imploded as there was record type of “90" that shared the same offset as the YY field in the non-type-90 records. (The relevant code was also in Assembly. Yay) When the Y2K FUD started all we had to do was remind management of the 1990 mess, and they were behind getting everything updated. So code from ~1970, in Assembly, helped us avoid any Y2K issues.
Show threadThe Penguin of EvilJan 1, 2024

@fifonetworks I did the website tracking for Linux Y2K issues and there were a lot of userspace fixes. Come midnight we all celebrated a perfect score ruined about 30 seconds later by receiving a mail from [email protected] for the year 19100. But nothing bad broke 😀

2038 is rather more concerning because a lot of the old stuff in 2000 didn't do dates or they didn't matter except as cosmetics. There's a lot of old embedded 32bit where that's no longer true.

Show threadAlaric Snell-PymJan 1, 2024
@fifonetworks ha yes, I was tasked with making a major magazine's (you'd have heard of it) online subscription system work for Y2K. It was CGIs written in C with C string handling so lots of "char year[2]". No dev/staging site, no source control, just a ssh login to a shell account with the source code and a makefile that installed to prod :-| first step was to do "cp -R" to back the sources up before I touched anything, and so I could make a diff to share with colleagues for a review...
Show threadclacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛Jan 1, 2024

@fifonetworks The company I was working at was doing certifications of PCs, upgrading BIOSes, putting Y2K thumbs-up stickers on units as they proceeded.

Our ERP was a homegrown one from 1980, running on the mainframe. We knew it would be ok, because it used *one* digit for the year and it had managed the 89/90 transition just fine. 😃

Show threadchxJan 1, 2024
@fifonetworks second hand, alas, but I knew someone who was on the JMU Y2K task force, they started in 1994(!) and replaced accounting and HR software by 1998 when I lost contact. They were working with PeopleSoft at the time on the student admin system.
Show threadKevin LydaJan 1, 2024

@fifonetworks "The GPS satellites run on UTC"

Um... No they don't. In 1999 GPS time would be 13 seconds behind UTC. It's currently 18 seconds behind. GPS time only includes leap seconds up to 1980.

Show threadjavigusJan 1, 2024
@fifonetworks thank you! I like many others thought the problem had been overblown.
Show threadSensible CroneJan 2, 2024
@fifonetworks thank you for the history. ♥️
Show threadtomkeddieJan 2, 2024

@fifonetworks
100%

I fixed a y2k bug in January 1997. Debit cards with 3 year expiration were being issued with 01/00 as the year of expiry and the credit union ATMs were eating all of them. Y2K was a thing but the actual event went smoothly because people took it seriously. They didn't panic, just took it seriously.

Show threadBob YoungJan 2, 2024
@tomkeddie
I hadn't heard of that situation before. Thanks for sharing.
Show threadGemma 👽Jan 2, 2024
@fifonetworks
Good time to remind everyone that the year 2038 will be here in just short little 14 years.
https://en.wikipedia.org/wiki/Year_2038_problem
Year 2038 problem - Wikipedia

Show threadClifford AdamsJan 2, 2024
@fifonetworks
I worked at a small telco (500k-ish users). We filled several (recycling) dumpsters with equipment that was not Y2K certified. In the end, I noticed one system that printed 19100 in its daily logs. It was an old minor part of the 911 system that we kept running as a third-level backup.
Show threadExtreme ElectronicsJan 2, 2024

@fifonetworks

Not much of a story, worked with a number of customers checking/updating their servers and too many PC's to remember.

I was "on call" 31st Dec 1999 and didn't get to celebrate the new year until 01:00 2000

We had a single failure, a clocking in system (out of or control) that needed rebooting to allow clocking in.

The process gave me enough work/money to start my company, that is still going...

Show threadSodomy Non SapiensJan 2, 2024
@Extelec @fifonetworks I was a little more fortunate in that I worked with PBX's where a failure would not be a disaster, but would have had problems with any time profiles thinking it was the wrong day.
various test were made on all equipment, most passed, some needed software updates & for the rare system that was so old it could not e patched we advised the customer to change to a year that would match
we were on standby over the NYE, NYD with a huge bonus for doing so
Show threadExtreme ElectronicsJan 2, 2024
@hollowman I worked on a number of PDX's early/mid 90's. Other than logs I'm surprised they would care about the date. I do remember the smaller systems with "smart" handsets having strange dates after Y2K, which I put down to moving to a working year :)
Show threadSodomy Non SapiensJan 2, 2024
@Extelec the potential problems were quite simple. systems that had time based time profiles for night service ect. as the weekend would fall on the wrong dates so thy could find themselves "open" @ the weekend but "closed" Monday & Tuesday
Show threadAndy 🌎Jan 2, 2024

@fifonetworks My first job was at a small startup in 1995 which was formed out of a university software engineering lab to tackle the Y2K problem. We create automated tools for syntactic & semantic analysis, design recovery, language migration (e.g. COBOL II -> COBOL/370), and automated maintenance of large COBOL (all flavours), RPG, and PL/I systems.

1/2

Show threadAndy 🌎Jan 2, 2024

@fifonetworks We initially ran our tools on multi-processor Macs & AIX machines and eventually used Linux (Yellow Dog & Red Hat).

Our UI was an HTML interface! The tools ran as cgis on an Apache webserver.

We used our tools to process hundreds of millions of lines of banking, airline, and insurance company code, and also licensed them to IBM for their Y2K consulting.

I can tell you we found some very "creative" code in there that needed fixing!

2/2

Show threadScarabJan 3, 2024
@fifonetworks I'm another person who had to travel to multiple states to make sure Y2K didn't break us. I had to install updated software at different sites so our timestamp-reliant version control system didn't break. That worked. ... We sysadmins DID get paged that our server room had an emergency shutdown the night BEFORE or the night AFTER Y2K (I forget which, now), but that was due to a cleaning person's broom hitting the emergency shutdown button (we got a cover for the button after that).
Show threadBob YoungJan 3, 2024
@Configures
Thank you for your story! Have you looked through all of the comments? I posted it here, on LinkedIn, and on Facebook, and I've got 20-30 stories. I'm going to copy and paste them all into one document and put it on my blog. This has been great fun.