@SmudgeTheInsultCat
I experienced things like this multiple times.
@SmudgeTheInsultCat The problem with attributing the AT&T cell network outage to a minor operational detail--expired cert, etc.--is that a number of other networks were hit hard at the same time.
@vandys @SmudgeTheInsultCat The other carriers attribute this to secondary effects, because they used some parts of AT&Ts infrastructure. (There is another discussion about all the problems with the american network infrastructure here...)

@vandys @SmudgeTheInsultCat

Interconnected. A few years ago, dozens of sites went down because of one change to Node.js.

@SmudgeTheInsultCat
Two years after Carl was laid off, while taking a shower, he suddenly remembered the certificate was going to expire, and he's been gleefully waiting for it ever since.
@SmudgeTheInsultCat so true i see those certificate issues all the time πŸ€ͺπŸ˜‚πŸ˜‚πŸ˜‚ i have so much stories about that πŸ˜‚πŸ˜‚πŸ˜‚
@SmudgeTheInsultCat Or where I work: β€œSome certificate expired because people can’t use a fucking calendar. The length of the outage was length of time it took to connect the problem with the expired cert. Then it was quickly solved, but should have never happened.”

@SmudgeTheInsultCat

Did it happen at midnight when the midnight munchers come out to play?

@SmudgeTheInsultCat

Feels like it could be so...

What's the ChatGPT prompt to regenerate keys on your multi-billion dollar phone/data network? Wonder if the kid just hired out of school can figure that one out...

@SmudgeTheInsultCat nah. Too complicated. More likely is that there are spaces instead of a tab in a config file pushed out to thousands of routers.
@SmudgeTheInsultCat certs can only be renewed for a year at a time now. So plan for yearly outages like this.
@SmudgeTheInsultCat Funny how it seems almost impossible not to let at least a few certs expire in any given time span.