fraggleOn January 15, 1990, a single line of buggy C code brought down AT&T's entire long-distance network for 9 hours, affecting 75 million phone calls. The culprit? A misplaced "break" statement in a software update for their 4ESS switching systems. The disaster began when a switch received a second call-attempt message while it was already in the middle of resetting its internal logic - something the software wasn't designed to handle.
Here's the kicker: the bug was in error-recovery code that was supposed to make the system MORE reliable. When the switch's software encountered this unexpected second message during reset, it couldn't process the conflicting instruction and declared its own processor "insane." To prevent spreading the problem, the switch shut itself down and notified other switches it was out of service. But when it came back online and started sending normal call processing messages again, those messages arrived at neighboring switches that were themselves in the middle of resetting after hearing about the outage - triggering the exact same bug nationwide.
The incident taught the telecom industry crucial lessons about software testing and gradual rollouts. AT&T had tested the code extensively, but never under the specific real-world conditions that triggered the bug. It also demonstrated how a single line of code could paralyze critical infrastructure that millions of people depended on daily.
This event helped shape modern software deployment practices and redundancy planning that we still use today.
#att #softwarebug #telecom #infrastructure #cascadingfailure
Here's the kicker: the bug was in error-recovery code that was supposed to make the system MORE reliable. When the switch's software encountered this unexpected second message during reset, it couldn't process the conflicting instruction and declared its own processor "insane." To prevent spreading the problem, the switch shut itself down and notified other switches it was out of service. But when it came back online and started sending normal call processing messages again, those messages arrived at neighboring switches that were themselves in the middle of resetting after hearing about the outage - triggering the exact same bug nationwide.
The incident taught the telecom industry crucial lessons about software testing and gradual rollouts. AT&T had tested the code extensively, but never under the specific real-world conditions that triggered the bug. It also demonstrated how a single line of code could paralyze critical infrastructure that millions of people depended on daily.
This event helped shape modern software deployment practices and redundancy planning that we still use today.
#att #softwarebug #telecom #infrastructure #cascadingfailure
Ari [APz] Sovijärvi
Doug Grinbergs
N-gated Hacker News
chribonn
Krutonium://
apfeltalk 
