So, I'm doing some homework on the Southwest disaster — probably more tomorrow. And I hate to say this, but it doesn't look like a simple morality play about greedy capitalists 1/
A lot of the issue was apparently that SW does point-to-point, not hub and spoke. This left planes and especially crews stranded far more than other airlines 2/
But p2p has advantages in normal conditions. In fact, other airlines have to some extent been moving back to the direct flight system over time 3/
Worth noting that before this happened, SW was ranked #1 in customer satisfaction (in basic economy — it doesn't have business class) 4/ https://www.jdpower.com/business/press-releases/2022-north-america-airline-satisfaction-study
2022 North America Airline Satisfaction Study

TROY, Mich.: 11 May 2022 — The crowds are back at the airport, those empty middle seats are occupied again and airlines in North America are raising ticket prices in response to soaring fuel costs and continued strong leisure travel demand—all at the expense of passenger satisfaction. While dramatically higher prices could harm airline brands in the long term, for now, load volume is continuing to climb and passengers are willing to be assigned a middle seat in exchange for getting out of their houses, according to the J.D. Power 2022 North America Airline Satisfaction Study,SM released today.

J.D. Power
Now clearly SW had an inadequate scheduling system; there may be a story of corporate penny-pinching here. But hardly unique 5/
@pkrugman biggest penny-pinching aspect is total refusal to rebook passengers on other carriers, ever

@pkrugman I work on distributed systems meant to help with this category of problem. It is a hard problem but the way check-in needs to be done for crew is absolutely solvable with a moderate investment for a company their size.

But ...they pay peanuts for IT, and don't allow remote work, so they are very limited in their ability to find people who know how. And from what I hear in backchannels management didn't even know they SHOULD want to fix the issue before now. All their staff sure did.

@reneestephen @pkrugman yeah the idea that the system “normally works pretty well” doesn’t mean it wasn’t glaringly obvious to many many people that it was, in fact, fundamentally inadequate to its basic purpose, which has to include recovering from mass cancellations
@reneestephen @pkrugman that’s not like, a luxury deluxe extra feature

@mtrkdjoyce I've been mulling a theory around management solving old problems because those were the things they experienced before they became management. In SW's case the system was probably truly fine at 1/3rd the size they are. So nobody who made decisions have lived insight into the scaling issues since they happened too recently.

@pkrugman

@reneestephen @mtrkdjoyce @pkrugman This all feels very reminiscent of other failures where managers were not aware of how backwards they were, and so were unable to hear it from their better-informed staff. The key question, why? Was the staff afraid, as they were in Nokia's inability to transition to a touch-enabled operating system? I love this history of it: https://alumnimagazine.insead.edu/who-killed-nokia-nokia-did/
Who Killed Nokia? Nokia Did

Despite being an exemplar of strategic agility, the fearful emotional climate prevailing at Nokia during the rise of the iPhone...

INSEAD Alumni Magazine

@sladner 'This was made worse by the lack of technical competence among top managers, which influenced how they could assess technological limitations during goal setting.'

I briefly worked for a Nokia supplier in Nokia's salad days (well, slightly wilted salad days), and this rings true. I certainly had some head-scratching meetings in Tampere.

@stacey_campbell I'm sorry to hear that but also gratified it rings true
@sladner @reneestephen @mtrkdjoyce People who have worked in large companies know very well that "leaders" often know very little about the products they are leading.
@david1 were that only not the case!
@sladner @reneestephen @mtrkdjoyce @pkrugman interesting article, I worked for orange UK in those wilted salad days. I wonder if the upper management behaviour described in the article was a telco thing?
@reneestephen @mtrkdjoyce @sladner @pkrugman I am getting Elon Musk vibes here in this paragraph

@stefan_grosse I struggle to understand what they are afraid of. Getting screamed at? I mean, that's gonna happen anyway. Being brushed off? Oh, well. Fired? Not like they were effective there anyway: work somewhere better. All of this behaviour just keeps the wrong people in the wrong positions working on the wrong things.

But yeah the "mistaking wealth for competence" category of leader is all too common.

@mtrkdjoyce @sladner @pkrugman

@sladner @reneestephen @pkrugman @mtrkdjoyce getting screamed at frequently in front of many others is quite a thing I believe. Firing is not that usual in Europe. More likely you get transferred to minor positions. Taking hits in your variable payment, getting fewer resources (people), different tasks. In a toxic environment people tend to move as little as possible an only in the direction they are told in order to prevent
@sladner @reneestephen @pkrugman @mtrkdjoyce punishment. Another extreme example for toxic environment I believe would be Putin where punishment even included mortal „accidents“. Consequently his collection of fearful yes sayers did prevent him from realizing how bad his army was and that Ukraine government was way better organized. If you don’t want bad news, the truth is bent.
@stefan_grosse @reneestephen @mtrkdjoyce @pkrugman My fav detail in that study was the executive who pounded the table so hard that "fruit flew out of the fruit bowl."
@mtrkdjoyce @reneestephen @pkrugman “it normally works pretty well” absolutely describes every system ever before an unplanned-for catastrophe.

@mtrkdjoyce @reneestephen @pkrugman

...also, making a system that only works when everything is going fine, a system with near-zero resilience, is how the care-homes were run, despite reports of them being inadequate and brilttle for decades - and it was considered the very soul and definition of "capitalist greed".

Without regulation, capitalists always choose "efficiency" (cheapness, greed) over "resilience". Choices they would never make for their own family.

@mtrkdjoyce @reneestephen @pkrugman this happened to SW airlines Oct 2021. A Washington Post article from that time noted staffing shortages and p2p flights likely contributed to cancellations above and beyond other airlines. SW had time to fix the problem but clearly chose not to https://www.washingtonpost.com/transportation/2021/10/11/southwest-airlines-flight-cancellations/
Southwest Airlines flight woes extend to Monday as industry scrambles to meet passenger rebound

The carrier is working to return flight operations to normal as it deals with a backlog of passengers from nearly 2,000 canceled flights.

The Washington Post
@reneestephen @pkrugman What does the solution look like at a high level? Does it boil down to adding extra redundancies?

@AlexandreZani Currently their system assumes that if one part of the workflow is scheduled, it happens. If it doesn't happen staff need to call into the scheduling centre to let them know their current location so it can be entered and accoms rerouted. During mass cancellation, tho, staff flood the scheduling team with calls... some were on hold for 20h. Nobody knew where staff were 🤷

Redundancies come later, once SW actually has *any check-in system for staff whatsoever.*

@pkrugman

@reneestephen @pkrugman Oh wow, this is way worse than I thought. I assumed they had a somewhat wrong network topology or something, not "rescheduling requires calling humans" and "let's assume nothing ever goes wrong".
@pkrugman Maybe the story isn't a simple morality tale of greed, but rather a more complex one about the importance that systems be resilient to unusual, but predictable events like winter storms. Southwest's system collapsed either because of inadequate analysis and planning for such events or because of management decisions that resilience would make for a less profitable airline.

@pkrugman I think we need deeper understandings of resilience. We build brittle systems and act surprised when they break under stress. Some systems likely need to be re-conceptualized. Other systems may be inherently "fair weather" friends.

My late husband, an orchestral tubist, lived in Austria and Mexico. He said that when the train was a minute late in Austria, people became impatient. But in Mexico, when the train arrived, people cheered.

@pkrugman - https://www.seat31b.com had a fairly detailed and plausible account of the problem and it does look awfully like a system that failed to pay for the spare capacity needed for resilience.
Seat 31B – The World In Economy Class

@pkrugman to say Southwest’s tracking and scheduling program is inadequate is the understatement of the year.

If you are going to run an extremely tight ship, with little tolerance for error (tolerance is not priority #1), then recovery capability has to be priority #1. There is no excuse for a lack of disaster recovery and business continuity planning, preparation, and testing.

Based on a lot of reading, much of it with claimed insider insight, they had poor, if any, business continuity planning and practice. In so many other critical industries we not only have BC processes, but we also practice worst case disaster recovery (eg cold boot with expected data loss) as a at least yearly exercise.

It not like they shouldn’t expect a major multi-state winter weather event. This fridged blast is not a “100 year storm”. They are becoming yearly events.

You either invest in resiliency or invest in recovery, or a ideally a balance of both. SW seems to have chosen neither. And all those pinched pennies and far more are now going to be spent on buying back a not insignificant portion their customer base plus some amount of the original investments they should have made in handling such a predictable scenario.

I’d give SW credit if they could have cancelled a single day’s worth of flights as a reboot and automatically rebooked passengers, cross airline if necessary, but many-day operational failure and simply dropping your customers is inexcusable.

Their business continuity plan seems to consist entirely of “try harder”.

@pkrugman this is the best overview and explanation I’ve seen re: SW meltdown. https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/ Noteworthy issue: markets don’t work when companies are shielded from financial liability
The Great Southwest Meltdown Of 2022 – Seat 31B

@pkrugman Southwest has always had plenty of money. Their problem seems to be one of inadequate staffing in addition to flawed scheduling software...
@pkrugman consider taking a look at https://how.complexsystems.fail
How Complex Systems Fail

TProphet (@[email protected])

1/ A lot of people have been asking for an explainer on what is going on with Southwest Airlines and the massive meltdown that occurred. Hi, I'm TProphet. I write the Seat 31B travel blog (https://www.seat31b.com) and closely follow the airline industry. More importantly, I have a friend whom Southwest abandoned in Las Vegas until New Year's (along with his cat), and there was literally nothing I could do for him. Ready? Let's dive in.

DEF CON Social
@pkrugman old legacy systems and other "independent" features?

@pkrugman

As a question of corporate risk management, it's an epic failure, as in three-mile-island size failure.

To not understand how point-to-point unravels under stress is to not understand your business, right?

Q: You wrote that we could privatize traffic control, if done right. Does SW experience change your view, or is it tangential?

@pkrugman
This survey also has JetBlue as #1 so you have to ask yourself "Who are these people who are responding?"
Have they ever been on a GOOD airline?
The best of the worst is still horrible.
@pkrugman they’re also non-union.

@pkrugman

...and well deserved. as someone who, in a past life, flew at least 4 flights a week, sw and midwest express quality stood out (domestic). especially post 911. in particular, beyond the better seat rake, the employees of both companies seemed happy, which always reflects well on the employer, and leads to a better customer experience.

@pkrugman For a time, Southwest was top-rated, but I don't think that was true this year. In fact, a top rating from JD Power just means they were willing to pay for the honor; for three years in a row, TWA was JD Power's top-rated airline. That still didn't stop them from going broke...
@pkrugman Is hub organization more vulnerable to a single point of failure, e.g. a storm or emergency that happens to hit the hub?

@pkrugman But every airline ought to know that conditions are not always normal. Storms coinciding with peak holiday traffic have happened before and will happen again.

In general, Greedy Corporations should not be allowed to rake in profits in the easy situations w/o also being held accountable for having contingency plans in place for the more difficult situations and for the long term consequences of their actions.

@pkrugman @pkrugman I assume it might also be about antiquated systems that require more humans to run them than they were willing to have on hand/able to have on hand in the context of COVID/holiday
@petnoodle @pkrugman this is what I heard from a Southwest pilot - their crew scheduling system is out of date and requires far too many manual overrides and inputs in the event of any cancellation - compounded, and very hard to fix quickly.
@petnoodle @pkrugman (The spoke and hub model also has its disadvantages in a mass cancellation scenario - namely, too many planes in the same place at the same time, nowhere to put them all)
@mtrkdjoyce @pkrugman Aha, exactly then - reminds me of a situation at my office, which reminds me that I was going to do some work this afternoon despite it being holiday break
@pkrugman was it the p2p architecture itself or due to ill suited logistics software?
@pkrugman This could have a future ripple effect as airlines are abandoning their smaller hubs, like like Delta has done with CVG, MKE, and MEM (all former NWA hubs). Anti-trust laws matter.

@pkrugman Southwest would have coped with disruptions of flights entering or leaving a major hub even *worse*. Their dispatching software assumed each flight would take place, and had to be manually corrected on cancellation, so after many cancellations, it had no idea where the crews were.

If they had a United-style hub at Chicago or Newark, a shutdown of *just those airports* would have the same results every single time.

@pkrugman Yeah, P2P failed under this extraordinary weather event. OTOH, when it works - which it apparently has been - it gives SW the pricing advantage they enjoy over bigger airlines. This comes from the ability to fly one type of aircraft, reducing spare parts and maintenance costs and complexity; and also freeing them up to reschedule routes on the fly without worrying about having the right plane in the right place.
The Great Southwest Meltdown Of 2022 – Seat 31B

@pkrugman If you haven't seen this toot series, worth it for your research, I think: https://defcon.social/@tprophet/109588899812358967
TProphet (@[email protected])

1/ A lot of people have been asking for an explainer on what is going on with Southwest Airlines and the massive meltdown that occurred. Hi, I'm TProphet. I write the Seat 31B travel blog (https://www.seat31b.com) and closely follow the airline industry. More importantly, I have a friend whom Southwest abandoned in Las Vegas until New Year's (along with his cat), and there was literally nothing I could do for him. Ready? Let's dive in.

DEF CON Social

@pkrugman I listened to an NPR interview with the head of their flight attendant union yesterday and they discussed the issue of pre-emptive cancellations.

Other airlines were cancelling flights before the storm to prevent trapping crew and airplanes. Southwest didn’t do that. There’s maybe a little greed there.

@pkrugman If this had happened to SW's predecessor, the legendary PSA, the crew and passengers would have just gotten stoned and had a party.

(Which reminds me of the time I was snowbound one night on an Italian train with the Bologna womens' volleyball team.)

@pkrugman I guess the lesson here is not to attribute to malevolence that which can be attributed to incompetence.
@pkrugman this is a leadership error. C-suite, but also #SouthWestAirlines board of directors. They should have conducted scenario planning to see how to respond to some sort of “unknown”. That could be schedule disruption, massive spike in oil and gas, grounding of a aircraft type, IT failures etc. These exercises build mental muscle and help boards cope with a real crisis. And proactively identify areas of investment. Bad winter weather is a high probability event.
TProphet (@[email protected])

1/ A lot of people have been asking for an explainer on what is going on with Southwest Airlines and the massive meltdown that occurred. Hi, I'm TProphet. I write the Seat 31B travel blog (https://www.seat31b.com) and closely follow the airline industry. More importantly, I have a friend whom Southwest abandoned in Las Vegas until New Year's (along with his cat), and there was literally nothing I could do for him. Ready? Let's dive in.

DEF CON Social