Dare ObasanjoJul 21, 2024

There is a weirdly racist trend where people blame hypothetical minorities when systems fail because management cuts corners. We saw it with the Boeing door collapse and now with the Crowdstrike crash.

There is no mythical "DEI engineer" doing sloppy work at these companies.

It's down to leadership cutting corners to make a buck. A software update causing a crash that takes down all of your customers is due to inadequate testing. It's not because some anonymous minority wrote the code.

Show threadJon McLeod
@carnage4life You are totally correct with your analysis here. "Inadequate testing". DevOps and continuous deployment have created a paradigm where changes can be pushed to channel without adequate testing or even 'stop and think' assessments of possible dependencies impact. The product managers often don't understand what they're managing. The goal statements ten years ago included "fail fast, fail cheap" and "cut bureaucracy" but this is the result.
Jul 21, 2024 at 12:36amWeb
Show threadferricoxideJul 21, 2024
@milennialzero @carnage4life

A Real™ devops pipeline has the testing in place to catch things – at least those that are anticipatable. Competent organizations do what they can to minimize the blast-radius of any given failure. As part of that, you don't generally just blast something out to "all the things" at once and you certainly don't do it without a predefined, tested rollback process ready to go.
Show threadFrank QuednauJul 21, 2024
@milennialzero @carnage4life the “fail cheap” is the issue here, hence why a pure continuous deployment setup in this particular case might be missing an element.
Show threadThe Queen of Weltschmerz🏳️‍⚧️Jul 21, 2024

@milennialzero @carnage4life DevOps, Agile or Waterfall, testing *always* gets cut to the bone because management push for earlier and earlier delivery, they insist on ludicrous “happy day” estimation and they perceive testing time as “soft”- as in “if you code it right first time you don’t need all that testing”.

Ridiculous, yes, but I’ve experienced it so many times in 30+ years…

Show threadDries V.Jul 21, 2024

@milennialzero @carnage4life also at the client side, automated updates should be blocked or delayed so that non production env receive them first and can be tested.

I don't know the product but if it didn't allow for such a scenario, I wouldn't used it in a critical env.

Show threadTim Ward ⭐🇪🇺🔶 #FBPEJul 21, 2024
@milennialzero @carnage4life But but but ... the CI/CD pipeline contains testing steps, for example the step that signs the content should, surely, refuse to do so if the earlier step that validates format has failed? So it shouldn't matter what crap is fed into the beginning of the pipeline, it shouldn't make it through to the end.
Show threadJon McLeodJul 21, 2024
@TimWardCam @carnage4life