Thank you Crowdstrike for helping to illustrate that Open Source is not the problem.
@bagder storytime :)
@katchwreck @bagder approximately eight hours ago, an update in CrowdStrike software has caused a massive outage on Windows computers. https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/
@bagder very challenging situation ... distributed system design is hard.
@jimfuller @bagder is "test patches in a lab before rolling them out to millions of systems worldwide" really a distributed systems problem
@privateger @bagder I believe so ... distributed systems is not just about how the systems operates but also how rolling upgrades are applied to such systems ... in this case I would have expected containment/mitigation of a bad patch
@jimfuller @bagder It seems they've tried to do that actually. It stopped rolling out a few minutes after problems were reported, but it was far too late at that point.

@privateger @bagder @jimfuller

if that's true it wasn't a "rollout", or at least not a controlled one. A rollout would be turning off updates after small measured increments and checking that things were still going well before proceeding with the next chunk (increment size doesn't need to be constant—often isn't—but does need to start small).

If you're combating an active 0-day attack you might be justified going full-throttle right off the bat, but do so knowing you're rolling the dice.

@dveditz @privateger @bagder sounds like it was virus def that triggered some regression, of course no matter how rare that might cause a problem, does not explain not testing the batch against a 'canary set' of hosts and progressive rolling upgrade ... guessing the problem here is they (CS/M$) think a certain part of their system is totally safe

@bagder

saving that one for posterity !

@bagder @bagder
It immediately came to mind that this would never happen on #Linux
@bagder @bagder meanwhile #MSM hosts are blaming "technology" and "the internet" for the outages like boomers lol

@bagder

It is what you get when we are fully dependent on a few commercial players of some of our most important infrastructures. It should and can change. Let it be a lesson.

@bagder and nothing will change. Next week (or earlier), everybody has forgotten today.
@bagder but I bet that Crowdstrike uses curl inside to distribute its rolling patches, so they will anyway find a way to reel you in! 🤦‍♂️
@bagder beautiful, isn't it

@bagder

I'll send this to Mr. de Vries 😅

@bagder I like that fact, that cloudstrike layed out a perfect cloud strike 🤣
@bagder No just bad management and zero pretesting.
@bagder
Cough *OpenSSH* Cough.
@bagder Surely the cause of the crowd strike issue was an update to their vendored curl 😁
@bagder I am still getting to know the open source ecosystem, open source thinking and open source working. So for a better understanding, how does this issue reflect positive on open source?
@GroeneBij I don't think it does. But lately it seems we've seen people talk about FOSS sustainability and as if these events were somehow limited to underfunded open source projects. Crowdstrike helps show the challenges go beyond that.