e. hashman 🇵🇸Jul 19, 2024
Fun fact: when I worked in security I used to install and maintain Crowdstrike agents (among other security vendor products) on customer machines. It's not Windows-only, they also have a Linux client. Which runs as a kernel module and requires auditd.
Show threade. hashman 🇵🇸Jul 19, 2024

They also have a mac client, though it looks like neither Linux nor Apple machines were affected by the bad update.

I'd view that as pure dumb luck until I actually see an RCA. Because of the way their agents work, any system could be utterly borked.

Show threade. hashman 🇵🇸Jul 19, 2024
I would have thought at Crowdstrike's scale that SURELY they use a slow-rollout/canary model for global updates. But the scale of this outage suggests otherwise. There's no way the rollout should have continued with 100% of clients not checking in.
Show threade. hashman 🇵🇸Jul 19, 2024
I'm rolling my eyes at the people going off about "don't deploy on Fridays". This went out Thursday night. No matter what day of the week it went out, it's going to take more than a work week to fix and everyone's weekend would be toast. Sometimes you just ship real stinkers, lol
Show threade. hashman 🇵🇸Jul 20, 2024
Everyone loves to confidently proclaim that companies shouldn't do X or Y (don't use rootkits for security! audit checkboxes are useless!) but it's funny how there's always dead silence when you ask what companies who really don't have security as a core competency should do instead
Show threade. hashman 🇵🇸Jul 20, 2024

Genuinely trying to understand here why Starbucks should be investing in building a world-class computer security organization instead of just paying for the best option vendor product

Hard for me to see this as anything but self-serving for people in the security industry, lol

Show threadIrenes (many)Jul 20, 2024

@ehashman yeah there really is no realistic option

but please note that, like, these multinationals have the market power to insist that somebody make a realistic option for them, if they cared to

Show threadIrenes (many)Jul 20, 2024
@ehashman like, we agree that it's unrealistic for every company to specialize in information security, but these are not exactly struggling mom-and-pop shops
Show threadIrenes (many)Jul 20, 2024
@ehashman we were discussing with friends this morning what exactly would need to change about the law and/or the contracts for these security products to do real things that people could hold them to... we didn't come up with any solid conclusions as to what would work, alas, the incentives tug very strongly in the other direction
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista there's definitely something to be said about OS architecture such that agents like this are considered necessary for adequate security coverage, and I think I would start there. I have a bunch of folks in my mentions talking about this failure mode being impossible on macOS and work on Linux having been done that prevents it for future versions. But on the flip side, if you don't want a third party solution, you're putting all your eggs in the baskets of the OS makers
Show threadIrenes (many)
@ehashman yes, we're big fans of technologies such as language-level security features and formal verification which could someday tip the balance towards the defender
Jul 20, 2024 at 2:18amWeb
Show threadIrenes (many)Jul 20, 2024
@ehashman the problem with any such thing is that it requires companies to care. the reality is that executives and shareholders would prefer to risk outages like today's rather than take precautions. it's more of an ideological decision than a rational one: cut costs no matter what the consequences are.
Show threadIrenes (many)Jul 20, 2024
@ehashman Microsoft has been the only company that consistently funds formal verification research (we know this through our friends who've sought funding for their projects over the years). it's good that SOMEBODY does, but it's a lot less likely to amount to anything than if the market actually wanted better outcomes.