e. hashman 🇵🇸Jul 19, 2024
Fun fact: when I worked in security I used to install and maintain Crowdstrike agents (among other security vendor products) on customer machines. It's not Windows-only, they also have a Linux client. Which runs as a kernel module and requires auditd.
Show threade. hashman 🇵🇸Jul 19, 2024

They also have a mac client, though it looks like neither Linux nor Apple machines were affected by the bad update.

I'd view that as pure dumb luck until I actually see an RCA. Because of the way their agents work, any system could be utterly borked.

Show threade. hashman 🇵🇸Jul 19, 2024
I would have thought at Crowdstrike's scale that SURELY they use a slow-rollout/canary model for global updates. But the scale of this outage suggests otherwise. There's no way the rollout should have continued with 100% of clients not checking in.
BradJul 19, 2024
Show threade. hashman 🇵🇸
I'm rolling my eyes at the people going off about "don't deploy on Fridays". This went out Thursday night. No matter what day of the week it went out, it's going to take more than a work week to fix and everyone's weekend would be toast. Sometimes you just ship real stinkers, lol
Jul 19, 2024 at 4:38pmWeb
Show threade. hashman 🇵🇸Jul 20, 2024
Everyone loves to confidently proclaim that companies shouldn't do X or Y (don't use rootkits for security! audit checkboxes are useless!) but it's funny how there's always dead silence when you ask what companies who really don't have security as a core competency should do instead
Show threade. hashman 🇵🇸Jul 20, 2024

Genuinely trying to understand here why Starbucks should be investing in building a world-class computer security organization instead of just paying for the best option vendor product

Hard for me to see this as anything but self-serving for people in the security industry, lol

Show threadIrenes (many)Jul 20, 2024

@ehashman yeah there really is no realistic option

but please note that, like, these multinationals have the market power to insist that somebody make a realistic option for them, if they cared to

Show threadIrenes (many)Jul 20, 2024
@ehashman like, we agree that it's unrealistic for every company to specialize in information security, but these are not exactly struggling mom-and-pop shops
Show threadIrenes (many)Jul 20, 2024
@ehashman we were discussing with friends this morning what exactly would need to change about the law and/or the contracts for these security products to do real things that people could hold them to... we didn't come up with any solid conclusions as to what would work, alas, the incentives tug very strongly in the other direction
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista there's definitely something to be said about OS architecture such that agents like this are considered necessary for adequate security coverage, and I think I would start there. I have a bunch of folks in my mentions talking about this failure mode being impossible on macOS and work on Linux having been done that prevents it for future versions. But on the flip side, if you don't want a third party solution, you're putting all your eggs in the baskets of the OS makers
Show threadIrenes (many)Jul 20, 2024
@ehashman yes, we're big fans of technologies such as language-level security features and formal verification which could someday tip the balance towards the defender
Show threadIrenes (many)Jul 20, 2024
@ehashman the problem with any such thing is that it requires companies to care. the reality is that executives and shareholders would prefer to risk outages like today's rather than take precautions. it's more of an ideological decision than a rational one: cut costs no matter what the consequences are.
Show threadIrenes (many)Jul 20, 2024
@ehashman Microsoft has been the only company that consistently funds formal verification research (we know this through our friends who've sought funding for their projects over the years). it's good that SOMEBODY does, but it's a lot less likely to amount to anything than if the market actually wanted better outcomes.
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista the ones hitting the news, not so much, but when I was installing security vendor agents for a living, the vast number of our customers numerically were very tech illiterate smaller businesses
Show threadIrenes (many)Jul 20, 2024
@ehashman yes, certainly. we want to be clear, it's our position that the product does very little that anyone should want in the first place. it's the (pardon the jargon) hyper-real simulacrum of security, not the real thing.
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista I think it's fine to argue that these products are overkill/expensive/poorly engineered/excessive for security monitoring but I think that's a different argument than saying these do nothing at all. Because the alternative in my experience is unmonitored boxes getting popped without detection, and most shops don't want to or know how to roll their own security monitoring. Maybe in the best case one of their infra providers will let them know about it
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista I think the customers of these vendors are doing the best they can given the available options and expertise they can afford and basically any other solution being proposed right now is demanding they spend a whole lot more on computer security. Maybe the biggest companies should be! And that would be great for people who work in computer security, but idk if that actually improves the world we live in
Show threadIrenes (many)Jul 20, 2024
@ehashman it's rational for the companies buying these products to cheap out, because it's really THEIR customers who suffer. the airlines and retailers that were hurting today won't really be punished much for it, because everyone else was having the same problem.
Show threadIrenes (many)Jul 20, 2024
@ehashman the market doesn't care about humanity's well-being.
Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista 💯
Show threadIrenes (many)Jul 20, 2024
@ehashman so it's not really that we can blame these companies for choosing to do things that they were definitely advised would be inadequate (we know this because we've seen how these kinds of decisions get reviewed - somebody always raises the concern, the only variation between companies is how seriously they're taken).
Show threadIrenes (many)Jul 20, 2024

@ehashman .. in exactly the same sense as, if we wandered into a tiger's habitat, we couldn't blame it for eating us

but we can maybe suggest that perhaps that means decisions that affect the public this severely shouldn't be in the hands of entities that have no reason to care

Show threadCassandra is only carbon nowJul 20, 2024
@ireneista @ehashman Irrespective of context, evergreen toot.
Show threadIrenes (many)Jul 20, 2024
@xgranade @ehashman thank you
Show threadIrenes (many)Jul 20, 2024

@ehashman yes, fine, reactive monitoring for existing threats is better than nothing. that is fair.

we would never argue that this stuff is OVERKILL. we don't think a real solution would end up looking anything like it, really, but our belief is definitely that the tools do not do enough, not that they do too much

Show threade. hashman 🇵🇸Jul 20, 2024
@ireneista I feel a little silly because I'm not trying to argue that Crowdstrike's product is good or anything, just that there are genuinely good reasons that people pay for it, even if a better solution probably exists out there :)
Show threadIrenes (many)Jul 20, 2024
@ehashman well, for what it's worth the discussion helped us understand our own position better. it's not that we think there's a better solution out there - there isn't. it's that we think the entire topic is drastically under-funded, and what funding does exist goes mostly to rent-seeking products with minimal utility, rather than to making real progress
Show threadIrenes (many)Jul 20, 2024
@ehashman if you're silly but it helps yourself or others to express themselves better, it wasn't silly, in our view. or at least not silly in a bad way
Show threadCassandrichJul 20, 2024
@ehashman You don't need a "world class computer security organization". You just need to not be doing stupid shit and running workstations like they're employees' personal computers. A workstation run like a workstation doesn't get viruses. A POS run like a POS doesn't get viruses.
Show threade. hashman 🇵🇸Jul 20, 2024
@dalias ah yes, "just not do stupid shit", a thing large organizations that employee thousands of non-computer literate are known for their expertise in
Show threadCassandrichJul 20, 2024
@ehashman If you're a large company, you hire competent ppl who work for you and whose obligation is to your interests rather than paying a Security Product vendor whose obligation is parting you with your money to CYA.
Show threade. hashman 🇵🇸Jul 20, 2024
@dalias can you name 5 real world examples where this has demonstrably worked?
Show threadGlyphJul 20, 2024

@ehashman I realize that this is not a *realistic* suggestion within the constraints of the market pressures to reduce safety margins as far as possible, but it is a *serious* one:

“For the same reasons that Waffle House invests in disaster preparedness rather than simply ticking compliance boxes from the fire marshall”

Show threadKevin RiggleJul 20, 2024
@ehashman cloud native companies are _head and shoulders_ above anybody still doing on prem anything. Let Google’s security team do pager duty for your login form
Show threadKevin RiggleJul 20, 2024
@ehashman at least judging by my clients (people can also hire me, I mean)
Show threadKevin RiggleJul 20, 2024
@ehashman a lot of people who don’t think that they are running distributed systems find Tenerife’s running distributed systems