Mastodawn

kuzushi 🇲🇽 🇺🇸Sep 4, 2023

A long while back I got into a conversation with @wendynather on why you can't treat security standards like building for elemental risk (fires, earthquakes, etc) and safety standards because unlike these things, we are talking about adversaries that are intentionally working to circumvent such controls for their own benefit.

While I still contest that is true, lately I have this feeling that I am finding hard to shake... we are a professionally negligent industry. Let's say I was wrong, and indeed cybersecurity could be thought of as tolerances against predictable events... do we have any evidence that what we are doing is working? Where are the objective studies that show that compliancy standards have net positive reduced the very outcomes they are designed around?

The sheer lack of evidence of effectiveness, coupled against clearly growing threats and consequences is honestly alarming. I get that risk reduction still leaves opportunity for occurrence. Like, if I could reduce the risk by 80%, that still means that there is a 20% chance a thing could still happen. But as far as I have seen, I've only run into one study done on effectiveness that would even begin to direct people.

Maybe I am wrong, and all of this is just my own ignorance into the modern state of enterprise security. but, at least as far as breach reports I've been reading seem concerned, whatever we are doing isn't really tracking.

Show thread

Taggart

@kuzushi @wendynather To my mind, much of this disconnect concerns the indexing of risk to monetary damage. Which, for a company, I totally get. But then risk analysis is playing a different game than security analysis.

If the concern is about letting the bad guys do the thing, then of course compliance is insufficient. If, however, the concern is about maintaining regulatory compliance so that, in the event of a compromise, the correct protections are in place to limit the cost to the company.

That's not to say real risk is all monetary, but I do think security thinks a lot more about the intangible value of preventing a compromise than other groups, who focus on bottom-line costs.

The belt-tightening of insurance companies will likely shift this dynamic.

But either way, you are 100% right that we don't have a lot of evidence to point to to support any kind of claim.

Show thread

Jarrod Frates Sep 5, 2023

@mttaggart @wendynather @kuzushi

I think the lack of studies comes from two things:

1. A reluctance on the part of companies to take part in such studies.

2. A lack of reliable lab environments to conduct studies separate from live environments.

In the first case, no one wants to end up in a scientific paper as an example of how taking an action or set of actions led to catastrophe, even if they were part of a minority that landed in the failure group and a large majority was successful.

In the second case, building a lab environment to simulate an enterprise environment is incredibly expensive, and that's without all the legacy software, quirky configs, users bypassing security, and all the other fun things we deal with on a daily basis. A change to the fire codes can be tested in a lab and then checked out in a model construction or even a house scheduled for demolition for a much more realistic test. I would love to get such a luxury, but I'm doubtful I ever will.