Mastodawn

[ Edit: I expanded this and turned it into an article at https://lcamtuf.substack.com/p/product-security-barking-up-the-wrong ]

The basics of infosec - such as meaningful asset inventories, privilege reduction and separation, or solid access control - are *not* actually basics. They're not something you start with and then are done with. They're unsolved problems in computer security. Companies mess this up not because they're careless and incompetent, but because we don't know how to do these things right.

Yes, it's easy on my Linux laptop. It's not easy when you have 10,000 employees. It only takes one person who, for the sake of expediency, puts a bootleg AWS instance on a corporate credit card and does some "prototyping" there. It only takes one person who does something creative with SSH tunnels to be able to "work from home". It only takes one person who installs a sketchy browser extension, or outright goes rogue.

At a scale, stuff like that happens *every day*, and even if your world-class tooling and education efforts get you to 95%, there's still that 5% that every organization is bound to miss. And 5% is enough. Heck, 1% is enough.

Yep, you try to lower that number, but it only gets you so far. The most successful security programs I've seen are not built around having perfect defenses. They're built around the assumption that you're gonna get compromised - and you need to detect it, respond to it, and contain it faster than the attackers can achieve their goals.

Product security: barking up the wrong tree

AppSec is fine. We're not paying enough attention to corporate infrastructure risks.

lcamtuf’s thing

SpaceLifeForm Mar 4, 2024

If you keep things offline, it really helps.

cate Mar 4, 2024

@SpaceLifeForm @lcamtuf read again. for one person it may work, but for a company no: paper will be misplaced, a lot of errors and so less care on checking if it is a forget paper. and offline makes checks very difficult: who did the error? when? offline doesn’t scale and it is difficult to get good security. and “backups”?

SpaceLifeForm Mar 4, 2024

Valid points.

Somehow, years ago, this is how it was done.

cate Mar 4, 2024

@SpaceLifeForm @lcamtuf So it means i’m older than you. It “worked” but very inefficient, slow and inaccurate. Banks needed to close few days in January to update accounts. And there were a lot of counters, and guess what? a large team of “fixers” which corrects errors, and still a lot of errors and missing transactions. You had various officials documents with different day of birth. sometimes fires and you lost a lot of officials documents (maybe just postal vehicle)

@lcamtuf I mean I agree with everything but it being easy in your laptop where you have 7 pyenvs, 14 forgotten docker containers, and more. 🤷

shellsharks Mar 4, 2024

@lcamtuf I've always said something very similar with regard to infosec disciplines that many regard as “junior” or "easy”. Vulnerability Management is one such role that I think is pretty easy to get started in (and many in security do) and for many considered to just be something that is easy/junior when in reality, doing *advanced* VM is something that takes a lot of finesse, organizational knowledge, cross-disciplinary skills, coding chops, etc... Same could be said for things like "SOC Analyst”. Sure you can run junior folks through that role but there is definitely a spectrum of proficiency that should not be overlooked.

So to pivot back to your point, sure you can check boxes for some of these controls but to do things “right" (for your org), or at-scale, or in a way that provides value without too much of a cost-sink, you usually have to go beyond the basics.

Gary McGraw Mar 4, 2024

@lcamtuf the only solution is to build security in.

Tim Bray Mar 4, 2024

@lcamtuf Was talking to a Very Senior Person in the security group of a Very Big Internet Company and they said “You know, we have thousands and thousands of engineers. I’m sure that at least one of them has another employer.”

Jacob | Five Eye Tea Mar 4, 2024

@lcamtuf This is a very fair assessment, but I do think there are definitely many instances where corporate data breaches are either due to incompetence or due to an unnecessary amount of data collection. I think there's definitely a balance.

I don't believe in shaming companies on data breaches unless it's clear that it was due to one of the factors named above, and even then, we have to be careful to attack the policies, not the people.

Jeff Grigg Mar 4, 2024

@ChallengeApathy @lcamtuf

Who made the policies?

Who resisted improving them?

Jacob | Five Eye Tea Mar 5, 2024

@JeffGrigg @lcamtuf In most cases, it's upper management seeking to cut costs or to make a pretty penny off of selling user data.

okanogen VerminEnemyFromWithin Mar 4, 2024

@lcamtuf
Sometimes things stay broken because our business's tools are broken. Legacy software with vital data or processing that no one knows how to upgrade and it only runs on 2012r2. Proprietary blackbox estimating/job costing/production software the vendor won't allow root access to and is available company wide, with known Linux vulnerabilities that they won't upgrade.
Then the things nearly impossible to find, former IT employee added a dialog script to an ordinary looking cronjob?

Ľuboš Moščovič

@lcamtuf
Well, but can the quick and effective incident response be deployed and run without those (indeed imperfect) foundations?

bassplayer Mar 4, 2024

@lcamtuf A lot of people want to do the right thing but it's difficult to get story points assigned to non-release items.

InfoSec4All Mar 4, 2024

@lcamtuf Perfect security is a unicorn, and you reach a point of very diminishing returns.
But of course that doesn't as mean we do nothing as I have to constantly explain to leadership.

lj·rk Mar 4, 2024

@lcamtuf Although I wish we had at least proper asset inventory for the "official" infra in many companies.

And companies not buying every new fancy tool because it's cheap and comes with a fancy PowerPoint.

Martin Seeger Mar 4, 2024

@lcamtuf The number I tell people: Calculate with about 50 significant security deviations per 1.000 employees and year. So 10.000 employees would be about 500 per year or 2 per workday.

As @isotopp once put it: I call it Tuesday.

Lazy B0y Mar 4, 2024

@lcamtuf agree getting things right 1000% is hard.

but are the real events when things happen really mostly overly „creative“ use of ssh tunnels to be able to work from home or rather „setting up an outside reachable service without changing the default password because time pressure and not thinking it through and not applying due diligence“?

really curious if there are statistics about that.

Nathan Byrd Mar 4, 2024

@lcamtuf Great post! Have you read Kelly Shortridge’s great book “Security Chaos Engineering”? I’ve found these concepts, while valid, are also hard to communicate. It can come off as “don’t bother trying to secure things, people will get in anyway” which is not the message. It’s more like “secure as much as you can, avoid eggshell security, and also focus on finding the attackers and response.” If the attackers are the cats you should be the room full of rocking chairs.

Nightyear Mar 4, 2024

@lcamtuf elegantly stated and so true. When vendors sincerely ask for input on how they can improve their products, I've said as much with very little gain. “If you don't make it easier for me to try and achieve the fundamentals (and you're not at the moment) then I'm not sure why I should keep buying from you."

cchelberg Mar 13, 2024

@lcamtuf thanks for expanding this out!