@blinkygal

There's a 1992 paper/keynote by Nancy Leveson [1] that I think about sometimes, in which she compares software development to steam engines. In the 19th century, people developed high-pressure steam engines despite not having the metallurgical, engineering and manufacturing process knowledge for how to make boilers that don't occasionally explode, killing people. It took almost a century to really solve the problem.

She was arguing that software and steam engines share similar relationships between economic usefulness, technological limitations, and safety, and she supposed that software might follow a similar trajectory as we improve engineering practices. For boilers, regulation was a part of it.

I think the comparison still holds up, over 30 years later.
[1] http://sunnyday.mit.edu/steam.pdf

@vriesk @kenrb Yes! So it is a design choice being made, now, rather than unknown unknowns.

We have learnt a lot in 21 years since the comparative paper on steam engines was published, and attackers make use of new knowledge. Software vendors keep trying to find silver bullets to avoid changing their designs.

Jen Easterly has this right I think in her two-decade-later followup. Incentives have to change.

@blinkygal @kenrb design choice, yes, but as in "process design choice", as there many aspects involved, and software design being one of them, there's also a whole richness of technology choices, and then the choice of due processes as well.

And then there's the problem that the whole software engineering world depends on a huge amount of legacy software that really deserves to be rewritten/hardened. And who's gonna do that?

@vriesk @kenrb Thanks for the thoughtful comment, I've been mulling over what I think here.

Yeah it is a process design choice, with a ton of inertia into using tools that existing software was built with which then propagates the same security properties into new code as well.

And you're absolutely right, there's a whole world of legacy software, most of the large software we use day-to-day falls into that category. Some small things are being rewritten (https://www.memorysafety.org/). But there's vast amounts of code that is not.

Part of me says it doesn't matter for the purpose of the above; users are harmed, thousands of people lose control of their digital lives every year, some end up in jail or at risk in other ways for it, millions or billions are lost in ransoms. The point in some sense is just that this is happening, that vendors are aware of it, and users don't really know what's being given to them and that it was a choice.

Right now there's not the right level of acknowledgement of the problem, in my opinion, outside of CISA and the intelligence agencies. So of course nothing is moving to fix it in a hurry.

If the vendors who are writing on top of legacy software started investing into rewriting it into memory safe languages, or hardening it with tools within the same language, things would start to get better, maybe fast enough that regulation and liability laws wouldn't be needed to protect people.

@blinkygal @kenrb I agree, but I would like to stress out that memory safety is just one aspect of the general software safety/security; on the strictly technical level, there's also concurrency safety (both on the scale of threads and distributed systems), whole aspect of proper authentication and authorization management, and then the more general aspect of logic safety - making sure the tools used are used properly.

1/2