To be 100% clear: I think both positions are in error. We have decades of evidence to suggest that theoretically being able to write memory-safe code with C/C++ does not prevent dangerous bugs in production code. The core design of the tool does not lend itself to safety, and now we know that organizations either can't or won't put in the requisite work to make it safe.

We have every reason to expect the same of LLMs. Even if guardrails mature around the models, and even if code correctness—such as it is, based on insecure code in the training corpora—improves dramatically, the requisite safety apparatus will yet again be ignored, deferred, downplayed by organizations who see security as a barrier to shipping. And because the tool in and of itself does not tend toward safety, its manifestations rarely will.

An engineer designs an oscillating fan that spins quickly and moves a lot of air. A designer puts a cage around it so nobody gets mauled by the expertly-engineered machine.

In this realm, we think too little of design.

@nakal @realn2s So, let's assume for the moment that the developers who are creating real applications with these tools are not suffering a mass delusion, and that they are not "useless." But indeed, that those developers are using "advanced toolchains" to ensure model output meets certain standards.

Do you see the congruence when they say of these toolchains, "You just need to know and use them?"

@delta_vee Okay we are talking about two different things. I am not talking about prompt injections, which is absolutely an unsolvable problem from a structural perspective. There are, however, controls available about what context is added—but again, discipline is the issue.

As for generated code correctness, discipline of process looks fairly similar to what you'd want in a normal SDLC, but refactored for agentic work. Yet again, the trick is discipline. Any shortcut, any change without verification/attestation, anything unreviewed for security, becomes a failure point. Which brings us back around to the original thesis: if it's theoretically possible to do it right, but practically impossible, then you have an unmitigated liability.