Look this isn’t at all a defense of slop code, but it has me thinking — how much does code quality matter, and why?
It’s maintenance, right? We care about readability because we know we’ll have to make changes, fix bugs, etc.
But so … imagine a codebase that’s magically bug-free and feature-complete. (I’m aware this is a strawman - that’s the point, it’s a thought experiment.) Does it matter if this codebase is well-written? I’m not sure it does! (1/5)
@jacob Responding to the thread as a whole, I think code readability will matter just as much as cockpit black boxes matter.
The universe is against us even with bug-free hardware and software. Even the perfect self-driving car will kill some people. The perfect AI doctor will lose some patients. The perfect automated factory will assemble some lemons.
We will need to be able to learn why and how it happened. Step by step. So, logs and readable code. Otherwise trusting automation is impossible
I was going to say something about accountability, but this is pretty much my thought too.
Especially as it's incredibly unlikely that anyone would one-shot a perfect implementation, even more so for something that is extremely critical.
If we can't understand "why" something went wrong, how do we account for it? Was it negligence? Can someone be held accountable? Is the prompting wrong or is the code wrong?
I can see why that's a dream scenario for a corporation, though.
@jacob I keep wanting a deterministic layer in there somewhere. Letting an AI take over the test suite is the piece that feels riskiest to me. That's always been the layer of trust and confidence.
A steadily evolving test suite that I understand is my best guarantee that things will continue to work at least as well as they have in the past. If I give over control of that layer, I don't know where my confidence that things will work comes from.
@jacob yeah, exactly. Lots of AI coding boosters are talking about the importance of tests, documentation, small modules… On one hand, yes! Great ideas! OTOH why did we have to wait for LLMs to prioritize this? (And as soon as we can avoid it, will we?)
But, I do like the move to smaller PRs, smaller files, more tests. My biggest challenge is that PR review has become a bigger bottleneck.
@jacob Never thought about the provably correct path. I have seen the "who cares what the code looks like when it's only the AI that sees it" though.
But yeah, that's intriguing.
@jacob A thought when reading this: even if it were possible to make something perfect (serving your straw man) by today's version of perfect, we know that definition will change over time (e.g. https everywhere today, but not before Firesheep).
I think there are some parallels here to “I'm not doing anything wrong, so privacy doesn't matter to me.” The definition of “not doing anything wrong" has changed.
Also difficult maintenance via LLM will get *expensive* when it's unsubsidized.
@jacob code quality does not matter *if the code never has to change*. it’s always a forward looking statement: we want easy to understand code because we will have to modify it later.
if later never comes then i agree entirely;
but in my own very limited explorations imho the LLMs also find good code easier to read and reason about…
@jacob Jumping on one specific thing: Rice's Theorem means that any "interesting" program has limits on the extent of static analysis, so "provably correct" is not a thing you can get.
The things that would improve product quality the most are things that provably can't actually be automated in the general case. Those are the limits we have to reckon with.
@janl *sigh*
If having anything positive to say or any curiosity at all about LLMs means I’ve fallen for a scam than ok I guess. I wish there was room for more nuance here.
@frank you can ask and the answer is never.
I do not think evaluating these things based on their merits is ethical.
The tool you are using has been built by stealing* from me** and my friends and it offers to rent their theft back to me, so excuse me when I am not interested.
*and that’s only one of the many issues
**I’d be entitled to thr Anthropic lawsuit money if I were a US citizen
@frank @janl @jacob I'm obviously not Jan, but maybe Jacob gets pushback like that because stuff like "imagine a codebase that’s magically bug-free and feature-complete. " and going on from there to talk about LLMs is practically indistinguishable from the propaganda and grandstanding Altman and Amodei and Musk peddle. Just imagine what would be possible if. Fascinating.
Yeah it's fascinating but so is imagining what we could do if we could photosynthesize. And...
@jacob "If the roof stays up and the floor is dry, does the build quality of my house matter?"
Yes. Yes, it does.
@jacob The view from the top of the mountain is great -- but it feels better (more "accomplishment") if you get there on foot instead of a chair-lift or helicopter.
I at least put non-zero value on "the process" and the methods used. I don't necessarily have a great way to articulate this feeling right now.
In climbing, one may "climb a climb" on toprope, on lead or even free-soloing. These come with different intrinsic rewards, even if the end-point is the same ("alive, and on top").
@jacob I believe some of the more scholarly posts I've read about the value of "trying, and failing" in learning apply here too: that being told the right answer by an oracle machine doesn't lead to the same learning as "doing the work" yourself (i.e. making mistakes, etc).
Everything I've learned about mentoring seems to back this up: if you just tell people the right answer, they do not learn it as well (nor appear to feel the same) as if they fail, then find the answer.
@jacob (...and of course there's some balance: continuing to flail and fail is not going to produce learning either, unless there's a TON of motivation).
Coming back to a climbing analogy, if you try to climb something _way_ too hard for you, it's just going to be frustrating (or fatal) instead of producing accomplishment.
So two "equal" programs (that produce same/similar output for inputs) will still be judged differently (by humans) based on other factors (like "process"). IMO.
@jacob Yes, there's no absolute scale here. Someone finishing something important to them is no less "deserving" of feeling accomplishment.
I'm not sure why I went to "physical" examples, but this applies to lots of areas. I should have kept it about 'craft' (in the widest possible sense) I guess
I do believe that people will feel _less_ "accomplishment" if the thing wasn't "hard" (for them!) though. (That is, climbing the stairs can be just as much an accomplishment as any mountain)
@issackelly @jacob The assembly was replaced by higher level code that still needed to be maintained and understood.
I’d argue that sufficiently-specific and structured prompts will still need to be maintained, too. But with the caveat that it will be run through a fundamentally non-deterministic interpreter.
@jacob @issackelly Yes, which I think is why they’re not suitable for what we’re trying to do with them.
The output of the code it writes might not matter, but consistency of it _is_. I don’t think we can trust them to produce code that retains functionality without a ton more tooling, which you alluded to earlier.
@palendae @jacob Jacob I think your original point is approximately where I am with it.
* Humans own/maintain the specs, and
* have a lot of oversight/responsibility over a test suite.
I don't know that "prompt maintenance" is really where my mind is at with agents + LLMs.
If the spec represents what you want, and the tests represent the spec, then that's the API to the product, like the function body is the API to other functions, what's on the inside can change or be ambiguous.
@issackelly @jacob Specs and the tests would be the prompts here; the instructions we give to a machine to do something. Whether it’s a REPL or a set of files given to agents isn’t really relevant, IMO.
I’m not especially optimistic that the industry will apply much rigor to the testing or the specs, to everyone’s detriment.
@palendae @jacob I dunno. I think specs (formal specifications) and tests (acceptance, regression, property-based, and unit, as appropriate) should _not_ be prompts, but rather barriers code should pass, human written or otherwise. The more complete and accurate they are the more likely you are to get value out of LLM code long term.
LLMs can help a skilled human write them; but humans must own the responsibility for them
"Product specs" can be llm prompts or jira tickets for engineers.
@issackelly @jacob I’m curious - are the specs and enhanced tests what you’re focusing on?
Because at my current job, I’m hearing a lot of people saying this, but the emphasis (and measured outcomes) is around throwing agents at code generation and automated review without bringing the testing up at the same time.
@palendae @jacob For me, in the last ~9 months of engineering; "It depends" -- If it _is_ or _is a dependency of_ an API or security critical code, yes. Both for me and the code I review. If it's feature work, especially experiments, on top; then "wing it" has been useful.
At the moment I am not beholden to anybody but myself, so I feel like I'm sharing the bounds of what seems _right_ not what others are currently doing.
I do feel there is far too much exuberance compared to capabilities.
jacobian
Łukasz Langa
Mario Munoz
Alyssa Coghlan
neoluddite
Glyph
Eric Matthes
Ian Bicking
it's B! Cavello 🐝
ash furrow (still spooky)
Troed Sångberg
Sean Coates
filipa mv
Gonzalo
Christopher Neugebauer
Jan Lehnardt 
Frank Wiles
Adriano
meejah
Andy Dustman
José Albornoz
Issac Kelly
Nolan B