David Beazley

So, I wonder which is harder to understand?

In one corner: A million lines of code from a CMM Level 4 organization generated by Rational Rose.

In the other corner: A million lines of code vibe-generated by four levels of agents.

LET'S GET READY TO RUMBLE!!! .... Ding!

Feb 26 at 11:34amWeb
Show threadDavid BeazleyFeb 26

Hearing about developers becoming exhausted by reading the output of LLMs has me wondering if it's exposed the fallacy of that whole "programmers spend more time reading code than writing it" trope.

No, they don't. And they never have.

Oh, sure, developers spend a lot more time "seeing" code than writing it. I'll give you that. Kind of in the same way that someone looking out the window while riding on a train is going to see far more gardens than they'll personally ever plant.

Show threadDavid BeazleyFeb 26
Sometime people ask me why I don't want to do any more "expert witness" gigs. The main reason is that it was entirely focused on reading code. Actually *reading* code that I didn't write and trying to understand what it was doing. It was exhausting as all hell. No. I don't need that kind of day-to-day stress.
Show threadDavid BeazleyFeb 26

This seems interesting and relevant to my exhaustion comment.

Source: https://jenniferplusplus.com/what-is-a-token/

Show threadDavid BeazleyFeb 26

If I install a third party package on a project, I'm probably not going to look at its code at all. Similarly, for a coworker, I'm probably not going to look all that closely at their code either.

In both situations, I'm starting from a place of "trust" which is very different than starting from a place of "suspicion."

In any case, having to keep an eagle-eye on everything all of the time doesn't seem very enjoyable to me.

Show threadRobert BrewerFeb 26
@dabeaz This is all I've really ever wanted from a job (mostly lucky to have found it). It takes work to maintain a high-trust environment, the most important and difficult being fighting off the religions that want to turn you into a low-trust environment.
Show threada wandering happenstanceFeb 26
@dabeaz it’s a very libertarian mode of victim blaming. “Harmed by a bug in someone else’s code? You shoulda reviewed it. Poisoned by salmonella in your lettuce? You shoulda had it tested. Chump.”
Show threadDavid BeazleyFeb 26
@joXn Those who don't heat their lettuce to 165F get what they deserve to be honest.
Show threadMarcos DioneFeb 27
@dabeaz in my recent experience, we do read a lot of code, based on all the cross reviews we do. This allows us to build that trust (or lack thereof) more rapidly.
Show threadOmar AntolínFeb 26
@dabeaz I can understand that, it definitely sounds exhausting. But hey, at least you got one the best programming talks ever given out of that kind of work! You know what? I'm gonna go rewatch it right now. https://youtu.be/RZ4Sn-Y7AP8
David Beazley: Discovering Python - PyCon 2014

Speaker: David BeazleySo, what happens when you lock a Python programmer in a secret vault containing 1.5 TBytes of C++ source code and no internet connectio...

YouTube
Show threadOmar AntolínFeb 26
@dabeaz I finished rewatching it and confirmed my opinion: it's a great story very well told and there's lots of wisdom sprinkled throughout.10/10, will watch again.
Show threadOwen NelsonFeb 26
@dabeaz I'm impressed that people are asking this
Show threadChristopher NeugebauerFeb 26
@dabeaz that naur paper I've been ranting about lately suggests that the most important act of programming is *modification* -- the understanding lies in what the program does, rather than just how it's written
Show threadCalebFeb 26
@dabeaz I have to do a lot of reviews of ai-assisted changes in an older codebase where I wrote a lot of the code. Surprisingly, but in hindsight understandable, the coding agents mimic the style of the surrounding code when they make changes. So it turns out the code sometimes feels like mine. I didn't expect that.
Show threadWilliam BlackerbyFeb 26

@dabeaz This is one of the things that perplexes me about LLM coding. It seems like the easiest way to read your ("your"? if it's LLM-generated) code is to write it in the first place.

As https://bodil.lol/parser-combinators/ by @bodil says:

"highly recommended that you start a fresh Rust project and type the code snippets into src/lib.rs as you read (you can paste it directly from the page, but typing it in is better, as the act of doing so automatically ensures you read the code in its entirety)."

Learning Parser Combinators With Rust : Bodil dot lol

Show threadI ForgetFeb 26

@dabeaz Maybe it would be helpful for us to have more clear language around kinds of reading. When my humanities friends read an article they'll skim intros and critically read analysis, but a close reading can spend much longer on a single passage.

I skim a ton of code. I critically read less, and I closely read even less. I don't think it's contradictory to say both that well-written code can help each level of that, and also that closer readings take more work regardless of style.

Show threadArtyom BologovFeb 26

@dabeaz novelty of the code and shared patterns might play a role. With others’ code, you read something written by people you know and work with, or, in case of Open Source, by people you share idioms and contexts with. This is possible to get used to and stay on top of. Even better: this context stays in your head to use and reference later!

It’s not possible to get used to LLM output, because it’s stochastic and often overwhelming in volume. Nothing to ground oneself in, nothing to learn, nothing to feel belonging to.

Show threadɗ𐐩ʃƕρʋFeb 26
@dabeaz I suppose it all depends what you mean by "understand". For example, just knowing that the code is generated allows me to perfectly well understand that there is no intention behind it, I don't have to actually look at the code for that.
Show threadTim ChaseFeb 26

@dabeaz

One should reliably (re)produce the same output from the same input every single time.

The other has an internal/hidden "temperature" knob that changes the output every time it's run.

I know which I'd prefer to try to understand & debug, especially over the long term.

Show threadnobodyzhome Feb 26
@dabeaz I’ve had to use rational rose, it was not fun. But its code looked more “canned”. AI slop looks more wishful.