this spring I've been teaching undergrads to use LLM agents. my rationale for doing this was that it would give me a chance to covertly teach lots of real software engineering, which is what I've done.
meanwhile, I've been watching the students closely to try to figure out whether a coding agent is a leveling factor (reducing differences in effectiveness between different students) or an anti-leveling factor (amplifying differences). at this point I'm 99% sure it's the second thing.
@ryan so you know how one student will have a bug, form a wrong hypothesis about it, get on the wrong track, and take a very very long time to track down the bug, whereas another student will just sort of home in on the issue right away?
I feel like it's just more of the same. it's a difference in how effective people's mental models and hypotheses are, and the available tools simply amplify whatever effects are already there.
but I have no hard evidence for anything
@regehr @ryan This matches what I hear from friends in industry.
I hear a lot of stories of projects where someone outsourced thinking to an LLM and spent days or weeks debugging something and not fixing it when applying a bit of human judgment would solve the problem. A friend of mine makes good money as a consultant who cleans up people's LLM-assisted messes.
On the flip side, the growth rate of startups is faster than ever because you can ship at previously impossible velocities.
@danluu @regehr Yeah that matches what I see at Mozilla. If you just want to write code quicker, you can do that.
But if your goal is to write better code, you can do that too. It’s so easy to write fuzzers or exhaustive test suites now. And LLMs are getting really good at finding defects too.
Which way you choose is a bit of a reflection on what you want. Unfortunately most people in our industry just want more code quicker..
John Regehr
Ryan Hunt
Dan Luu