John Regehr4d ago

this spring I've been teaching undergrads to use LLM agents. my rationale for doing this was that it would give me a chance to covertly teach lots of real software engineering, which is what I've done.

meanwhile, I've been watching the students closely to try to figure out whether a coding agent is a leveling factor (reducing differences in effectiveness between different students) or an anti-leveling factor (amplifying differences). at this point I'm 99% sure it's the second thing.

Show threadRyan Hunt4d ago
@regehr do you have an example for why that is? I’ve been wondering that myself and leaning towards (2) as well, but it was just a gut feeling with no evidence.
Show threadJohn Regehr4d ago

@ryan so you know how one student will have a bug, form a wrong hypothesis about it, get on the wrong track, and take a very very long time to track down the bug, whereas another student will just sort of home in on the issue right away?

I feel like it's just more of the same. it's a difference in how effective people's mental models and hypotheses are, and the available tools simply amplify whatever effects are already there.

but I have no hard evidence for anything

Show threadJonathan Yu

@regehr Sounds like you have the makings of some kind of experiment that you can run on your students...

@ryan

Apr 2 at 4:54amWeb