@krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek we have been claiming for decades that we are not just educating coding monkeys, so it shouldn't really matter that LLMs can now do all the coding. As far as I see it, it's still necessary to identify and clearly formulate verifiable requirements and specifications, come up with a modular design, and verify the whole thing, because I still believe the ultimate responsibilty lies with the developer. So students still need to understand the fundamentals. But yes, it has become much harder to check *at scale* whether they actually grasped them.

@GeorgWeissenbacher @[email protected] @jfdm @csgordon @lindsey @jeremysiek
Yes to most of that. I think it's not that hard to assess if that is what people were always assessing that.

I actually disagree w/ your opening comment. Most intro CS educators will say (and have said), "I don't teach programming, I teach *problem solving*" (whatever the fuck that is). My response is, "great, this should be your liberation! Programming got easy, what are your «problem solving» ideas?"

@shriramk @GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek ... did programming get easy? Can one be said to be programming if one asks someone else (or an LLM) to write a program for you? Or is some other kind of (not- or not-quite-programming) interaction going on?
@tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @lindsey @jeremysiek
I very much think of what I'm doing with Claude Code as a kind of programming — indeed, the kind of programming I always wished I could do! But if it makes you happier to use a different term for it (not "vibecoding", that has too many specific connotations and is definitely not how *I'm* doing things), and it's *useful* to have that other term…that's fine by me. I guess my slogan is: "Philosophy…but not too much".
@shriramk @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek This comment made me realize something about myself: this is *not* a kind of programming I always wished I could do. I really only like programming because I like manipulating formal systems. That might explain a lot about why this kind of programming doesn't appeal to me, aside from all the bad externalities.
@lindsey @shriramk @tonyg @GeorgWeissenbacher @jfdm @csgordon Same. I got into CS because I liked craft and determinism. Never aspired to write more code quicker. Software is excessively buggy and voluminous already!
@aarbrk @shriramk @tonyg @GeorgWeissenbacher @jfdm @csgordon Just to be clear, I would never in a million years say that Shriram doesn't care about "craft". (I don't know what his stance is on determinism.)
@lindsey @aarbrk @tonyg @GeorgWeissenbacher @jfdm @csgordon
Well, if people who don't know me want to jump to absurd and insulting conclusions about me, I guess that's what social media are designed for. 🤷
@shriramk I did not make any statement about you, who I also do not know, but I am sorry to have apparently insulted.
@lindsey @shriramk @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek another issue (at least for me) is it's far easier to write code than it is to read somebody else's code, and i think ppl using llms to write code are optimizing for the wrong kind of thing (also negative externalities aside, carbon tax wen)

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Yes, I'm sure people like me know so little about software that we're all "optimizing for wrong kind of thing". 🙄

You do know how absurd these kinds of remarks look, right?

@shriramk @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek oh apologies, I didn't mean to infer that you were optimizing the wrong thing, I guess I was refering to the vibecoding community

im also not saying that what you're doing is vibecoding either, my remark was more for the ppl using llms to vibecode

i should have been more specific 🫠

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Implicit apology accepted, but I still don't agree with you. The vibecoders aren't out to read any code at all *by definition*. So who cares whether the code is readable?

There are lots of tasks that computing can help with. I keep describing programming as a "superpower": there are things you wouldn't even do if you didn't know you about programming. As an example: ↵

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
In my courses, I make up all kinds of wacky grading schemes for assignments, because I have full knowledge that come semester's end, I can figure out how to turn these into grades — using code. Sometimes it takes me 2-3 hours to wrangle, but I *know* can do it. Most profs don't have that power, so they just use whatever crappy grading scale Canvas or Gradescope or whatever give them. ↵
@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Likewise, I teach my students SQL and Unix commands because they are another kind of superpower. Even if you know to code, trying to write what a 5-Unix pipe does could take a half day of Java, so you would never get started; whereas you'd bang out the code on the shell and get on to your next task that depends on that output. ↵

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
If, now, *everyone* can have that superpower within reach — why are we not all *CELEBRATING*?!?

We should be using out knowledge and strength to figure out how to enable people to do this safely and responsibly, not acting as high priest gatekeepers like the men standing around IBM mainframes locked away in "computer rooms", dressed in such a way you can't tell if the photo is b&w or color.

@shriramk @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek >why are we not all *CELEBRATING*?!?

ik this is a rhetorical question, but id have to do some introspection to answer this, apologies \:

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Please do, I welcome it. Hearing thoughtful other views helps me sharpen my thinking.

To be clear, I see problems too. But at least *some* of those problems have been around from the dawn of time: arguably the whole field of software engineering (beyond PL) exists because "humans always fuck up, how can we help them".

Of course, speed, scale, etc. are all different, and there are subtleties like copyright. ↵

@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
To me, a useful thought experiment is, "What is a line in the sand behind which we are not likely to go back?" That would be some combination of an open-source agentic loop and open-weight model, running on reasonable modern hardware. Those all exist or can easily exist. That doesn't require trillion-dollar investments, destroying the earth, etc. Can we not do a lot with that? ↵
@nemo @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
At that point, you also aren't externalizing impacts. If you use a lot of power, it's on *your* machine and *your* power bill. No amount of subsidy is going to help with that. (I sometimes run ollama on my laptop and think "whoa, this is taking a while, that's why they want me to run things on their cloud".) So it's reasonable to answer (not entirely rhetorical) questions against that baseline.

@shriramk To me, the answer of "why [am I] not celebrating" is that it feels like LLMs are an abjuration of the idea that we should care about what we're working on; that the question of "so why or how does it work" can and should be safely delegated to the machine and we go on our merry way, filling an ever-diminishing gap.

In my personal experience it's very easy to beget software with an LLM that I don't understand. I find it really shocking to take a project I made with an LLM originally and then start coding in it by hand - it's like I'm using a library whose documentation I read, but whose code I've never worked with, which is... imo quite accurate. There are alternative workflows for using LLMs, but those also get rid of a lot of the productivity improvements at the same time so it's hard for me to see the point.

A colleague of mine, who works in physical modeling, said this:

> Trying to help out in such a situation [where there's an issue in an agent-built model] can be a uniquely frustrating experience since you're not actually talking with the brain that built the model, there is now a clueless human in between. "I don't know, the agent did it" as an answer to each and every question is a clear indication that the human modeler has outsourced too much of the understanding to the agent.

I see a lot of LLM systems where the author doesn't in any meaningful way understand how their system works and feel like ultimately this is a step backwards in how well people understand the software and overarching systems that they're building. The answer the community seems to be moving towards is ultimately that this doesn't matter, that you don't need to understand how any of the systems work because the agent will deal with that for you, but I personally find this an extremely unappealing and objectionable outlook.

@ckfinite Yes to all that, but I also don't understand how a lot of the code I work with works, and I struggle to understand my *own* code from, say, six months ago.

There are certainly programs I've written that were t the very, very edge of my mental powers when I wrote them, that I now just take on faith, and *others* also take on faith. ↵

@ckfinite Most code isn't that important or difficult or interesting. A lot of code is just reproducing things in the "thick of the distribution". I think it's great if that can be made quick and cheap and easy to produce.

As someone who's spent a whole lifetime caring about meaning and correctness, I can separate that from stuff that Really Matters. (Of course, *many* things are "mission-critical", it depends on one's mission!) ↵

@ckfinite We've never before had the opportunity to ask "If code were cheap to come by, what are the interesting problems?" That was a sci-fi question until now. Now that it's some approximation to reality, I want to think about how we can move up the reasoning chain. The need for meaning and correctness don't go away, but maybe they can be tackled in different, new ways.
@shriramk
> As someone who's spent a whole lifetime caring about meaning and correctness, I can separate that from stuff that Really Matters.
What gives me pause about this is that frequently the stuff that I didn't think really mattered - the unknown unknowns, if you will - ended up really surprising me. It's the loss of the bits of friction I didn't expect that makes me worried about the ability to maintain context long term while using LLMs.

@ckfinite Both points are valid.

But I feel a bit like how I feel about being contrarian (while not being truly contrarian) re. self-driving cars: *have you seen the alternative*? I think we often construct mythical baselines, when all the time I've spent studying humans and understanding the literature on humans leaves me way, way less optimistic and more despondent about that baseline.

@lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
You were formal about some things and utterly informal about other things. Where, for instance, did you document your software architecture, your temporal properties, your concurrency requirements, etc.? I'm going to guess a whole bunch of things were *not* captured by your "formal systems" and were left to documentation, comments, hopes, dreams, and aspirations. ↵
@lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
This is a classic PL community flaw: we think in terms of languages and ignore the rest. I know you are better than most in these regards, but your programs also lack several formalizations. (To be fair, I too suffered from this flaw until Gregor Kiczales kindly beat it out of me with a few pointed remarks.)

@shriramk To me this is an odd (and, frankly, extremely personal-seeming) attack. It is possible to have a formal system that one enjoys manipulating (a subjective assessment) that is limited in scope or does not capture other aspects of the system. I, fundamentally, do not think that you must formalize your entire world in order to pick parts of it off into interesting and fun abstractions that can be manipulated into and of themselves.

I read this as arguing that Lindsey's personal, subjective, enjoyment was invalid due to not having a complete formal world model, and I ultimately think that this isn't a valid position. You might not have the same opinion, but I find it bizarre to argue that the very enjoyment is wrong.

@ckfinite Nowhere did I question Lindsey's right to, or sense of, *enjoyment*. Nor would it make any *sense* for me to say "no, you didn't really enjoy the thing you said you enjoyed". I'm pointing out that Lindsey was already informal in various ways, and PL people have a particular blind spot about how formal/informal we really are (as in, we think we're super-formal when we're actually only 1-2 degrees more formal than the entirely-informal).
@shriramk @ckfinite I am, indeed, informal in various ways! In fact, I just added it to my Mastodon bio in case there is any danger of confusion on that point.

@shriramk To me, I read quite a lot of the terminology you're using as being very specifically judgemental, and when combined with the second person framing it becomes very personal.

When contextualized with a statement about a subjective position, about a personal feeling, it comes off as either:

* A value judgement of the feeling that she's expressing in and of itself; that it is wrong to feel enjoyment from a partial formalism and that your viewpoint is categorically better, or
* A category error where you're making a point about the community come off as a judgement about a person.

I agree with your point about the community, but think that the way that this is framed is coming off very strongly as the former where your response to me was that you were intending the latter.

> as in, we think we're super-formal when we're actually only 1-2 degrees more formal than the entirely-informal

I would contrast this answer with what you said earlier

> To be fair, I too suffered from this flaw until Gregor Kiczales kindly beat it out of me with a few pointed remarks.

to imply that no, you are different, you are entirely formal because you are disclaiming yourself of the flaw.

I'm not different; I'm very informal. These days I work mostly in physical simulation and embedded control systems, and there's tons of informality in both settings. What I think that those domains illustrate, though, is that even for a "fully formal" software system, where the architecture, the temporal properties, the concurrency requirements are all fully specified large parts of the system aren't and go quietly unnoticed.

A classic example is Therac-25, where the race condition was between the speed the user typed at, the UI state, and the speed the machine's turntable rotated at. A more modern example would be rowhammer. Even in systems where everything's formally verified and proven down to RTL you still have informality in the analog domain.

@ckfinite Thanks for this critique. I definitely did not mean to make this *personally* about @lindsey . Since I have given that impression — sorry, Lindsey. I always think of Lindsey as one of the "good eggs", so I *especially* don't mean to call her out personally.

I do mean to call out the PL community broadly, though. And I say this as someone acknowledging my own flaws in this regard. I'm not sure how you read my Gregor comment as me saying I am "entirely formal", but probably no matter.

@shriramk @ckfinite I did think your comment was a weird non sequitur, Shriram. I made an observation about myself and what I like about programming, and you responded to that with a broad critique of the entire PL community, with me as a proxy for it. I'm not even sure that what I said is even really an example of the community-wide tendency that you're critiquing. It seems more like you had something queued up to say and I accidentally pushed the particular button that made it come out.
@lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
One thing I've learned over the past few months is that there seem to be two kinds of "programmers" even in the very rarified space of "profs with a PhD in PL/FM/…": those who really like code and those are indifferent to it in light of other aspirations. Until now we had no way of telling ourselves apart. Now we do. It's been really interesting, revealing, and fun to see this split.

there are maybe a lot of different dimensions here i think. personally one of the main reasons i do a lot of programming is motivated by dislike for how programming feels to me now versus i i feel it could feel, and my desire to make systems which feel different. my main personal interest is in ways of interacting with computing that are more spatial and tangible, which led to me initially disliking chat-based approaches, which seem to be taking me farther away from the goal state.

i've basically since done a 180 on this as the combination of increased model capabilities and refining my processes have led to a point that, despite not being what i want at an immediate level, the 'new programming' feels like a more happy intermediate to me than the old way of doing things. it seems to allow me to interact with code at a bit closer to the conceptual level i wanted to, even if the cost is a layer of intermediation that feels very different than i imagined. it still feels very much like a k step forward m steps back situation when k and m are very rapidly in flux, even as the ratio has been trending mostly positive.

@shriramk @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek

@disconcision @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Personally, I'm excited that I can use this weak-ass text interface to try to build the medium I wish I had!
@shriramk @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek i'll admit that recent developments have given me pause about whether structured editing qua structured editing is worth pursuing, even as my experiments in this space have been significantly accelerated. but then i remember that structured editing was already utterly quixotic
@shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek based on available evidence so far (I'm enjoying the hell out of LLM-assisted programming) I'm not as much of the first kind as I had previously supposed.
@shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek like Csmith had sort of slowly decayed over the last 12 years and last weekend I went and did a bunch of much-needed cleanup and modernization. would I have done that w/o LLM support? absolutely not
@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
This is the key. The things that were never being done. Maybe it's a prof thing? I remember having all the time in the world to code, as a grad student, and now I have virtually none, and building the things I want involves not just thinking but reading 50 APIs and using 5 languages with 3 syntaxes and 2 partridges in a pear-tree.

@shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek right, I hadn't realized how bottlenecked I was by lack of student + my own time. like I have one million ideas and now I can pursue 0.0007% of them instead of 0.0003%. or whatever.

LLM coding is perfect for profs-- we're time-limited experts and we're mostly not expected to produce really awesome code anyhow.

@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
In particular, it's letting me make a lot more progress on my 1/10th-baked ideas, helping me take them to at least quarter-baked ideas, before I ask a student to think about them. I've definitely had situations before where the lack of that prototyping meant I would give a poor, unsuspecting student some "brilliant" idea and a semester later find out it didn't go anywhere. ↵
@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
I feel like I can be a more responsible advisor now — at least giving it a whirl on my own time and seeing how it pans out. If I'm not willing to do even that, then I certainly shouldn't be foisting it on a student, for whom failure is infinitely more problematic. ↵
@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
One "problem" I've had is that the prototypes emerge so quickly that the blocking function is my thinking. I'm working on a new PL/medium(!) that I've never been able to interest a student on; and I've quickly hit the limits of how far I'd thought things through. Now the repo tab just sits there in my shell staring at me accusatorially, whispering, "Okay, what next, smart guy?" Gulp. ↵

@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Papert, Kay, etc. used the phrase "tools for thought". (@tonofcrates is teaching a course by that name!)

This feels like a new tool-for-thought. Including exposing both the weakness and incompleteness of my thought, which is what a good tool ought to do.

As a PL person, I'm excited to be able to rapidly prototype a PL and actually *use* it, not just reason through calculi (a different tool for thought).

@shriramk @regehr @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek @tonofcrates i've had a similar experience, but have found it really awkward to use these tools -- what am i supposed to do in the ~(small. number of minutes) during which claude is doing stuff. think? follow along? mostly i go and check mastadon lol

@jbigham @regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek @tonofcrates
I really like Jakob Nielsen's summary of 30 years of UX research on 3 levels of interactivity:
0.1 second = instantaneous
1.0 second = uninterrupted
10 seconds = keeps attentionª
ª still true?

Anything beyond that, and your brain wanders. These things have the *worst* response time, returning in minutes rather than a day.

That's why we have *three* social media.
https://www.nngroup.com/articles/response-times-3-important-limits/

Response Time Limits: Article by Jakob Nielsen

How users react to delays in a user interface, whether website or application. The 3 main response time limits are determined by human perceptual abilities.

Nielsen Norman Group

@shriramk @regehr @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek @tonofcrates

i think the big opportunity, which is hard, is how to usefully keep people attending with these things. i'm starting to see work where humans are still in the loop. but, that requires a lot more focus on updating the human's mental model (need way more than just following along the chat), and then how would you usefully intervene?

@shriramk @regehr @lindsey @tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @jeremysiek @tonofcrates regardless, most of the steps are pushing on the 10 second mark, and when they are just coming in sequence every 5-10 seconds, it's super boring. so many of hte updates are meaningless anyway, how many synonyms for "thinking" do i really want to see flash on the screen lol

@jbigham @regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek @tonofcrates
To me a really exciting open research question is how can we keep humans in the loop in a way that their work is
- minimal/moderate
- meaningful

Either one is easy. Security alerts were minimal, but useless because not meaningful. "Review all the generated code" is (maybe) meaningful, but not moderate.

In specialized sub-domains we have some answers. Don't yet know how to generalize.

@shriramk @regehr @lindsey @GeorgWeissenbacher @jfdm @csgordon @jeremysiek @tonofcrates That's so interesting that you see it as a tool for thought. To me, it has felt predominantly like a tool for escaping thinking. One delegates the thinking. And then, having not done the thinking, has trouble understanding the result in anything other than a superficial way... But it's definitely a tool for achieving some effect in the world! I just wouldn't characterise that effect as "thought". Perhaps it's a kind of lever/fulcrum: some kind of force or distance advantage, more like a (mechanical) machine than like a mental augmentation??

@tonyg @shriramk @regehr @lindsey @GeorgWeissenbacher @jfdm @csgordon @jeremysiek

A simple, albeit terribly incomplete, analogy is that an LLM is a fancy search engine. Like any search engine, it can hinder thinking (e.g. searching for solutions to a homework problem, plagiarizing sources for an essay) or support thinking (e.g. facilitate debugging, surface unexpected sources).

@tonofcrates @tonyg @regehr @lindsey @GeorgWeissenbacher @jfdm @csgordon @jeremysiek
As someone who thinks about metaphorical and analogical thinking, I'd say this is a terrible analogy in general and a fantastic analogy in this specific case and for this audience. (-:

That is, it fails very badly as an analogy for understanding mechanism, but it works very well as an analogy for understanding use/effect. I hadn't really thought about that distinction before: analogies for purposes.

@shriramk @regehr @lindsey @tonyg @GeorgWeissenbacher @jfdm @csgordon @jeremysiek this!

I’ve been exploring language design (a language I’ve been brewing for 30ish years)

I’ve been exploring a new operating system (first bring-up on hardware yesterday)

The flexibility of “yeah, this is a dead end, try that” is liberating

@regehr @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek
Over the past 3 months, I too am finding out I'm far less of the first kind than I thought. I don't know whether to be happy or sad about that, as someone who literally used to describe his job to others as "designer of programming languages".

@shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek it is what it is, and I suspect we'll all feel differently (not sure how though) once the technology stabilizes and the novelty wears off.

but I am sure happy at the prospect that I might never have to fight with tikz, cmake, or some fiddly LLVM API the hard way again

@regehr @shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek but all of the above-mentioned problems—

-APIs too fiddly (bc you don't have any say in which development efforts are funded)

-student-to-instructor ratio too high (bc you don't have the power to set limits & force hiring of instructors to match enrollment)

-not enough time for interesting research/experiments

...could be better solved with a union & no LLMs, than with no union & unregulated LLMs.

@regehr @shriramk @lindsey @tonyg @GeorgWeissenbacher @[email protected] @jfdm @csgordon @jeremysiek Regarding the two camps: I believe there also is a temporal aspect. Say, 110% in the code camp as a grad student and assistant professor, and maybe one grows a little bit out of it later on.

LLM-based coding allows me to do much more prototyping and playing around with new ideas that I had written down into my notebooks over ten years ago. So the alternative would just be to have nothing instead...