Thanks to everyone who has responded so far.

To focus on a specific definition of "it works", take this post that I boosted:

https://toot.cafe/@nolan/116185451572229163

He has seen coding agents fix bugs with minimal prompting, and it's effective enough that he finds it terrifying. What should we make of that? He's ambivalent, but clearly feels that we should take seriously the demonstrated abilities of these tools, and as a result, he's using them, but not happily. I'm trying to figure out what to do with that.

@matt Apologies for wall of text, but it's a nuanced topic which I am not able to do justice succinctly.

This is hard to give a precise answer to because “work” is not a discrete binary category here. (I suspect this might also be driving some of the disagreements). Personally I am specifically worried about code that compiles and seem to do the right thing… yet have security, reliability or other quality problems that are not caught. Sure human programmers also make that kind of mistake

I suppose the short honest answer for I think I both don't think it works well enough to my satisfaction. Specifically, to me, one of the great things about software is how it's deterministic (well, mostly). Considering that randomness is core how what makes LLMs work well (to the degree they do), I have a hard time accepting that as “working” I suppose.

I think this is interesting, because I suspect a lot of us are looking at the same data, but we're interpreting it differently (think of two researchers categories statistical numbers differently… they have different viewpoints despite looking at the exact same raw data).

I *do* know of devs who are very clear that if it worked as claimed it would not change their opinion a dime. I would like to be believe I am one of those, but I have not been in the situation where I have been genuinely convinced they work well. A lot of aspects of tech are already highly immoral yet I still participate in at least some of that tech (but I also reject some and have made some parts of my tech life significantly less convenient because of it).

Another aspect of why I don't use LLM is neither about its ethics or how well it works, but the fact that I'm apprehensive of the cognitive effects it will have on *me*. Whether it's human language or writing, both of those are useful way to *think through problems* and carefully consider ideas and how they connect. I am not confident that my abilities for abstract reasoning will not be negatively affected by LLM usage, so that is also I reason I refrain.

@matt
Question for people who choose to use generative AI: Do you make that choice despite accepting the growing evidence that it incurs frankly terrifying external costs? Or do you use it because you ignore those costs *and* believe it actually works?

I have yet to see an application for whatever you want to call these "AIs" where their performance is worth the cost. Yet for some reason that makes me an ethical purist in an ivory tower who ignores "growing evidence" [citation needed] that "gen AI" works for anything but the most trivial cases. Meanwhile, "AI" boosters are allowed to flout ethics, flout externalities, and flout logic, but you don't accuse them of "belief" in something contrary to evidence. Why is that, I wonder?

If you're genuinely interested in asking the question, you need to rethink your phrasing. If you're not, you might still want to rethink it because it smells funny. Hope that helps! 🙂

@matt many self reported claims of something working is not the same as evidence of it working.

AI research is (obviously, the field is evolving fast) lagging far behind. Even ignoring external harms, it's unclear whether it causes personal long term harm and what it means for the maintainability of projects.

It's entirely possible that it causes *important* skill atrophy (every new tool causes skill atrophy of some kind, many are often irrelevant). Of course last year's studies on this topic doesn't apply to the current models. Just like studies done on these models won't apply to next year's models. But patterns are appearing.

It's also possible that large projects where everyone uses AI heavily won't have anyone understand the details anymore, only the broad designs. Of course an AI can always explain those to you or you just regenerate them with a bigger model in the future.

But all that said... If we had a performance enhancing drug that allowed ppl to be x times more productive, would we really be that uncareful with it just because it's chemical makeup hasn't been restricted yet?

GenAI is basically a nonchemical drug (listen to the LLM maximalists, not me), and I am worried about heavy users, I am worried about companies forcing employees to use them everywhere, I am worried about ppl getting addicted to them and frying their brains (burnout as a service)

And at this point we haven't even talked about the damage all the mis-use (be it malicious or ignorant) has caused or the training causes to the world and to ppl in the training mines.

@matt I have staked out the position that I will do my own thinking. The hard work, the blunders, and the late nights will be the point of it all. When I mess up, I will not blame it on "AI", but at the same time it cannot claim credit for my successes.

Since I'm a Colorado guy, I'll give this analogy: it's like people who drive their car on the road up Pikes Peak, and then say, "I climbed Pikes Peak."

@matt I see growing evidence that it does, & I see growing evidence that it doesn't.

I don't really care either way.

@matt How high is the bar for "works"? Code can compile and be a security nightmare, for example.

Relatedly, how long must it "work" for it to count? Do you have to be able to maintain the software?

Is the code still licensed properly? Did it "work" legally?

Did the chatbot give you mental illness which interferes with your ability to discern reality, and therefore to tell whether anything works?

@matt

Firstly, the ethical problems are so vast that whether it 'works' is irrelevant.

Secondly, I have never seen or imagined a use case that made the technology interesting for me to use.

To me, it is horribly unethical AND uninteresting.

So umm… all of the above?

@matt I've used it and I've seen it work - it clearly has value. But it also has a cost - to creators of original works, to the environment, and to society when megacorporations are monopolising them - which I'm not willing to pay.

Local LLMs are better on the cost front and sometimes satisfy the value side of the equation, so I'm more forgiving of them. But overall, to answer your question, I'm the former, not the latter.

@matt GenAI is incredibly young. The fact that we're having discussions about if "it works", shows the software development industry doesn't think beyond the time frame of the proverbial goldfish's memory.

We're celebrating the disappearance of junior devs, the validity of lines of code as a metric, and the viability of reviewing instead of writing code and we call that: "it works!"

@matt this decision is ongoing, but personally I think about it with a matrix like this one:

- risk: what harms might I personally face?
- reward: what benefits might I personally accrue from using it?
- externalities: what harm might I be doing to others as a result of using it?
- systems: what harms (or benefits!) might develop as a result of *everyone* using this tech in the way that I am?

@matt I reject it for ethical reasons, the same way I avoid shopping with Amazon.com for ethical reasons more than any pragmatic reason.

Is Amazon often the cheapest, fastest way for me to acquire a thing? Yes. Is it also a terrible company? Yes. If I can acquire something in another way, I look elsewhere. (I am not perfect about this.)

Even if you could show me evidence that, somehow, generative AI produced 100% accurate code or text, I'd still be against using it on ethical and social grounds.

Do I use some LLM-driven software? Yes. I use some local models for transcription. Do I double-check the results? Also yes, because I cannot be 100% sure its results are accurate, and I don't want a fabricated quote winding up in an LWN article.

Might I use LLM-driven stuff at some point for grammar checking? I already use LanguageTool, so ... maybe?

But purely generative AI stuff... I have too many ethical, social, etc. qualms against it to make it part of my work even if I was confident it was 100% accurate. (This is not all, strictly speaking, ethical - I also have qualms about its impact on FOSS development from many other angles, such as increasing the velocity of PRs and putting maintainers under even more stress.)

I also, currently, reject it partially out of spite/stubborness -- there is far too much "peer pressure" and pushing to accept it. I cannot claim this is a logical stance, but when this much money is being spent to push something, I feel like somebody should be pushing the other direction. I'm just dumb enough to be that somebody.