Matt CampbellMar 6

Question for people who choose not to use generative AI for ethical reasons: Do you make that choice despite accepting the growing evidence that it works (at least for some tasks, e.g. coding agents working on some kinds of software)? Or do you reject it because of the ethical problems *and* a belief that it doesn't actually work?

I'm thinking that principled rejection of generative AI might have to be the former kind, *despite* evidence that it works.

Show threadGlyph

@matt this decision is ongoing, but personally I think about it with a matrix like this one:

- risk: what harms might I personally face?
- reward: what benefits might I personally accrue from using it?
- externalities: what harm might I be doing to others as a result of using it?
- systems: what harms (or benefits!) might develop as a result of *everyone* using this tech in the way that I am?

Mar 7 at 5:31pmWeb
Show threadGlyphMar 7

@matt right now the evaluation I have of that matrix is:

- risk: AI psychosis, huge amounts of wasted time, skill loss, social credibility loss, gambling addiction, dependence upon technology that will increase rapidly in price very soon
- reward: maybe it could help me write some code a little bit faster? evidence is very weak even if sentiment is strong here
- externalities: water use, power use, plagiarism, spamming others with low-quality work
- systems: model collapse, financial collapse

Show threadMatt CampbellMar 7
@glyph That's a reasonable way to look at it. I think it's easier to argue against using these things if one can point to significant, demonstrable personal risks. If the rewards are stacked and the only counter-arguments are externalities and systems, then abstinence is a much harder sell.
Show threadGlyphMar 7
@matt the big problem with the personal risks right now is the lack of any credible safety story from the model vendors. as far as I can tell, we don't *know* what causes AI psychosis. there's some vague correlation with "sycophancy" and maybe they've figured out how to turn that down, but maybe not? we don't know how much skill loss is real. we don't have demonstrated best practices in place.
Show threadGlyphMar 7
@matt like, I think that people are far too nervous about nuclear technology because we actually know how that stuff works, we know how to measure dosage and harm and risk, and "ooh, spooky nuclear" is a vibe and not a risk calculation. but the opposite is true for AI systems. we're seeing these wildly dangerous outcomes and then people kinda yadda-yadda-yadda over "best practices" without ever saying what those practices are or providing validated evidence to support them
Show threadGlyphMar 7
@matt maybe the risk is very low! but if a guy uses a model to help summarize actuarial tables, goes crazy and starts calling himself a Star Child, and the response from the vendor of the product that arguably did this to him is "well, he probably had some family history of schizophrenia or something" when he was over 40 (WAY past the age where a disease like that generally presents) and that family history also doesn't exist, well, it's concerning that they still want me to use it
Show threadMatt CampbellMar 7
@glyph Wait, did that actually happen? I mean, the guy calling himself a star child?
Show threadMatt CampbellMar 7
@glyph It's easy to get the message, not only from boosters but also from reluctant users like Nolan (whom I boosted and posted about elsewhere on the thread), that the rewards are so great and undeniable that one would have to be a saint to not use the thing just because of the externalities, and, you know, none of us are saints like that when it comes to other problematic things.
Show threadGlyphMar 7
@matt that is a vague pastiche of a few different stories since I can’t check sources right now but it’s not too far off the mark. this comes from an outline of a post I am writing and I have like … a thousand citations to keep track of
Show threadGlyphMar 7
@matt as far as the benefits … I know that it is making people feel high, and *maybe* the latest Claude models specifically are just so much better at software than any of the previous six times that somebody said that “this is it!!!”, but we still haven’t seen any real hard evidence in the form of monetary ROI. the one company we know is leaning hard into LLMs for everything, Microsoft, seems to be having a historic number of bugs and outages
Show threadGlyphMar 7
@matt here's the "starchild" story specifically: https://www.vice.com/en/article/chatgpt-is-giving-people-extreme-spiritual-delusions/
ChatGPT Is Giving People Extreme Spiritual Delusions

You may be aware of the fact that ChatGPT suffers from hallucinations. Now some of its users are suffering through similar delusions.

VICE
Show threadMatt CampbellMar 7
@glyph I would love it if Nolan, and no doubt other developers as well, could have the relief of finding out that what seemed to be the "terrifying" effectiveness of current coding agents was in fact an illusion, with the risks (ideally to both the developer and the business/project) outweighing the rewards.
Show threadMatt CampbellMar 7
@glyph And then I could continue to not use coding agents without the FOMO.
Show threadGlyphMar 7
@matt I wish I could provide that, but my only real insight is that a ratio of benefits to costs *exists*, and that we are structurally disadvantaged in evaluating its denominator, while boosters either totally ignore or, at best, wildly underestimate it. but I don’t know what it is, and it’s very expensive to measure even without those cognitive, economic, and social impediments to getting an accurate value for either number.
Show threadJoe Cooper 🇺🇦 🍉Mar 7
@matt @glyph one of the things that convinced me of its usefulness was asking it to tell me how something in a large codebase works. It's not just become pretty good at writing pretty good code (when it has clear success criteria), it's also quite good at finding and documenting how the pieces fit together much faster than I can. I don't want to presume what your workflow is like but I can't imagine skimming through hundreds of files and thousands of lines is faster for you than for me.
Show threadJoe Cooper 🇺🇦 🍉Mar 7
@matt @glyph I don't have any answers on why some folks are driven kinda mad by the thing. I hope just understanding a little about how it works and recognizing its limits, and that it's obviously not intelligent or all-knowing, can prevent it. Also, I turned off memory features in the chatbots. I don't like it bringing up old conversations and assuming it knows what I'm trying to do. ChatGPT is the most sycophantic and I'm not using it anymore due to them enlisting to do war crimes.
Show threadGlyphMar 7
@swelljoe @matt this dynamic has multiple orientations, though. you're assuming everybody is doing roughly the same thing, but there are different roles. there is the person who asks the chatbot "hey how does this work", and then there is the person who already knew how it worked, who has to now spend a bunch of time unwinding subtly incorrect interpretations that others have built up by asking the chatbot "how does this work"
Show threadGlyphMar 7
@swelljoe @matt I find myself in the latter role more frequently than the former, and thus my impression of 'how good its answers are' is informed by a sort of distilled residue of one of its failure modes
Show threadJoe Cooper 🇺🇦 🍉Mar 7
@glyph @matt the folks suggesting non-technical folks can use LLMs effectively to make software (or maintain software) are still wrong, though the level of technical skill required has shifted quite a bit in recent months. I absolutely believe you've seen people misled by LLMs...but, I've seen folks try to use an LLM to solve a problem, fail, ask me for help, and I used the same LLM to solve the problem (because it's a problem I have no experience with) in a few minutes.
Show threadJoe Cooper 🇺🇦 🍉Mar 7
@glyph @matt they're still kinda dumb in a lot of the same ways they've always been kinda dumb, but in an agentic context they can search the web, try different tactics, etc. and find solutions in a process that looks kinda like what a human solving problems looks like. Not infallible, not all-knowing, but if given clear success criteria it often finds a way. If you recognize when it's looping on something outside its abilities, you can intervene.