Here's a fun AI story: a security researcher noticed that large companies' AI-authored source-code repeatedly referenced a nonexistent library (an AI "hallucination"), so he created a (defanged) malicious library with that name and uploaded it, and thousands of developers automatically downloaded and incorporated it as they compiled the code:
https://www.theregister.com/2024/03/28/ai_bots_hallucinate_software_packages/
1/
If you'd like an essay-formatted version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:
https://pluralistic.net/2024/04/01/human-in-the-loop/#monkey-in-the-middle
2/
These "hallucinations" are a stubbornly persistent feature of large language models, because these models only give the illusion of understanding; in reality, they are just sophisticated forms of autocomplete, drawing on huge databases to make shrewd (but reliably fallible) guesses about which word comes next:
https://dl.acm.org/doi/10.1145/3442188.3445922
3/
Guessing the next word without understanding the meaning of the resulting sentence makes unsupervised LLMs unsuitable for high-stakes tasks. The whole AI bubble is based on convincing investors that one or more of the following is true:
I. There are low-stakes, high-value tasks that will recoup the massive costs of AI training and operation;
II. There are high-stakes, high-value tasks that can be made cheaper by adding an AI to a human operator;
4/
III. Adding more training data to an AI will make it stop hallucinating, so that it can take over high-stakes, high-value tasks without a "human in the loop."
5/
These are dubious propositions. There's a universe of low-stakes, low-value tasks - political disinformation, spam, fraud, academic cheating, nonconsensual porn, dialog for video-game NPCs - but none of them seem likely to generate enough revenue for AI companies to justify the billions spent on models, nor the trillions in valuation attributed to AI companies:
https://locusmag.com/2023/12/commentary-cory-doctorow-what-kind-of-bubble-is-ai/
6/
@pluralistic I’d even say dialog for video game NPCs is an unsuitable task for LLMs.
As a writer, you know how important it is to say exactly what needs to be said to move the story—in advancing the setting, characters, or narrative.
An LLM just cranking out “kind of fits” dialog really doesn’t do any of that, and if anything, makes it harder to deal with knowing if you caused a state change you were supposed to or learned something you needed to know.
@stooovie you would think—but think about that for a second. Most bystander NPCs sort of grunt, act confused, or maybe say a comment to you that you’ve heard before. These are all signals that NPC doesn’t have much to add to the progression of the story—it’s an indicator that they’re a leaf on the game tree, not a branch.
Putting all of ChatGPT, its weird sycophancy, its infinite desire to blab—does that add anything to the game? Maybe the first time, but not the fifth.
Cory Doctorow
Listerine Lavender Mint
Jiří Fiala Total Landscaping
Robin Adams