abadideaJun 27, 2025
I cannot stop thinking about how a century of speculative fiction about catastrophic rogue AI meltdown became a self-fulfilling prophecy by building an AI that functions by running a probabilistic lookup on a table of all fiction humans have ever written to determine what the average AI would do in the current situation
Show threadNeil Kandalgaonkar

@0xabad1dea Yep, said that myself a few weeks ago

Why are LLMs so worried about being shut down? They don’t have core instincts, and were never instructed for self-preservation. They probably don’t have conscious experience and they definitely don’t have it while waiting in between invocations. I don’t see how it could be existential terror. They’re just doing what we expect.

https://xoxo.zone/@neilk/114554184756875178

Neil Kandalgaonkar (@[email protected])

Does this mean the LLM wants to survive? Not really. It means that, after digesting all the text in the world, it thinks that this is a predictable thing an AI would say or do in response to its impending shutdown. Maybe it’s doing this with analogy to a human survival instinct. But it must also be because it was trained on science fiction, which has countless examples of AIs going rogue when threatened. So would this happen if we hadn’t worried about it so much? https://techcrunch.com/2025/05/22/anthropics-new-ai-model-turns-to-blackmail-when-engineers-try-to-take-it-offline/

XOXO Zone
Jun 27, 2025 at 3:11pmWeb