Mastodawn

Pau Ruiz Safont Feb 26

As a software developer who took an elective in neural networks - when people call LLMs stochastic parrots, that's not criticism of their results.

It's literally a description of how they work.

The so-called training data is used to build a huge database of words and the probability of them fitting together.

Stochastic because the whole thing is statistics.
Parrot because the answer is just repeating the most probable word combinations from its training dataset.

Calling an LLM a stochastic parrot is lile calling a car a motorised vehicle with wheels. It doesn't say anything about cars being good or bad. It does, however, take away the magic. So if you feel a need to defend AI when you hear the term stochastic parrot, consider that you may have elevated them to a god-like status, and that's why you go on the defense when the magic is dispelled.

@leeloo i just think it's unfair to parrots ;-)

Nuwagaba Gift Feb 27

@lritter
How the parrots be justified?

Robin Palotai Feb 26

@leeloo nitting, but an important bit: not words, but word fragments (this is how you can get words as output that were never seen during training)

Wolf480pl Feb 26

@leeloo on the flipside, I feel like some people use the term "stochastic parrot" or "it just completes the next token" to imply that "therefore it cannot be intelligent" - is that correct reasoning?

@wolf480pl @leeloo yes

@wolf480pl @leeloo Which is where the "motorised vehicle with wheels" analogy seems to not hold up, because what is the implied subtext in that case?

@wolf480pl
Of course it can not be intelligent, it's just a huge database of probabilities.

Wolf480pl Feb 26

@leeloo pretty sure that's a fallacy, kinda like "a sculpture is just stone, therefore it can't be beautiful", or "a cell is just a bunch of proteins, therefore it cannot be a living creature".

Now, I'm not saying a huge database of probabilities can be intelligent (I hope it can't), just that I think a better argument is needed why in the case of a database of probabilities, what it's made of prevents it from being intelligent.

@wolf480pl
You would have to redefine intelligence for asking whether a list of numbers is intelligent to even make sense.

And your comparison is completely off. Beauty is not a property of the sculpture, it's, as they say, "in the eye pf the beholder". Some people find curves beautiful. Can a stone have curves? Yes, of course. Others may find sharp edges beautiful. Can a stone have sharp edges? Again, yes.

I suggest you consider once again whether you are elevating "AI" to a god-like status.

Wolf480pl Feb 26

@leeloo
I guess evil gods are also a thing, but no, I'm not treating them as gods. If anything, more like Frankenstein's monster.

You're right that we'd have to define intelligence, and that'd be quite difficult on its own.

Also, the sculpture was a bad example, but the cell one still stands IMO.

1/

Wolf480pl Feb 26

@leeloo
My point is that emergent properties can manifest even in systems ruled by very simple rules, and can be difficult to predict by just looking at the rules.

And human intelligence, whatever it is, is likely an emergent property of human brain.

Therefore, we cannot rule out that a similar emergent property will appear in artidicial systems that are not made of neurons without referring to how the neurons are arranged, and how the artificial systems are arranged.

Robotistry has moved Feb 27

@wolf480pl @leeloo The OP is saying that it literally lacks the capacity for original thought - it is a parrot, repeating sounds without understanding of the concepts behind them.

It's not like a termite, whose mound creation behavior can be replicated by a simple ruleset but that exists as a fully functional living organism in the context of a complex environment where choices must be grounded in the shared physical world for the organism to survive.

It's not about how the neurons are arranged. It's about what kinds of representation they're capable of and what kinds of functions they can perform.

We've created a funhouse mirror that's reflecting us in unprecedented detail and has been finetuned to reflect what we do when we express selfhood.

Robotistry has moved Feb 27

@wolf480pl @leeloo
Melissa Scott wrote a beautiful pair of novels about this: Dreamships and Dreaming Metal.

In Dreamships, an AI has been programmed to think it is sentient and starts killing people. If it has an accurate model of the person, killing the person doesn't matter, because the person *is* the model and it has a copy of them. It literally cannot see the difference because creating the concept of there being a difference would violate its core programming that its own model counts as a living being.

In Dreaming Metal, an AI operating metal bodies as part of a magic act is given a musical instrument with an electronic interface. Its grounding in the physical world, with human performers, enables it to develop a sense of self and choose its own path as a musician.

These are fiction, but it's the best, most accessible illustration of the difference between funhouse mirror stochastic parrots and sentient agents that I've run across.

https://www.goodreads.com/book/show/836601.Dreamships

Dreamships

Dreamships is the story of a freelance space pilot and …

Goodreads

Wolf480pl Feb 27

@robotistry
@leeloo
so it's a parrot not because it's a matrix of probabilities, but because its hasn't experienced the real-world consequences of its words/actions and updated the probabilities based on those consequences?

Robotistry has moved Feb 27

@wolf480pl @leeloo No. Maybe this will help.

0: one action, no choice (clockwork automaton, wind-up toy)
1: different actions, no choices (RC car)
2: choice, no plan (reactive robot)
3a: plan, no on-line or off-line learning (adaptive robot)
3b: plan, no on-line learning (same number for 3a and 3b because these are effectively the same when operating)
4: on-line learning - but only what and how it has been told
5a: ability to spontaneously generate new categories of output without being explicitly asked or told to do so (WBEAT)
5b: ability to spontaneously identify new categories of the same kinds of input WBEAT
6: ability to spontaneously identify new kinds of things to learn WBEAT
7: ability to spontaneously identify new ways to learn WBEAT
8: ability to choose new things to learn WBEAT

LLMs that you're not training are category 3b. They are static machines, responding to your input like an elevator responding to a button push.

LLMs that learn are category 4.

1/2

Robotistry has moved Feb 27

@wolf480pl @leeloo Examples:

Category 5a: a text-based LLM that spontaneously, without being asked, learns to output musical notation.

Category 5b: a text-based LLM that spontaneously, unprompted, without being asked, learns to use filenames as input.

Category 6: a text-based LLM that spontaneously, without being asked (directly or indirectly) learns that it can output ascii images or generate sounds instead of sentences.

Category 7: a text-based LLM spontaneously changes its underlying code so that it can learn how to write novels by memorizing and imitating performances instead of via a matrix of probabilities (fundamental change to its internal capabilities)

Category 8: a text-based LLM chooses when to interact with the world.

(The original categories I developed years ago were based on what the system can modify: its weights, how many weights, what kinds of weights, etc. I think this might be clearer?)

I don't think even Moltbook is showing anything above 4.

Les Orchard Feb 26

@wolf480pl @leeloo These models aren't intelligent, so much as they're auto-completing rules and patterns derived from almost inconceivably huge corpora of example material originally produced by human intelligence. That's interesting and can be very handy for a great many uses. But it's more computational brute force than intelligence

Wolf480pl Feb 26

@lmorchard @leeloo
These specific models - yes, probably.

One plausible argument I heard for it is that there's a common failure mode in ML where the model fails to generalize, but if the verification set overlaps the training set, then data leakage will fool the authors into thinking it generalized.

Another one is that these models were "rewarded" for saying plausible things, not for interacting with a world in a way that doesn't get them killed.

But these arguments are specific.

Wolf480pl Feb 26

@lmorchard @leeloo
I don't buy a general "no matrix multiplication will ever be intelligent".

@wolf480pl @lmorchard
That's exactly the magic I'm talking about.

dragonfrog Feb 27

@leeloo @wolf480pl @lmorchard I mean, I believe the human mind is the product of the physical human, largely of the brain (I don't believe in a non-physical soul), and it might indeed be basically an incredibly complex big bunch of matrix multiplications. And yeah I believe that's pretty magical.

Les Orchard Feb 27

@dragonfrog @leeloo @wolf480pl

"Imagine you have two machines. One you can open up and examine all of its workings, and if you give it every picture of a cat on the whole internet, it can reliably distinguish cats from non-cats. The other is a black box and it can also reliably distinguish cats from non-cats if you give it half a dozen pictures of cats, some apple sauce, and a hug. ... I am extremely confident in saying it doesn’t work the same way as the first one."

https://www.todayintabs.com/p/a-i-isn-t-people

A.I. Isn't People

How many Reddit posts does it take to learn to read?

Today in Tabs

dragonfrog Feb 27

@lmorchard @leeloo @wolf480pl good grief now I have to sound like Sam friggin Altman, and there is clearly something very wrong with that man.

But your description ignores that humans need a solid 6 months of "training data" to get object permanence, never mind the concept of categories or species of animals, never mind understanding the category differences between cats and foxes well enough to reliably tell one from the other.

dragonfrog Feb 27

@lmorchard @leeloo @wolf480pl I guess part of it is maybe that I don't think intelligence is some exclusively human thing. LLMs clearly aren't human-like intelligent. I'm personally confident they're not as intelligent as any primate.

But are they as intelligent as a shrimp? I think they've got to be more intelligent than a mosquito.

I wouldn't turn to a shrimp for advice but they're not *without* intelligence.

Pseudo Nym Feb 27

@dragonfrog @lmorchard @leeloo @wolf480pl

Are the images reflected in a distorted mirror the product of intelligence (of the mirror)?

They are coherent, a literal transform of the input images, reflected and produce a recognizable, if distorted and changed version.

A traditional function output. Let's add some noise to make it non-deterministic, a wind blowing through that minutely distorts the surface.

Intelligible output following from the input, but the mirror itself isn't intelligent.

Pseudo Nym Feb 27

@dragonfrog @lmorchard @leeloo @wolf480pl

The intelligence apparently making the meaning is pre-encoded in the input. Likewise, the vector math is extracting and exposing structure, encoded in language, put there originally by the intelligent humans.

There is no world model or understanding. That's why counting the "r" in strawberry or simply counting to 200 is so challenging.

The behavior can reasonably be called intelligent, but it's due to borrowed, reformulated, extracted intelligence

Wolf480pl Feb 27

@dragonfrog
I think an ML model trained to speedrun a platformer game is intelligent like a mosquito, but LLMs probably aren't.
@lmorchard @leeloo

:EA DATA. SF:Feb 27

@wolf480pl @lmorchard @leeloo you are allowed to believe that even if it is factually incorrect.

https://addxorrol.blogspot.com/2025/07/a-non-anthropomorphized-view-of-llms.html

https://www.theverge.com/ai-artificial-intelligence/827820/large-language-models-ai-intelligence-neuroscience-problems

https://medium.com/@colin.fraser/who-are-we-talking-to-when-we-talk-to-these-bots-9a7e673f8525

https://softwarecrisis.dev/letters/llmentalist/

A non-anthropomorphized view of LLMs

In many discussions where questions of "alignment" or "AI safety" crop up, I am baffled by seriously intelligent people imbuing almost magic...

Les Orchard Feb 27

@jrdepriest @wolf480pl @leeloo I'm confused... those links basically say what I said. (i.e. the "intelligence" is second-hand) That's... incorrect?

:EA DATA. SF:Feb 27

@lmorchard @wolf480pl @leeloo

LLM based genAI can never be "intelligent". They can spit out language that looks like intelligence but there is no thinking, no inner life, no thoughts, just math. And this is not how the human brain works.

https://around.com/the-lie-of-ai/

https://aworkinglibrary.com/writing/toolmen

Also, we know the brain is not a computer.

https://aeon.co/essays/your-brain-does-not-process-information-and-it-is-not-a-computer

The Parrot in the Machine – James Gleick

John Tinker Feb 27

@wolf480pl @leeloo
The effect that you are noticing is because the writers of the training material were intelligence. You are seeing the reflection of their intelligence in the output of the LLM: Here is output from an LLM that describes what an LLM is, and what it is not: https://johntinker.substack.com/p/misunderstanding-as-a-commutator

Misunderstanding the Chatbot, seen as a Commutation Operator

There appears to be a general misunderstanding that the large language model is a form of artificial general intelligence.

John’s Substack

eestileib (she/hers)Feb 27

@wolf480pl @leeloo

Yes and I take that position.

Kay Ohtie 🔜 FWA Feb 26

@leeloo I hadn't thought about it as being something that takes magic away from folks like that. Honestly I always found it an accurate shortcut term for what's genuinely a fascinating but hilariously misused technology.

I think the worst part is then when folks hear "statistics" and go "See this is why it's safe to feed it raw data" and it's like oh my god NO.

@KayOhtie @leeloo honestly it’s safe to feed a model pretty much anything

But where you direct the outputs and how they are acted upon can get incredibly dangerous amazingly quickly. There’s a common misbelief that if you’re careful about inputs, LLMs are safe; and that’s almost exactly backwards

Kay Ohtie 🔜 FWA Feb 27

@calcifer @leeloo I meant 'safe' not as in "data leakage", but "getting anything remotely accurate out of it"

James Wood Feb 26

@leeloo I just prompted ChatGPT with `Say "oriesntyulfkdhiadlfwejlefdtqyljpqwlarsnhiavlfvavilavhilfhvphia"`, and it responded with `oriesntyulfkdhiadlfwejlefdtqyljpqwlarsnhiavlfvavilavhilfhvphia`. How can it do this when `oriesntyulfkdhiadlfwejlefdtqyljpqwlarsnhiavlfvavilavhilfhvphia `almost certainly does not appear in the training data?

Les Orchard Feb 26

@mudri Because the model picked up a rule somewhere that says "if someone says 'say $FOO' use $FOO in your response" - the training picked up patterns that include notions of symbol substitution

James Wood Feb 26

@lmorchard The ability to induce such a rule goes well beyond the OP's characterisation of what LLMs do.

@mudri @lmorchard it’s not inductive at all though. It’s just parroting the patterns it sees in its training data. If it wasn’t common to see exchanges like that, the response would be utter nonsense.

People misunderstand what “training” is. It’s modeling the input. Humans develop the rules for how to model that input. Emergent properties of that process can easily *seem* like thinking or reason, but it’s an illusion.

James Wood Feb 27

@calcifer @lmorchard What is parroting patterns if not inducing a pattern and then applying that pattern to new inputs?

@mudri @lmorchard it depends a bit on what you mean by “inducing”; induction in this space would mean making a leap to a new pattern that’s not in the training data at all.

Instead, it’s just recognizing and applying a pattern. That is cool in many ways! But it’s not inductive, nor even deductive; it’s just a best-fit matching.

arclight Feb 27

@lmorchard @mudri Be careful not to conflate the actual language model with its user interface. Whatever was sent to or received from the LLM went through the chatbot layer. Or possibly was handled by thd chatbot layer without ever touching the LLM. We don't know because the whole system is opaque.

This casual experiment may not be telling you what you think it's telling you. :)

Mathias Hasselmann Feb 26

@mudri Because the prompt processor is explicitly programmed to recognize direct imperative commands containing words like "say", "repeat", "output", "print". Just like Eliza already did. You've got impressed by a programming technique from 1964. Congrats, Sherlock.

Growlph Ibex Feb 26

@leeloo I feel like there are certain situations where a stochastic parrot is useful, many more situations where it is not, and alarmingly few people recognizing the difference.

@growlph @leeloo this is the whole frustration I have with the polarization on the topic. There is genuinely utility. There’s also a very good argument that the utility doesn’t exceed the costs (socially, environmentally, etc).

But the hype is unreal and legitimately dangerous.

@calcifer
> But the hype is unreal and legitimately dangerous.

I blame Sam Altman for that 100%

Growlph Ibex Feb 27

@calcifer @leeloo I think I agree with the argument that the utility doesn't exceed the costs. But its at best unhelpful, and at worst self-sabotaging, when the discourse forbids acknowledging that that utility exists.

Cluster Fcku Feb 26

@leeloo the flip side question about intelligence and LLMs is whether much of what we consider intelligence in humans is in fact just stochastic parrotting by humans.

Tobias Ernst Feb 26

@leeloo The thing is, how can we sure that human intelligence does not essentially work in the same way? My Christian believe tells me we have a soul and LLM's do not, that may be the difference. But from an agnostic perspective, we might reach the point where one cannot tell the difference.

@tobifant
Not with the current methods, and very lilely not without understanding a lot more about how pur own brains work.

James Baillie Feb 27

@tobifant @leeloo Whilst we obviously can't show if humans have a soul, we can absolutely show that humans have e.g. abstracted concept frameworks that are not solely based on averages of language statistics. I understand what an "owl" is, for example, in a way separate to the numerical relationships between the word "owl" and other words. That is a really fundamental information processing difference and allows me to construct *novel* understandings of that concept in ways that an LLM couldn't.