Linaerys
Prof. Emily M. Bender(she/her)A thought experiment in the National Library of Thailand—or why #ChatGPT (or any other language model) isn't actually understanding.
Osma Suominen@emilymbender
Thank you for this enlightening thought experiment! This is like Searle's famous Chinese Room example, except that it aptly describes modern LLMs instead of old rule-based AI systems.
Thank you for this enlightening thought experiment! This is like Searle's famous Chinese Room example, except that it aptly describes modern LLMs instead of old rule-based AI systems.
Bornach@osma @emilymbender
The difference is that the Chinese Room has a big book of instructions telling you how to create a response using Chinese characters. The Thai library doesn't even have that and you must somehow write this big book of instructions yourself.
I imagine archaeologists discovering a library of a lost civilization who had figured out how to communicate with an alien race. This alien race has now transmitted a question to modern humans. How do we formulate a response?
@osma @emilymbender
Also missing is an analog of Reinforcement Learning with Human Feedback and the hidden economy of low paid gig workers labelling the books in the Thai library. We'd have to imagine there was computer in the library that had a broken monitor, so all it did was beep annoyingly when you entered something it didn't like.
Also missing is an analog of Reinforcement Learning with Human Feedback and the hidden economy of low paid gig workers labelling the books in the Thai library. We'd have to imagine there was computer in the library that had a broken monitor, so all it did was beep annoyingly when you entered something it didn't like.
@bornach @emilymbender RLHF is relevant for ChatGPT and other instruction tuned models for sure, but not for basic foundation LLMs such as GPT-3/4 and LLaMA, which are only fed tons of text. I understood the Thai library analogy to be about plain LLMs that simply predict text, not the enhanced kind that try to perform tasks.
@osma @emilymbender
Although the now very closed "Open"-AI keep the nature of GPT4 a proprietary secret, it is widely assumed the same InstructGPT training procedure was used at some point in its development
Or perhaps like the fine tuned derivatives of the original LLaMa which have earned their own name (Alpaca, Vicuña, etc), the instruction following aligned GPT4 derivative that everyone is using via web API, officially goes by a different name but is erroneously referred to as GPT4
@osma @emilymbender
And a key flaw of these thought experiments is that we still can assume sentient being communicating with another sentient being and having a lot in common - needing to eat, having to ask for things of others, forming collectives, motivation vs instinct. I thought this scene in Arrival was most enlightening:
But how does a LLM on a computer that has never had to beg to be provided with electricity understand the concept of a child asking for food?
Arrival: The Nature of a Question (Amy Adams, Jeremy Renner) 4K HD Sci Fi Clip
JW prince of CPH@emilymbender A Thai library, or a Chinese room... https://en.m.wikipedia.org/wiki/Chinese_room
Also, reminds me of the time an astronomer thought he could see canals on Mars which appeared to be created by an intelligence - there aren't, of course, so another guy later observed that sure, the lines he thought he saw in his telescope were indeed signs of intelligence, but on the opposite end of the scope...
Matunos@emilymbender I see a problem with asking a reader to analogize their idea of themselves understanding a language to the capabilities of an computational model.
I understand things with my brain, through mechanisms that I can barely understand, and which at any rate are vastly different from how an LLM operates.
The best I could do in the library to truly test the capabilities of an LLM would be to create one and use it. But now the thought experiment has become circular.
@emilymbender In other words, this would seem to eventually devolve into a variant of a Chinese Room experiment, with a precondition that whatever the computational rules are that will be employed in the "Thai Room" if you will, they must be computed from the data in the library.
But can a Chinese Room be said to understand Chinese? The best response to that is: how would one know of it did?
@emilymbender if, instead of a subjective sense of understanding, we had a list of tasks that I would be expected to perform to demonstrate my understanding— tasks we expect that humans could only perform well if they understand a language— then I suspect I could theoretically achieve success in at least some them, without actually understanding the language myself.
@@emilymbender The obvious counter to this is that LLM are not strictly given _text_ but also _communications_ and in particular through RLHF _actual communications and feedback on its "utterances"_.
I agree with the basic point that LLM are far more appearance than substance, and that people are very good at finding meaning where there is none, but there are pathways open for actual LLM as they exist or might exist that are not present in your example.
Mog-Pharau@emilymbender Thanks a lot for hitting the nail on the head so precisely! Will bookmark this and forward it to anyone still claiming ChatGPT would understand or help in anything ...
Jörg Knappen@emilymbender In fact, unsupervised machine translation is possible, given large enough language models. For references, see https://linguistics.stackexchange.com/a/34102/9781
Could the Ancient Egyptian hieroglyphs have been deciphered without the Rosetta Stone with modern tech?
The Rosetta Stone was one of the most important documents in the history of linguistics. Discovered around 1800, it allowed Ancient Egyptian to be deciphered. Let's say that the stone didn't exist,...
Viktoria Pammer-Schindler@emilymbender Many thanks for sharing these thoughts, so well explained! A follow-up thought: In order to make sense of generative AI output we as humans recipients need to be capable of making sense of the language - but what if we humans over the next hundreds of years "just" write prompts? these could become less similar to what we think of as "meaningful text" as we understand it now,. then the generative AI still writes in the "old" language (which is now our language)...
Ray Jepson@emilymbender How would this differ from code breaking ?
I've thought the same thing using AI for art and design. It can spit out things that look like design, but it has no understanding of the human context: why shape a handle à certain way, why use a material like gold on jewelry, etc.
*_jߍyrope@Ray Jepson
has no understanding of the human context
That's the main flaw, It doesn't seem that AI devs care about that, though. Pretty dangerous situation.
has no understanding of the human context
That's the main flaw, It doesn't seem that AI devs care about that, though. Pretty dangerous situation.
FranklinMaillot@emilymbender Fascinating thought experiment. Here's my take. An often overlooked feature of LLMs is word embedding which encodes words relationship relative to each other as vectors. Vectors for 'good' and 'bad' are opposite to each other. Vectors for dog breeds are clustered in the same region, etc. I believe that this vector space captures something from reality and that's what we call "meaning" and "understanding". Couldn't we build that vector space from the library?
@emilymbender Another way to look at it. Let's assume an entity that receives inputs, has computing power and a reward function. It can detect patterns and can respond as to maximize its reward. Does this entity understand its environment and can it reason? If your answer is no, then you'll have to admit the same for humans. Aren't we only able to access the world through sensory inputs and do we have any mean to make sense of the world other than pattern recognition and a reward function?
@FranklinMaillot We could also work more with intention, meaning and understanding of actual humans instead. Much less effort.
n0body@emilymbender In Edgar Rice Burrough's "Tarzan of the Apes" (1912) young Lord Greystoke teaches himself to read by recognizing patterns in the "bugs" on the pages of his father's library.
As a young child, I wondered how he could have related those patterns to the physical world without a Rosetta stone, e.g. pictures.
This is the same thought experiment. Turing answers this question by rejecting it. The real question is one of *indistinguishability*, and LLMs are only at the T2 (pen-pal) stage.
As a young child, I wondered how he could have related those patterns to the physical world without a Rosetta stone, e.g. pictures.
This is the same thought experiment. Turing answers this question by rejecting it. The real question is one of *indistinguishability*, and LLMs are only at the T2 (pen-pal) stage.
Michael Busch@n0body @emilymbender Even Burroughs in his imagination found it necessary to give Tarzan an extensive collection of picture books and primers to learn English from - and he only learns to read and write a bit; not to speak.
@michael_w_busch @emilymbender Yes you are right, didn't he also learn to read French? This was the source of Jane's confusion who thought he was illiterate. (edit: the premise of the next book! it's been many decades since I read it)
Adam Wulf@emilymbender very interesting! It’s hard to wrap my brain around the difference of form and meaning, as they’re so closely tied together in my mind.
Even though current models work only on form, could models that work on both images and text uncover some meaning? How does this thought experiment change if picture books remain?
Thanks for your work!
Ján Bogár@emilymbender
- this assumes that because the task is hard for a human, it would also be hard for a machine.
- it implies that the only meaning is human-like meaning, which is not strictly true.
Let's instead try to learn arithmetic from a long list of solved equations - possible even for a human.
Meaning is how things relate to other things (?). But for LLM, the whole universe consists of words, understanding how they relate to each other is all the meaning there is.
@emilymbender Conversely, we humans are able to assign meaning to things in our sensory universe without any Rosetta stone, just by observing patterns.
We are just so predisposed for our sensory universe that discovering meaning of patterns in other universes - for example long lists of numbers describing turbulent flows- is comparatively really hard for us. But might not be for a machine.
Joe ❌👑@emilymbender I think this argument makes sense when the model has only text and is learning the structure of text. But we also have models that generate images; they are trained on both images and text where the text describes the images. We could bring in other "senses" as well: spoken language as well as written language. This would allow the model to learn how language correlates to other things. But you do have a good argument for the limitations of a pure LLM.
Lukas Galke@emilymbender I wonder if, in general, it is fair to conclude that only because it is not imaginable that we can do something, some other learning system (as an LLM) cannot do the thing?
On related note: I would love to hear your take on @lampinen et al.'s recent work: https://arxiv.org/abs/2305.16183
Passive learning of active causal strategies in agents and language models
What can be learned about causality and experimentation from passive data? This question is salient given recent successes of passively-trained language models in interactive domains such as tool use. Passive learning is inherently limited. However, we show that purely passive learning can in fact allow an agent to learn generalizable strategies for determining and using causal structures, as long as the agent can intervene at test time. We formally illustrate that learning a strategy of first experimenting, then seeking goals, can allow generalization from passive learning in principle. We then show empirically that agents trained via imitation on expert data can indeed generalize at test time to infer and use causal links which are never present in the training data; these agents can also generalize experimentation strategies to novel variable sets never observed in training. We then show that strategies for causal intervention and exploitation can be generalized from passive data even in a more complex environment with high-dimensional observations, with the support of natural language explanations. Explanations can even allow passive learners to generalize out-of-distribution from perfectly-confounded training data. Finally, we show that language models, trained only on passive next-word prediction, can generalize causal intervention strategies from a few-shot prompt containing examples of experimentation, together with explanations and reasoning. These results highlight the surprising power of passive learning of active causal strategies, and may help to understand the behaviors and capabilities of language models.
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛@lpag Something unimaginable might be able to learn to understand the meaning behind language from just studying texts.
But we are the ones who imagined the LLM.
Parade du Grotesque 💀Thank you - that entire post should be required reading everywhere.
Aaron D'Souza@emilymbender An interesting test is whether any of these systems would be able to *initiate* a conversation (absent any other input). How could they, when they have no idea what it means to say "hello"?
This is where "distributional semantics" fails us. Even if the protagonist in the library mastered the distributional semantics of the Thai linguistic form, they would only be able to initiate a conversation by spitting out non-sequiturs that satisfied some marginal probability requirements.
Owen Anderson@emilymbender I wanted to ask a/b something I didn't see discussed before.
The article posits that all meaning grounds in external reference, and that in lang acquisition that grounding must be direct or indirect (from a prior language).
ISTM *some* meaning can be grounded in self-reference: basic arithmetic for example (see "Contact) which yields meanings of truth and falsity.
Self-referential meaning could also arise out of language instructional texts, such as for children.
@emilymbender For a concrete example, one could derive a concept of the number 3 from observation of a "counted list" pattern:
"Here are three examples of reptiles: snakes, lizards, and turtles."
@emilymbender So, my question is, does that demonstrate that there exists a category of meaning (self-referentially-derivable meaning) that could potentially be learned purely from textual observation? If so, do we know anything about that category, and does it provide useful bounds on what kind of intelligence we could expect from a hypothesized omnipotent language model?