| Pronouns | she/her |
| Website | https://anna-ivanova.net/ |
| Google Scholar | https://scholar.google.com/citations?user=hBUjCB0AAAAJ&hl=en |
Anna Ivanova
@neuranna@mstdn.science
- 439 Followers
- 180 Following
- 37 Posts
Postdoctoral Associate at MIT. Language and thought in brains & in machines.
Ben WaberNext was a fantastic talk by @neuranna on what language understanding actually means, also at @sfiscience. Through a number of brilliant experiments, Ivanova wonderfully picks apart the notion that language processing is enough to achieve any complete notion of full understanding. Highly recommend https://www.youtube.com/watch?v=KKuu57FXd0g&t=1s (6/9) #AI #LLMs
Anna Ivanova: Language Understanding Goes Beyond Language Processing
P.S. Although we have >20 pages of references, we are likely missing stuff. If you think we don’t cover important work, pls comment below! We also under-cover certain topics (grounding, memory, etc) - if you think something doesn’t square with the formal/functional distinction, let us know.
It’s been fun working with a brilliant team of coauthors - @kmahowald @ev_fedorenko @ibandlank @NancyKanwisher & Josh Tenenbaum
We’ve done a lot of work refining our views and revising our arguments every time a new big model came out. In the end, we still think a cogsci perspective is valuable - and hope you do too :) 10/10
Similarly, criticisms directed at LLMs center on their inability to think (or do math, or maintain a coherent worldview) and sometimes overlook their impressive advances in language learning. We call this a “bad at thought = bad at language” fallacy. 9/
The formal/functional distinction is important for clarifying much of the current discourse around LLMs. Too often, people mistake coherent text generation for thought or even sentience. We call this a “good at language = good at thought” fallacy. 8/
Google's powerful AI spotlights a human cognitive glitch: Mistaking fluent speech for fluent thought
Fluent expression is not always evidence of a mind at work, but the human brain is primed to believe so. A pair of cognitive linguistics experts explain why language is not a good test of sentience.
ChatGPT, with its combination of next-word prediction and RLHF objectives, might be a step in that direction (although it still can’t think imo). 7/
We argue that the word-in-context prediction objective is not enough to master human thought (even though it’s surprisingly effective for learning much about language!).
Instead, like human brains, models that strive to master both formal & functional language competence will benefit from modular components - either built-in explicitly or emerging through a careful combo of data+training objectives+architecture. 6/
On the other hand, LLMs are still quite bad at most aspects of functional competence (math, reasoning, world knowledge) - especially when it deviates from commonly occurring text patterns. 5/
Armed with the formal/functional distinction, we thoroughly review the NLP literature. We show that, on one hand, LLMs are surprisingly good at *formal* linguistic competence, making significant progress at learning phonology, morphosyntax, etc etc. 4/
We ground this distinction in cognitive neuroscience.
Years of empirical work show that humans have specialized neural machinery for language processing (reading, listening, speaking, etc), which is distinct from brain mechanisms underlying other cognitive capacities (social reasoning, intuitive physics, logic and math…) 3/