Chris Hanson@siracusa I was really disappointed to hear you say, shortly before 1hr40m in the latest released episode of ATP, that LLMs are “really good at understanding” which is an enormous category error. They don’t *understand* anything! What they’re good at is *using statistics to predict what text could plausibly come next*.
@siracusa You cannot actually trust anything coming out of an LLM. There are AI systems that do have actual understanding, like Cyc and other systems built on huge ontologies in knowledge representation systems, and they can do some pretty amazing things. For an example, look into Cyc’s participation in the “Battlefield of the Future” wargames a couple decades back to see how it compared to other battlefield decision support systems.
John Siracusa@siracusa Thanks for covering that later in the episode, I turned it off and messaged you when I heard you ascribe understanding and only later later listened to the rest. But think about this: If you, so careful with language, make such statements, what chance do “normal” people have?
@siracusa Also, Robot Or Not topic: Should an ontology be represented by a hierarchy, a heterarchy (“multiple inheritance”), or a graph of relational tuples? In Robot or Not you seem to default to hierarchy. ;)
Damien Sorresso@enhancedscurry @siracusa Yes, but I don’t think it goes the other direction—being able to make predictions about something doesn’t necessarily mean understanding it.
@eschaton @siracusa But if you have a system which can make predictions about arbitrary subjects with sufficient training, why does the training mechanism matter? It really gets into "quacks like a duck" territory. Not there yet, mind you, but I don't think a reliance on statistics is all that different from we do things, which is basically pattern recognition.
Sören@enhancedscurry @eschaton @siracusa I’m not an expert, but these don’t feel like something sufficiently good models will fix:
• hands with too many or too few fingers
• distortions in image enhancements
• ChatGPT guessing how many letters a word has
These are all “human looks at it for mere seconds and seems the error” mistakes that such a model doesn’t seem to grok
@chucker @enhancedscurry @siracusa Exactly; while I think statistical prediction models can get really good, ultimately they’re going to be limited by the lack of knowledge unless the networks themselves happen to develop an underlying understanding as a side-effect of training.
Lennvor@siracusa @enhancedscurry @chucker
Considering LLMs has made me reflect on how our own brain has separate modules for "understanding" (predicting, modelling) different things, such that not only is our understanding not reducible to language, the very notion of "meaning" for a human appeals to those non-linguistic models. Language production & understanding for us consists of converting nonverbal internal states/model into language and back. Of course there are complex feedbacks where language influences nonverbal understanding but it still involves separate systems and understanding doesn't reduce to language.
What I'm not sure of is whether this is an inherent feature of anything that could act on its "understanding" as well as we do (which LLMs currently can't, whatever label we assign to their internal processes), or whether it's just the way we happen to do it and a system without such a separation could perform as well as we do. I can't help feeling it's an inherent feature but I haven't found a logical justification for it.