Poren ChiangJun 29
Greg Egan

“Potemkin Understanding in Large Language Models”

A detailed analysis of the incoherent application of concepts by LLMs, showing how benchmarks that reliably establish domain competence in humans can be passed by LLMs lacking similar competence.

H/T @acowley

Link: https://arxiv.org/abs/2506.21521

Jun 28 at 3:22amWeb
Show threadSteveg58Jun 28
@gregeganSF @acowley
Maybe this could lead to better exam questions?
Show threadKevin GranadeJun 28
@Steveg58 better exam questions for LLMs? Or for humans?
In the further case who cares. In the latter why would LLM error modes have to do with human error modes?
Show threadSteveg58Jun 28
@kevingranade
To create exam questions that require the sitter to demonstrate understanding, The current ones obviously don't. Any exam question that allows someone with no actual knowledge to pass is obviously flawed.
Show threadGreg EganJun 28

@Steveg58 @kevingranade

The paper makes the case that when humans misunderstand things, they usually do so in a very different way than LLMs — and that exams that are genuinely pretty good at assessing humans’ competence can nonetheless be terrible at assessing LLMs.

This doesn’t mean all current exams are perfect at assessing humans, but it’s a big [and unjustified] leap to go from “an LLM can ace this exam while having no competence at applying the relevant concepts” to “a human who aces this exam might understand as little as the worst LLM that could also ace this exam”.

We might have to change exams to catch out humans who are cheating with real-time assistance from concealed LLMs, but that’s a different matter.

Show threadDavid NashJun 28

@gregeganSF @acowley

This. So much this. I had the following exchange with Claude a couple of months ago (lightly condensed and paraphrased)

Me: draw the structure of naphthalene, please

Claude: (produces something with the right shape and general connectivity of the atoms, but one carbon atom has five bonds, a classic oopsie in Intro Organic Chem)

Me: one of the carbon atoms has the wrong number of chemical bonds. Please fix it.

Claude: I’ve redrawn it. (Its effusive self-directed praise about how perceptive I am and how it successfully redid the drawing has been deleted.)

(a human who understands carbons are limited to 4 bonds, while also usually having exactly 4 bonds in neutral molecules, would find the error and fix it easily. Claude did…not. Its revised structure has *four* carbon atoms with five bonds.)

Me: your structure has four carbon atoms with the wrong number of bonds.

Claude: I’ve redrawn it. (Again, much obsequiousness deleted.)

(Claude has drawn the same exact structure as the previous one, except a couple of the bonds have been shifted by a couple of pixels.)

That was Claude. I tried the same thing on Gemini and it was immeasurably worse, culminating in a structure that had carbon bound to three other carbons *and three hydrogens* and that wasn’t even the weirdest thing about it.

Show threadToni AittoniemiJun 28

@gregeganSF @acowley Ah this ratio of 19/20 accuracy explains a lot!

This kind of accuracy/inaccuracy ratio is actually really difficult for humans to understand.

It is accurate enough times to throw a facade over it’s nonunderstanding, but not accurate enough to trust in the long run.

It passes just under our instinctive thresholds of validating a person’s understanding of a concept. We let it pass, as a human who gets 19 out of 20 right ”gets it” and would improve, where an #ai doesnt

#llm

Show threadChaucerburntJun 28
@gregeganSF @acowley reminds me of Feynman's anecdote about physics students in Brazil? who could perfectly recite textbook statements about polarisation of light, but were incapable of applying them to a very simple problem.
Show threadAmir Livne Bar-onJun 28

@gregeganSF @acowley
It's great to have this work, to show how (depending on the viewpoint) models are able to solve some problems even though they don't understand, or some benchmarks are too easy to solve by pattern matching and don't measure understanding.

It would be even nicer if they used best practices when invoking the models, such as chain of thought prompting. Especially given that some of the problems require deliberate consideration when people do them. And some of the problems touch on known issues with language understanding, such as counting syllables. But most of the tasks are free of these.

Show threadGLCJun 28

@gregeganSF @acowley

"incoherent application of concepts"

Reminder that no concepts are involved. in a random (Markov) walk through word space. Shannon 1948.

From Pogo: "We could eat this picture of a chicken, if we had a picture of some salt."

Show threadl'empathie mécaniqueJun 28
@glc @gregeganSF @acowley But that is not the word space they're walking...
Show threadGLCJun 28

@dpwiz @gregeganSF @acowley

I suppose you must be referring to Pogo, which is not, for the present purposes, even a word space (or: not fruitfully treated as such).

Show threadl'empathie mécaniqueJun 28
@glc @gregeganSF @acowley no, the LLMs aren't operating in **word**-space.
Show threadGLCJun 29

@dpwiz @gregeganSF @acowley

Are you trying to distinguish tokens and words?

Or do you have a point? If so, what is it?

Show threadl'empathie mécaniqueJun 29
@glc @gregeganSF @acowley No, bytes/tokens/words/whatever is irrelevant. The important part that's wrong in the "word-space" model is that it misses the context. The "language" part is a red herring. What's really going on is a tangle of suspended code that's getting executed step by step. And yes there are concepts, entities, and all that stuff in there.
Show threadGLCJun 29

@dpwiz

I'd say there is syntax without semantics (in the traditional sense of formal logic, that is).

You have some other view evidently.
That much is now clear.

I don't see much difference from Markov and Shannon, apart from some compression tricks which are needed to get a working system.

Show threadl'empathie mécaniqueJun 29
@glc Perhaps. I just hope this not another "X is/has/... Y" claim.
What's your favorite or most important consequence of this distinction?
Show threadGLCJun 29

@dpwiz

That no concepts are involved, and the numerous corollaries of that, I suppose. At least, that's what I find myself harping on now and then.

I have no strong interest in the details. though considerable interest in watching this play out.

—Someone like Cosma Shalizi is going to actually get into the weeds a bit more:
http://bactra.org/notebooks/nn-attention-and-transformers.html

You'll probably find much to agree with and much to disagree with there. And at adequate length.

"Attention", "Transformers", in Neural Network "Large Language Models"

Show threadl'empathie mécaniqueJun 30

@glc > I find this literature irritating and opaque.

That's a promising start! (8

Show threadCegorachJun 28
@gregeganSF @acowley in short: a LLM stores data. It doesn't UNSERSTAND that data.

No surprise here.
Show threadMetafrastisJun 28
@gregeganSF @acowley I don’t know why so many scientists seem to suck at using LLMs, but Claude Sonnet 3.7 had no problem correctly answering these 3 questions at first try.
Show threadAriaflameJun 29
@dgavin @gregeganSF @acowley Well many of us prefer to think for ourselves. After all, that's our job.
Show threadMetafrastisJun 29
@ariaflame @gregeganSF @acowley That’s not the point. I prefer that too. But there are too many scientific studies that just dont stand up to peer review in the LLM field.
Show threadAriaflameJun 29
@dgavin @gregeganSF @acowley Well let's put it this way. If scientists suck at using it, what do you think the average person is going to be like trying to use it and interpret its output?
Show threadMetafrastisJun 29
@ariaflame @gregeganSF @acowley I started using computers in the 80ies and I am to this day still trying to teach scientists at our university to use Word. Most people suck at using computers in general and we still use them more than any other tool humanity has ever invented. We will have to tame AI, because people will use it, no matter what.
Show threadAriaflameJun 29
@dgavin @gregeganSF @acowley At least until the bubble collapses because it isn't magic and won't give the investors the financial rewards they are hoping for. And I started using computers then too. I am fully aware of GIGO and LLMs just magnify this.
Show threadDieuJul 4
@ariaflame @dgavin @gregeganSF @acowley anything in, garbage out
Show threadAriaflameJun 29
@dgavin @gregeganSF @acowley And it's as if the output of LLMs isn't consistent. So your chances of getting accurate information from it are unknown.
Show threadMetafrastisJun 29
@ariaflame That has been known for years. That’s why LLM are not a tool for scientific writing. You don’t use a hammer to drive in a screw. That doesn’t mean a hammer is a useless tool. LLM are extremely good at other things humans suck at, such as making sense of large amounts of data.
Show threadAriaflameJun 29
@dgavin No, they don't make sense of anything. They are a stochastic parrot predicting most likely words after another word. They don't think. They don't analyse. Now neural nets trained on specific tasks, there is some hope there, but LLMs are not an analysis tool and I would be extremely dubious of any data summaries it 'produced'/