| 🌏 | https://poren.tw |
| 🐙 | https://github.com/RSChiang |
“Potemkin Understanding in Large Language Models”
A detailed analysis of the incoherent application of concepts by LLMs, showing how benchmarks that reliably establish domain competence in humans can be passed by LLMs lacking similar competence.
H/T @acowley
想要偷偷分享我很喜歡的木吉他歌手 @chanwenju ！輕快的旋律不論是週末早晨或是午夜時分都很撫慰人心。
《快樂旅社》
• iTunes Store: https://music.apple.com/tw/album/%E5%BF%AB%E6%A8%82%E6%97%85%E7%A4%BE/1822804717
• Spotify: https://open.spotify.com/album/01wtvbyk51x128s26faatl
📢 We've sat down with our artist @dopatwo and created a sticker pack for @signalapp. Now you can send cute elephants to your friends, and promote the #fediverse at the same time. We ❤️ Signal, too!
The Third Edition of Portable Network Graphics (PNG) today became a W3C Standard.
It adds support for High Dynamic Range (HDR), and adds Animated PNG to the official standard.
This document describes PNG (Portable Network Graphics), an extensible file format for the lossless, portable, well-compressed storage of static and animated raster images. PNG provides a patent-free replacement for GIF and can also replace many common uses of TIFF. Indexed-color, greyscale, and truecolor images are supported, plus an optional alpha channel. Sample depths range from 1 to 16 bits.
If you're one of my academic publishing folks: sadly, it's true. Due to a funder's unexpected decision to pull support, we've had to make the incredibly difficult decision to wind down PubPub over the next 18 months and regroup to figure out how to best serve our mission.
I'm sure I'll have more to say, but today I'm feeling both gratified and saddened by the overwhelmingly supportive responses to our announcement.
I urge you to learn from our mistakes: https://www.knowledgefutures.org/updates/2025-06-update/
I rarely subtoot, but when I do just to say: if an open source project that your commercial project depends on breaks something in your software stack, causing you trouble, no matter how much, that's your problem and your problem alone.
"The software is provided as is" is a part of OSS licenses for a reason, and unless we have a contract that says otherwise, I'm not part of your bloody "supply chain".
“Potemkin Understanding in Large Language Models”
A detailed analysis of the incoherent application of concepts by LLMs, showing how benchmarks that reliably establish domain competence in humans can be passed by LLMs lacking similar competence.
H/T @acowley
The paper makes the case that when humans misunderstand things, they usually do so in a very different way than LLMs — and that exams that are genuinely pretty good at assessing humans’ competence can nonetheless be terrible at assessing LLMs.
This doesn’t mean all current exams are perfect at assessing humans, but it’s a big [and unjustified] leap to go from “an LLM can ace this exam while having no competence at applying the relevant concepts” to “a human who aces this exam might understand as little as the worst LLM that could also ace this exam”.
We might have to change exams to catch out humans who are cheating with real-time assistance from concealed LLMs, but that’s a different matter.
This. So much this. I had the following exchange with Claude a couple of months ago (lightly condensed and paraphrased)
Me: draw the structure of naphthalene, please
Claude: (produces something with the right shape and general connectivity of the atoms, but one carbon atom has five bonds, a classic oopsie in Intro Organic Chem)
Me: one of the carbon atoms has the wrong number of chemical bonds. Please fix it.
Claude: I’ve redrawn it. (Its effusive self-directed praise about how perceptive I am and how it successfully redid the drawing has been deleted.)
(a human who understands carbons are limited to 4 bonds, while also usually having exactly 4 bonds in neutral molecules, would find the error and fix it easily. Claude did…not. Its revised structure has *four* carbon atoms with five bonds.)
Me: your structure has four carbon atoms with the wrong number of bonds.
Claude: I’ve redrawn it. (Again, much obsequiousness deleted.)
(Claude has drawn the same exact structure as the previous one, except a couple of the bonds have been shifted by a couple of pixels.)
That was Claude. I tried the same thing on Gemini and it was immeasurably worse, culminating in a structure that had carbon bound to three other carbons *and three hydrogens* and that wasn’t even the weirdest thing about it.
@gregeganSF @acowley Ah this ratio of 19/20 accuracy explains a lot!
This kind of accuracy/inaccuracy ratio is actually really difficult for humans to understand.
It is accurate enough times to throw a facade over it’s nonunderstanding, but not accurate enough to trust in the long run.
It passes just under our instinctive thresholds of validating a person’s understanding of a concept. We let it pass, as a human who gets 19 out of 20 right ”gets it” and would improve, where an #ai doesnt
@gregeganSF @acowley
It's great to have this work, to show how (depending on the viewpoint) models are able to solve some problems even though they don't understand, or some benchmarks are too easy to solve by pattern matching and don't measure understanding.
It would be even nicer if they used best practices when invoking the models, such as chain of thought prompting. Especially given that some of the problems require deliberate consideration when people do them. And some of the problems touch on known issues with language understanding, such as counting syllables. But most of the tasks are free of these.
"incoherent application of concepts"
Reminder that no concepts are involved. in a random (Markov) walk through word space. Shannon 1948.
From Pogo: "We could eat this picture of a chicken, if we had a picture of some salt."
I suppose you must be referring to Pogo, which is not, for the present purposes, even a word space (or: not fruitfully treated as such).
Are you trying to distinguish tokens and words?
Or do you have a point? If so, what is it?
I'd say there is syntax without semantics (in the traditional sense of formal logic, that is).
You have some other view evidently.
That much is now clear.
I don't see much difference from Markov and Shannon, apart from some compression tricks which are needed to get a working system.
That no concepts are involved, and the numerous corollaries of that, I suppose. At least, that's what I find myself harping on now and then.
I have no strong interest in the details. though considerable interest in watching this play out.
—Someone like Cosma Shalizi is going to actually get into the weeds a bit more:
http://bactra.org/notebooks/nn-attention-and-transformers.html
You'll probably find much to agree with and much to disagree with there. And at adequate length.
@glc > I find this literature irritating and opaque.
That's a promising start! (8
Matthias Kirschner
Greg Egan
Mastodon
Chris Lilley 🏴
Gabriel Stein
Gina Häußge
Ian Campbell
Ars Technica
Steveg58
Kevin Granade
David Nash
Toni Aittoniemi
Chaucerburnt
Amir Livne Bar-on
GLC
l'empathie mécanique
Cegorach
Metafrastis
Ariaflame