rateexportpilot
Jason GormanWhat makes LLMs work isn't deep neural networks or attention mechanisms or vector databases or anything like that.
What makes LLMs work is our tendency to see faces on toast.
ggdupont@jasongorman
100%
I've been telling this for a while now.
Our tendency for pareidolia goes way beyond just seeing faces. We're desesperately trying to connect other intelligent beings...
100%
I've been telling this for a while now.
Our tendency for pareidolia goes way beyond just seeing faces. We're desesperately trying to connect other intelligent beings...
circfruit@gdupont @jasongorman I didn’t know there’s a word for that 🤨
@circfruit
@jasongorman
There is a large research corpus in human cognition which studies how people think and how our thinking can be (easily) fooled.
It is usually completely ignored by AGI fanboys (so it's easier to make grand claim about artificial cognition)
GLC@circfruit @gdupont @jasongorman
And a nice page for it as well.
https://en.wikipedia.org/wiki/Pareidolia
Pareidolia + theory of mind: mentalist effect
LLM: stochastic parrot
Mix and serve
Peter Middelkoop@jasongorman A thought that slipped trough my mind also.
JWcph, Radicalized By Decency@jasongorman No, seriously - that's exactly what it is & it's a mechanism exploited by charlatans since always: https://softwarecrisis.dev/letters/llmentalist/
Kyle Memoir 🍉🐧A bit long, but good reading.
maya🌻@jwcph @jasongorman I’ve always felt like the way chatbots speak was a little too general and assertive for me to believe/trust them, and I’m glad to learn *why* it is like that. The psychic comparison is very effective.
Timothy Wolodzko@jasongorman jokes aside, this is a very good metaphor
@tymwol I'm not even sure it's a metaphor 🙂
Infrapink (he/his/him)It's called the ELIZA effect, and we've known about it since 1966: https://en.wikipedia.org/wiki/ELIZA_effect
Pete@Infrapink @jasongorman @tymwol
I think it's also related to the #PeterPrinciple... that people get promoted to a level of incompetence.
The chat part of #ChatGPT is really important because it allows us to correct the LLM's output and give it another chance to completely fool us. When we're satisfied, the output is above our present ability to detect that it is BS.
By "present ability" I mean we might not feel bothered to check the output, or we genuinely might think it's correct.
Óscar Morales Vivó@pete @Infrapink @jasongorman @tymwol It should also be noted that the people most in love with the bullshit-generating machine make a living out of generating bullshit themselves.
ꓤɔᴉʇɐʇS@jasongorman A streamer I watch has a bot that's literally just a random number generator picking from a set list of predefined phrases that runs on a 10 minute timer or when prompted by chat and you wouldn't believe how shockingly often its totally on point "reacting" to what's happening on screen or responding to chat. So much so that we've been joking it is actually sentient.
Actual chatgpt which that streamer used before that had less hits than the literal RNG list on a timer.
Duco@StaticR @jasongorman do you have an example of a question answered by the RNG list?
@duco since they were all in twitch live chats no. There's probably a couple clips of particularly funny responses but digging them up might be a bit challenging.
Tane Piper ⁂@StaticR @jasongorman Cold reading
Stephen Bannasch (316 ppm)Pareidolia is fun and cool unless you are fooling yourself that the faces you see are real and they are talking to you.
Efi (nap pet) 🦊💤@jasongorman if it looks like a duck and it quacks like a duck, it can't possibly be an artificial duck
William Pietri@jasongorman Yes! This plus what I think of as the Stone Soup Effect. A claim of magic gets people to try it, but they don't notice that they're the ones doing most of the value-creating work.
Dec [{(
)}]@williampietri @jasongorman
"Finding books in babbling brooks"; that's either Shakespeare or Benford.
"Finding books in babbling brooks"; that's either Shakespeare or Benford.
Yvo Verschoor@jasongorman just as the “I want a lollipop” quote from the last #securitynow podcast*, this is also a nice piece of wishdom for Leo la Portes ( @leo ) quotes collection. I think 😇.
* = see page 15 of: https://www.grc.com/sn/SN-1027-Notes.pdf
* = see page 15 of: https://www.grc.com/sn/SN-1027-Notes.pdf
Max Leibman@jasongorman @purplepadma Right on. As I’ve said so many times, I hate the term “hallucination” because it implies there is something in there that perceives and that therefore can hallucinate. In fact, the only hallucination is happening on our side of the keyboard.
Allan@jasongorman oooh that’s good.
infryq@jasongorman unfortunately, I feel like our tendency to see faces on toast is also what makes human society work
Norm Tovey-Walsh@jasongorman And staggering amounts of theft, of course.
D2@jasongorman nice. Brilliant phrasing vs. we *are* pattern-seeking machines.
Arnim Sommer 🇪🇺@jasongorman
Well, computers are doing something that we interpret as mathematics…
Well, computers are doing something that we interpret as mathematics…
Tom Stoneham@ArnimRanthoron @jasongorman
We have good theories explaining how syntactic rules can implement semantic rules. In fact, nearly 200 years of formal logic.
We also have good theories explaining why a next token generator does not implement linguistic understanding. But sadly these are not well understood outside linguistics and philosophy of language.
Tattie@jasongorman are you implying that Toastface isn't real?
Don't listen to him, Toastface.
eyajpng@Tattie @jasongorman my favorite member of wu tang
Paul SomeoneElse
Raymond Russell@jasongorman
LLM's are doing an automated Mentalism act.
LLM's are doing an automated Mentalism act.
Earthshine@jasongorman I mean if we're really getting down to it, one might argue rather that LLMs just don't work, but our tendency to see faces on toast is what makes people able to sell them anyway.
Weekend Editor"The fault, dear Brutus, lies not in our stars but in us."
We are the ones hallucinating rationality. (In our defense, LLMs pretty much optimize for plausibility.)
Bill the Lizard@jasongorman The Gell-Mann Amnesia Effect is also doing some of the heavy lifting. https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect
Shafik YaghmourThis piece "Language Is a Poor Heuristic for Intelligence": https://ninelives.karawynnlong.com/language-is-a-poor-heuristic-for-intelligence/
Really stuck with me, that we have "fluency heuristic" and LLMs are really good at fooling that.
Which is similar effect but explains more.
It really makes you realize how naive the "Turing Test" really was.
jnpn@jasongorman on a slightly smaller scope, I also believe that LLM bring less directional and highly contextual interfaces, which is something I don't recall seeing really before and was also one massive constraint in all software engineering
diaeter@jasongorman Pareidolia?
Maverynthia🌱Seeing things in a random assortment of shapes. Like "That cloud looks like a sheep", "These stars look like a bear.", "This toast looks like the face of Mother Mary if you squint".
@Maverynthia @jasongorman Pareidolia helped us survive by discovering one more sabre tooth tiger in bushes than there actually was...
Arianne B@jasongorman Thanks. I've wondered about that. A close friend ADHD relies heavily on a LLM tool to support themself psychologically and regulate their nervous system, as well as help them write cogently about their work. The personal support stuff they get from it reminds me of like a Tarot reading: an array of symbols we interpret to to help orient us in an otherwise scarily disordered world. Or,as you say, like seeing faces on toast. IMO Tarot and similar tools are very helpful and can be used immensely positively - but lets not overrate what's happening there.
Dave C.@adb @jasongorman had someone come to my home recently to repair my HVAC system. Individual told me they were ADHD, they shared a lot of things I don't need to know. But he was using ChatGPT to try and troubleshoot my AC. I got tired of that real quick. He somehow heard logic in the word salad it produces. I heard meandering assembly of words that could take you any direction you want. I could see how it helping in one aspect of life would lead to it impinging on others.
Ozzelot 
@davesomebody @adb @jasongorman I may or may not have significant ADHD and autism traits, but it makes me seek controlled, intentional information, rather than the ad hoc generated texts produced by a bunch of GPUs.
@ozzelot @adb @jasongorman I think some of us know too well how it works to trust it. It is great at producing walls of word salad. And to some people that is great. If you value precision LLMs are just too random.
europlus 
@jasongorman dog butt Jesus.
@jasongorman just returned to my browser which still had that image search tab open from when I’d done it early this morning.
I was…taken aback! 😵💫
Greg@jasongorman It amazes me that highly educated CEOs see faces on toast. Or maybe they just say they do to mislead investors with plausible deniability.
Oatmeal RaisinThey don’t have to think, reason, or know anything at all to work, no more than any other algorithm. On the rare occasion I use them, they more often than not work quite well. Here’s an example from a couple of days ago that worked perfectly on the first try with 0 pareidolia.
Negative Kelvin@jasongorman Ceci n'est pas une pipe.
Marsh Ray@jasongorman What is the name for when otherwise intelligent people make absurd and easily disproven claims?
it's B! Cavello 🐝@jasongorman @adamr I agree, but I also think it’s lots and lots of data (and human effort)
Semitones
GeePawHill@jasongorman Fuck, Jason. That's quotable.
@GeePawHill I feel a t-shirt coming on 🙂
@jasongorman @GeePawHill i can imagine like
first panel has the first line as a caption for a picture of this big looming shiny metal machine
second panel has the last line and the same machine, but the toast is done and has popped up
first panel has the first line as a caption for a picture of this big looming shiny metal machine
second panel has the last line and the same machine, but the toast is done and has popped up
@jasongorman @GeePawHill (of course the toast has faces on them)
Moss (the τεχie) <(fsm)>{{