Meta. OpenAI. Google.
Your AI chatbot is not *hallucinating*.
It's bullshitting.
It's bullshitting, because that's what you designed it to do. You designed it to generate seemingly authoritative text "with a blatant disregard for truth and logical coherence," i.e., to bullshit.
@jimproctor @martinvermeer @dezene @bikerglen @ct_bergstrom IMO, this is the (awkward) way forward for these LLM-based attempts at "AI". Take the LLM generated content, run it through some sort of meta-analysis, and then present that to the user.
I mean, in a way they're already doing that to suppress the nazi stuff.
In your case, this hypothetical tool would be capable of doing a meta-analysis of your document and the LLM-generated document on the same subject.
@jimproctor @martinvermeer @dezene @bikerglen @ct_bergstrom Of course, you can't ask the LLM to do the meta-analysis. So we're relying on an as-yet non-existent technology to make the LLM useful in a practical way.
At which point, why are we using the LLM?
@tob @jimproctor @martinvermeer @bikerglen @ct_bergstrom
Really all I want is an AI that could retrieve, fill, and send forms to the next approval stage with a few simple typed or spoken commands.
@pjie2 I would not even put it that morally judgemental.
As long as you have no clue, you don’t know that you’re bullshitting.
I have quite a few old articles of mine that I changed over and over again, adding corrections, updates, or clarifications when I learned more about a topic.
But those LLM's do not know whether I have the expertise on a specific topic that my writing is actually well-founded.
And I found that the more I understood the less I wrote unasked.
@moultano Humans have an underlying knowledge model. They have beliefs about the world, and choose whether to represent those beliefs accurately or inaccurately using language.
LLMs do not have an underlying knowledge model, they don't have a concept of what is true or false in the world. They just string together words they don't "understand" in ways that are likely to seem credible.
It's not a matter of making better LLMs; it'll take a fundamentally different type of model.
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.
@TedUnderwood @moultano @ct_bergstrom @not2b
People who understand the technology of large language models aren't dismissing it as "inherently just bullshit generation", but they are warning that its output smoothly mixes both fact and falsehood with no distinction or care.
And they warn that the quantity and impact of this #bullshit could likely surpass that of #politics, #consumerism, and other forms of rampant #disinformation for which we humans have demonstrated we are poorly prepared.
@ct_bergstrom @moultano I'm starting to have doubts about the idea tha LLMs are "stochastic parrots" that can't generalize after watching a short talk from Francois Charton of Meta at NeurIPS 2022.
TL;DR - he trained a small LLM to learn how to diagonalize matrices using only triplets of the similarity transform. No hallucinations were observed.
The talk was "Leveraging Maths to Understand Transformers" https://neurips.cc/virtual/2022/workshop/50015#wse-detail-63846
@ct_bergstrom @moultano @marshall_0i
without doubt LLMs are not like humans, but this statement assumes we know what it means a) for humans to “understand” and b) what processes that rests on, so that we can compare which of these are and not present in LLMs, and c) seems to assume “understand” (whatever it is) is all or none.
As a cognitive scientist, I personally wouldn’t subscribe to any of the three.
see e.g., https://www.journals.uchicago.edu/doi/abs/10.1086/443791
@marshall_0i @ct_bergstrom @moultano
I don’t get that argument: they are incredibly powerful in what they do, well beyond what anything in NLP did 20 years ago. People now spend days trying to find, and then excitedly communicate things that ‘break’ them.
I think it’s a fundamental mistake to conflate whether they should be let loose on an unsuspecting public as a readily available tool (no!) or used in search (no!) with how they relate to human cognition.
1/
@marshall_0i @ct_bergstrom @moultano 2/ so much of the latter discourse seems to proceed *as if we understood human cognition*, which, as a cognitive scientist, I very much feel we don’t.
Top notch cognitive scientists consider them to be interesting and worthy of consideration as models of cognition. That should maybe give people some pause for thought here. I’ve linked to these two recent talks before:
by Ellie Pavlick
https://youtu.be/1_wTMxdVgOI
by Noah Goodman
https://youtu.be/dYXxkS4rrAs
@marshall_0i @ct_bergstrom @moultano 3/ I think they are demonstrably the closest anything has come to generating surface behaviours that look like human performance when it comes to generating and responding to language. So taking them seriously in that context makes sense to me.
Their scientific value and their suitability as online tools ready for current deployment are two separate matters.
That's fairly clever, and it could be implemented as a browser plugin.
All the search engines have old school tools for validating certain content on the web. Various sites are rated for higher accuracy than others and that has been tweaked and modified over the last two decades. Thats not the hard part.
The hard part is just as you say, they are trying to convince people ‘This one new tool is the answer!’ and slapping a 59% confidence metric on a statement about the JWST doesn’t look nearly impressive enough.
This is how Google could maintain its Search supremacy, and its golden goose. They list all the sites that back up their LLMs output with a page of links, or more like the sidebar thing that Bing is using.
I would love a Open Source rating system, but you are right it is hard to deobfuscate their inner workings.
My company, Tomorrowish, built a rating system for Tweets more than a decade ago, and though it gave good results, finding out WHY was very tough.
@jt_rebelo @ct_bergstrom And that was our own internal tool we had trouble understanding!
Admittedly, we were a tiny team running a startup on a shoestring, and constantly trying to adjust to pull in some more revenue, SO, we didn't have a lot of time to investigate rigorously something that already worked.
🤷
I think a Google, or Microsoft level company could pull it off. with their personnel and resources.
cjr
Carl T. Bergstrom
Glen Akins
Dezene Huber 🌻
Tom Bellin 
Martin Vermeer FCD
Jim Proctor
grob (teeth era) 🇺🇦🏳️🌈🏳️⚧️
FishMom2022
Supportive Tree-based lifeform
Kyozou
Sidney Alcantara
Ryan Moulton
Peter Ellis
ArneBab
Joe ❌👑
Leo
Ted Underwood
Jef Allbright
Ben Carver
Nicole Göbel
Gnome Kat
Frank Bennett 🇯🇵
Joe
Ulrike Hahn
Django
Pete Bleackley
Paddy
Dieu
Anca Buit
itatiVCS
Mick Darling
rastilin
João Tiago Rebelo