Super frustrated with all the cheerleading over chatbots for search, so here's a thread of presentations of my work with Chirag Shah on why this is a bad idea. Follow threaded replies for:
op-ed
media coverage
original paper
conference presentation
Please boost whichever (if any) speak to you.
Chatbots are not a good replacement for search engines
https://iai.tv/articles/all-knowing-machines-are-a-fantasy-auid-2334
The idea of an all-knowing computer program comes from science fiction and should stay there. Despite the seductive fluency of ChatGPT and other language models, they remain unsuitable as sources of knowledge. We must fight against the instinct to trust a human-sounding machine, argue Emily M. Bender & Chirag Shah.
Chatbots are not a good UI design for information access needs
https://www.technologyreview.com/2022/03/29/1048439/chatbots-replace-search-engine-terrible-idea/
Chatbots-as-search is an idea based on optimizing for convenience. But convenience is often at odds with what we need to be doing as we access and assess information.
Chatbots/large language models for search was a bad idea when Google proposed it and is still a bad idea even when coming from Meta, OpenAI or You.com
Language models/automated BS generators only have information about word distributions. If they happen to create sentences that make sense it's because we make sense of them. But dis-connected "information" inhibits the broader project of sense-making.
https://www.youtube.com/watch?v=VY1GHbU_FYs&list=PLn0nrSd4xjjY3E1qxXpWDoF7q-Q3d6g_A&index=17
We must not mistake a convenient plot device — a means to ensure that characters always have the information the writer needs them to have — for a roadmap to how technology could and should be created in the real world.
https://mindmatters.ai/2022/12/why-we-should-not-trust-chatbots-as-sources-of-information/
@arildsen @emilymbender Well the thing with chatbots like ChatGPT is that they are very good at exactly that: giving you an IMPRESSION that they are good at something.
But they will absolutely lie through their teeth to do it, and it will be believable lies.
@WAHa_06x36 @arildsen @emilymbender This is kinda a category error, isn't it. As well-argued here, language models are incapable of producing factual statements, correct or incorrect. They can only produce poetry.
Unfortunately, we lack the language and metaphor to talk about statistical text generators and the human tendency to see peopleness everywhere doesn't help.
@emilymbender @WAHa_06x36 @arildsen Urgh, sorry.
Thanks for the heads up. It's sad that some people have made AI a gateway to that cluster of terrible thinking.
That blog post only refers to a piece of Gwenn's work in the opening couple of paragraphs, as framing.
The author doesn't seem to be in the SSC/Rationalist/scientific racism orbit. Maybe they don't know? (I'll try to contact them)
Thanks again for the heads up.
@WAHa_06x36 I think it's quite an important distinction? It's fundamental to how you should interpret text generated by a language model.
If you paint two dots and a downward facing semicircle on a rock, people immediately interpret the rock as being sad - : (
But we all know rocks can't be sad.
Similarly, language models are a really complicated pattern painted on a rock. The text they generate isn't true or false statements; it's randomly generated truthy. Many of the texts it generates will be interpreted as true statements, because lots of truthy strings are representations of true statements.
But saying GPT-3 lies suggests that you could make a language model that doesn't lie, or that isn't cavalier with the truth, and that's the wrong way to think about them.
Everyone knows rocks can't be sad; they don't know that language models can't tell the truth, but it's the same human cognitive failing that generates both.
The positive/useful ChatGPT examples I’ve seen have mostly been coding examples.
@emilymbender The points you make about Trust are really critical. These language models are “often wrong, never in doubt.”
I have found that when asked to cite sources, OpenAI model will generate plausible-but-false URLs.
@emilymbender Hah. That's some... loaded wording, while we're speaking of trusting interactions by default.
Raw language models are a terrible way to acquire information, but they have lots of potential as an interface, given that there is already a lot of quick info search happening through language recognition through "dumb" assistant software.
The right place for a chatbot in the UX isn't as the search engine, it's as a parser of the query/result in some applications like text-to-voice.
@emilymbender And even then I'd suggest that for certain applications, like as accessibility tools, the downsides in information accuracy may be tolerable.
I'm also curious about the circularity of the argument that search engines share some of the same problems. Yeah, they do, and the pushback against them in the 90s looked a lot like this, too. But that ship has sailed, hit an iceberg and sunk in the middle of the ocean. We are starting from a search engine-saturated world already.
@emilymbender Which is to say, dumb algorithms are already pushing bias, we are already giving them too much credit and they already can be compromised by hostile techniques like SEO.
I don't think AI chat is inherently more believable because it sounds more human, that's just the uncanny valley of seeing new tech. We should hope to design these to do better than the old tech, but surely the bar for usability is to not do worse, which is much easier.
@emilymbender Alright, I'll stop threading and ranting, but just one more warning. A lot of both reasonable and unreasonable observations and criticism of generative AI, chatbots and the like is starting to degenerate into straight bias against ML, which is dangerous. ML is already ubiquitous and crucial to lots of fields, from astrophysics to game development.
Let's be careful to not let reasonable warnings about big data devolve into technophobia against the research field in general.
The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here, we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of GPT-3) on a range of analogical tasks, including a novel text-based matrix reasoning task closely modeled on Raven's Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.
@ar_lt I try not to spend too much time with preprints, i.e. work that has not been vetted by experts.
But: From the abstract, it sounds like they are trying to inject some external information (the meaning of negation).
Also: "latent knowledge" is an unfortunate overstatement.
Prof. Emily M. Bender(she/her)
Dag Ågren ↙︎↙︎↙︎
Carola Schubert
RAOF
halcy 
John Gordon
Beyond the Arc
Steve Torrente
MudMan
Antoine Rivoire
Matt Stegmeir
Red Faster