Super frustrated with all the cheerleading over chatbots for search, so here's a thread of presentations of my work with Chirag Shah on why this is a bad idea. Follow threaded replies for:
op-ed
media coverage
original paper
conference presentation
Please boost whichever (if any) speak to you.
Chatbots are not a good replacement for search engines
https://iai.tv/articles/all-knowing-machines-are-a-fantasy-auid-2334
The idea of an all-knowing computer program comes from science fiction and should stay there. Despite the seductive fluency of ChatGPT and other language models, they remain unsuitable as sources of knowledge. We must fight against the instinct to trust a human-sounding machine, argue Emily M. Bender & Chirag Shah.
Chatbots are not a good UI design for information access needs
https://www.technologyreview.com/2022/03/29/1048439/chatbots-replace-search-engine-terrible-idea/
Chatbots-as-search is an idea based on optimizing for convenience. But convenience is often at odds with what we need to be doing as we access and assess information.
Chatbots/large language models for search was a bad idea when Google proposed it and is still a bad idea even when coming from Meta, OpenAI or You.com
Language models/automated BS generators only have information about word distributions. If they happen to create sentences that make sense it's because we make sense of them. But dis-connected "information" inhibits the broader project of sense-making.
https://www.youtube.com/watch?v=VY1GHbU_FYs&list=PLn0nrSd4xjjY3E1qxXpWDoF7q-Q3d6g_A&index=17
We must not mistake a convenient plot device — a means to ensure that characters always have the information the writer needs them to have — for a roadmap to how technology could and should be created in the real world.
https://mindmatters.ai/2022/12/why-we-should-not-trust-chatbots-as-sources-of-information/
@arildsen @emilymbender Well the thing with chatbots like ChatGPT is that they are very good at exactly that: giving you an IMPRESSION that they are good at something.
But they will absolutely lie through their teeth to do it, and it will be believable lies.
@WAHa_06x36 @arildsen @emilymbender This is kinda a category error, isn't it. As well-argued here, language models are incapable of producing factual statements, correct or incorrect. They can only produce poetry.
Unfortunately, we lack the language and metaphor to talk about statistical text generators and the human tendency to see peopleness everywhere doesn't help.
@WAHa_06x36 I think it's quite an important distinction? It's fundamental to how you should interpret text generated by a language model.
If you paint two dots and a downward facing semicircle on a rock, people immediately interpret the rock as being sad - : (
But we all know rocks can't be sad.
Similarly, language models are a really complicated pattern painted on a rock. The text they generate isn't true or false statements; it's randomly generated truthy. Many of the texts it generates will be interpreted as true statements, because lots of truthy strings are representations of true statements.
But saying GPT-3 lies suggests that you could make a language model that doesn't lie, or that isn't cavalier with the truth, and that's the wrong way to think about them.
Everyone knows rocks can't be sad; they don't know that language models can't tell the truth, but it's the same human cognitive failing that generates both.
Prof. Emily M. Bender(she/her)
Dag Ågren ↙︎↙︎↙︎
RAOF