Prof. Emily M. Bender(she/her)Jun 8, 2023

From a live tweet of the proceedings around the lawyer caught using ChatGPT:

"I thought ChatGPT was a search engine".

It is NOT a search engine. Nor, by the way are the version of it included in Bing or Google's Bard.

Language model-driven chatbots are not suitable for information access.

>>

Show threadJW prince of CPHJun 8, 2023

@emilymbender - but the misunderstanding is understandable, as providers of these "AI"-driven services are extremely, seemingly deliberately obtuse about what they're actually for.

Because if they said "this is a machine that can imitate sounding factual but has absolutely no fact-checking layer or mechanism, nor are we planning to add any such thing", then most people would wonder what the hell they're supposed to use it for at all.

Show threadBrad EdwardsJun 9, 2023

@jwcph @emilymbender But they have said that. Repeatedly. And people have been yelling about that since ChatGPT was released. Repeatedly.

We can infantilize the public and put all of it on the companies, or hold people accountable for their mistakes. Disbar the lawyer, the others will learn right quick. Otherwise it's like banning Tide because some people thought eating Tide pods was fun.

Show threadJW prince of CPH
@bradedwards @emilymbender Please provide three examples.
Jun 9, 2023 at 5:56amWeb
Show threadBrad EdwardsJun 9, 2023

@jwcph @emilymbender

OpenAI ChatGPT:
"ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 24 Version"

https://openai.com/gpt-4

"GPT-4 still has many known limitations that we are working to address, such as social biases, hallucinations, and adversarial prompts. We encourage and facilitate transparency, user education, and wider AI literacy as society adopts these models. We also aim to expand the avenues of input people have in shaping our models."

GPT-4

Show threadBrad EdwardsJun 9, 2023

@jwcph @emilymbender

The news stories on hallucinations, much less the research papers, industry coverage, etc are endless:

https://www.bloomberg.com/news/newsletters/2023-04-03/chatgpt-bing-and-bard-don-t-hallucinate-they-fabricate

https://www.cspinet.org/blog/chatgpt-amazing-beware-its-hallucinations

https://www.nytimes.com/2023/05/01/business/ai-chatbots-hallucination.html

Law societies themselves have warned before this about problems with AI:

chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://www.lco-cdo.org/wp-content/uploads/2021/03/LCO-Govt-AI-Workshop-Report-%E2%80%94-March-2021.pdf

AI Doesn’t Hallucinate. It Makes Things Up

There’s been so much talk about AI hallucinating that it’s making me feel like I’m hallucinating. But first…

Bloomberg
Show threadBrad EdwardsJun 9, 2023

@jwcph @emilymbender

After a certain point, "Don't touch the electric fence" means don't touch it.

Show threadJW prince of CPHJun 9, 2023
@bradedwards @emilymbender I get what you mean, but at no point do the vendors (!) tell people that the thing *was not designed to be accurate, and has no mechanism for accuracy at all*. That's why they call it "hallucinations", implying that it's a glitch, and not the thing doing exactly what it's designed to do: Make. Sh*t. Up.
Show threadBrad EdwardsJun 9, 2023
@jwcph @emilymbender Sort of? Bing, you.com, Scite etc. They are explicitly positioning their tools as accurate within the bounds of what is accurate for their markets; search, academic databases, etc. And LLM systems can be engineered for that, primarily by using them only as summarization engines over standard search. I haven't seen any data comparing their accuracy to normal search.
Show threadBrad EdwardsJun 9, 2023

@jwcph @emilymbender ChatGPT, Bard, Pi, etc haven't been marketed as truth engines. There's a million pounds of info out about how they're not and how LLMs work. For free. Everywhere online.

I'm just entirely unsympathetic to a professional who abdicates their responsibility by adding a tool to their practice without even a modicum of research into how it works. I don't see how that level of negligence bears on the chat bot providers.

Show threadJW prince of CPHJun 9, 2023
@bradedwards @emilymbender Oh, me too; I'm not feeling sorry for that guy at all - but I don't agree that these things aren't marketed as truth engines, because that's exactly what is implied. Especially since, as I keep saying, the vendors never tell people that it is *not designed for accuracy at all & has no mechanism for fact checking, nor is it intended to have one*. Also, the whole "hallucinations" thing.