Florian KräutliDec 4, 2022
Prof. Emily M. Bender(she/her)

With #Galactica and #ChatGPT I'm seeing people again getting excited about the prospect of using language models to "access knowledge" (i.e. instead of search engines). They are not fit for that purpose --- both because they are designed to just make shit up and because they don't support information literacy. Chirag Shah and I lay this out in detail in our CHIIR 2022 paper:

https://dl.acm.org/doi/10.1145/3498366.3505816

>>

Situating Search | Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

ACM Conferences
Dec 4, 2022 at 6:09pmWeb
Show threadProf. Emily M. Bender(she/her)Dec 4, 2022

For a lay introduction to these ideas, see these two popular press pieces:

https://www.technologyreview.com/2022/03/29/1048439/chatbots-replace-search-engine-terrible-idea/

https://www.washington.edu/news/2022/03/14/qa-preserving-context-and-user-intent-in-the-future-of-web-search/

Chatbots could one day replace search engines. Here’s why that’s a terrible idea.

Language models are mindless mimics that do not understand what they are saying—so why do we pretend they’re experts?

MIT Technology Review
Show threadL Thorne McCartyDec 4, 2022
@emilymbender

This thread from Twitter makes a similar argument with some background information about Google's search strategy:

https://twitter.com/deliprao/status/1599098378172104704?t=QlQFV6P3OServvtYepryHg&s=19
@[email protected] on Twitter

“Despite the amazing results I’ve experienced with ChatGPT, this is not a correct way to look at LLM vs. Google search. Since several other tweets have made this equivalence and have been eager to spell doom for Google, let’s examine the details:”

Twitter
Show threadProf. Emily M. Bender(she/her)Dec 4, 2022
@ltmccarty The initial proposals that Shah & I were reacting to in our paper came from ... Google (including Sundar Pichai himself at GoogleIO).
Show threadElisa Beshero-BondarDec 4, 2022
@emilymbender Thanks for sharing these! I’m adding as good readings to my #digitalhumanities syllabi for spring.
Show threadrajesh bilimoriaDec 4, 2022
@emilymbender In addition to the TR piece, your interview from May in pnw.ai might also be useful here, particularly for its point about terms and framing (and reference to Stefano Quintarelli's alternative term for AI: “SALAMI” = “systematic approaches to learning algorithms and machine inferences”) https://pnw.ai/article/the-problem-with-overestimating-ai/121722775
The problem with overestimating AI

Discover the latest artificial intelligence news, jobs and more.

pnw.ai
Show threadUna MilutinovichDec 4, 2022
@emilymbender I've been trying to get it to explain physics topics, and what's it's good at is bullshitting. It uses the right terminology, but it usually gets the details wrong. Like a student who hasn't studied for the exam!
Show threadBobbi TowersDec 4, 2022
@emilymbender they just need to iron out a few bugs, like *making sure results are actually correct*
Show threadollibabaDec 4, 2022

@emilymbender

> Summarize the paper "Situating Search" by Chirag Shah and Emily M. Bender.

> I'm sorry, but I am not able to summarize the paper "Situating Search" by Chirag Shah and Emily M. Bender. As a language model, I don't have access to external resources like research papers, and my training only goes up until 2021, so I am not able to provide current information. Is there anything else I may be able to help you with?

Guess I actually have to read on my own ;-)

Show threadflintpopeDec 5, 2022
@emilymbender I came here for the art but find this fascinating. Thanks for this.
Show thread🌊 Jumpsuits For Trumpsuits 🌊Dec 5, 2022
@emilymbender Today it convinced me that 2+2=5.
Show threadArclightDec 5, 2022
@jumptytrumpty @emilymbender Are you sure it wasn't just explaining perturbation theory? 2 + 2 is approximately 5 for moderately large values of 2...
Show thread🌊 Jumpsuits For Trumpsuits 🌊Dec 5, 2022
@arclight @emilymbender No idea! Like I said, I'm no good at math. I'm just curious what others think. But now I'm going to ask it about perturbation theory of course!
Show thread🌊 Jumpsuits For Trumpsuits 🌊Dec 5, 2022

@arclight @emilymbender

🤯

Show threadArclightDec 5, 2022
@jumptytrumpty @emilymbender To be fair, approximate methods like perturbation theory and asymptotic analysis are difficult niche topics (barely survived them in grad school). They're mainly used to get close-ish answers to problems that can't be solved by pure math (symbol manipulation) or numerical approximation (sample everything very finely and add up all the samples). It's a technique that gets used when nothing else works; a cure as bad as the disease...
Show threadArclightDec 5, 2022
@jumptytrumpty @emilymbender That's not a bad answer tbh...
Show threadGiulioDec 5, 2022
@emilymbender
Sometimes chat bots can be better than humans when it comes to supporting about technical questions.
Show threadGiulioDec 5, 2022

@emilymbender

After hours and a few misleading replies, I found the correct solution by myself, as usual.

Asking open #AI chatGPT would have been likely better😂

On top of the current, good Google search engine: they are not mutually exclusive alternatives!

https://ask.fedoraproject.org/t/f36-boot-entries-after-new-fedora-37-installation/29411/37

F36 boot entries after new Fedora 37 installation

Yes, I’m very close to write a “fix” to the os_prober, just for the sake of it, lol. At least to simplify the manual management of the entries, in case I want to keep those two Fedora versions and those two partitions FYI, my fix: mykernel="${LKERNEL/"/vmlinuz-"/}" title="${LLABEL} ${mykernel} $onstr" instead of title="${LLABEL} $onstr" and I can refresh the entries with sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Ask Fedora
Show threadasha_wrenDec 5, 2022
@emilymbender this is very interesting!
Show threadpmcmDec 5, 2022

@emilymbender

Strong agree on this. I've been experimenting whith ChatGBT for problem solving as I'm learning #JuliaLang. Most of the time, it can give me a good answer to a question about syntax or available functions. But pretty regularly it will also just make up some example solution that looks like Julia code but produces garbage output.

Show threadMaxDec 5, 2022
@emilymbender Can we be excited about computers getting better at understanding natural language and its context as an input? Is that the big shift when it comes to #ChatGPT and #Galactica? Maybe they cannot provide accurate answers, but they help computers understand what we mean when we talk to them.
Show threadjust a💧in my 🍺Dec 5, 2022
@max @emilymbender in fact i tried out chatgpt for very different questions and it gave me correct, precise and understandable answers most of the time. Saying that they just make things up and cannot give correct answers just shows a lack of knowledge about this tool. Those AI agents will be the fundamental part of every human machine interaction in the near future.
Show threadZacDec 5, 2022
@max @emilymbender They can't do that Max because they are neural networks that just create an output without being aware of what they are doing.
Show threadMaxDec 5, 2022
@lolzac I know what they are and how they work, but this is not a reason why they cannot get better at understanding language. ChatGPT is one example. Image recognition is another one. Neural networks don’t need to be sentient or "know" what they are doing to provide outputs that are useful for us.
Show threadMike BellDec 5, 2022
@emilymbender Yep - here's a fairly simple maths question ChatGPT gets wrong while giving a long winded explanation that has a mistake half way through (bottom of the screen grab)
Show threadDavidDec 5, 2022
@emilymbender imagine an llm that cites its sources. the model would be sued out of all profits.
Show threadZacDec 5, 2022

@emilymbender People forget that these systems are just building output based on rules that it created to match data.

For some time I have been correcting people when they call these types of systems AI and not machine learning. There is no "thought" in any of the content being generated.

Show threadJ Paul GibsonDec 5, 2022
@emilymbender - they just “make shit up” is a great summary of what they do. so now all we need to do is to teach/inform how to detect such bullshit. unfortunately, in social media, good bullshitting is rewarded at least as much as good authoritative writing.
Show threadPiotr KaniaDec 5, 2022
@emilymbender In your opinion, can ChatGpt be considered a worthy rival to DeepL- when it comes to text translation?
Show threadMark McClellandDec 5, 2022
@emilymbender Discovering that a chat bot's data is inconsistent with its sources is often trivial. I've had fun asking GPT-3 questions about the rules of Scrabble, especially around how many tiles there are for each letter, and whether you can spell out certain words with and without blank tiles. The answers are internally inconsistent, don't correspond with actual Scrabble rules, and don't even correspond with the source material that GPT-3 pointed me to when I asked it how it "knew".
Show threadbenjamin melançonDec 5, 2022
@markproxy @emilymbender so there is not any setup for direct citation of sources in these models but you can ask it what it's sources are… but it's possibly making up things there, too?!
Show threadMark McClellandDec 5, 2022
@mlncn @emilymbender It's not fabricating sources entirely, but the content of the sources I've seen often doesn't match GPT-3's so-called knowledge.
Show threadbenjamin melançonDec 5, 2022
@markproxy @emilymbender Thanks! This more clearly asked the question in my head, and the answer—the model is not supposed to retain the source material—is elucidating and worrying https://fosstodon.org/@miklo/109462959890443389
💙💛:~/eu/pl/priv$:idle: (@[email protected])

@[email protected] Is this #ChatGPT able to return, as part of its reply, links to the sources on which it based its reply ?

Fosstodon
Show threadMark McClellandDec 5, 2022

@emilymbender And in related news... I'm encouraged to see Stack Overflow attempting to prevent answers fabricated by NLP models.

https://www.theinsaneapp.com/2022/12/chatgpt-ai-generated-answers-banned-on-stack-overflow.html

ChatGPT AI Generated Answers Banned On Stack Overflow

StackOverFlow have temporarily blocked users from posting responses / codes / answers created by the OpenAI's chatbot ChatGPT.

Insane
Show threadProf. Emily M. Bender(she/her)Dec 5, 2022
@markproxy I am well aware. I wasn't asking for experience reports with GPT-3 or any other LLM with that post.
Show threadMark McClellandDec 5, 2022
@emilymbender I meant merely to support your point with an illustrative example from my own experience, in the context of conversation around an important topic. I can see how it might have come across differently; sorry for that!
Show threadProf. Emily M. Bender(she/her)Dec 5, 2022
@markproxy Thank you for this. In the future, please consider whether such support was requested before offering it as well as how you present it if you choose to do so.
Show threadEric KansaDec 5, 2022

@emilymbender

There's a new NEH granting program that seems ideally suited for exploring this issue at least in the domain of the humanities. So interesting!!

https://www.neh.gov/program/dangers-and-opportunities-technology-perspectives-humanities

Dangers and Opportunities of Technology: Perspectives from the Humanities

Supports research examining the relationship between technology and society.

The National Endowment for the Humanities
Show thread💙💛:~/eu/pl/priv$Dec 5, 2022
@emilymbender Is this #ChatGPT able to return, as part of its reply, links to the sources on which it based its reply ?
Show threadRichDec 5, 2022
@miklo @emilymbender I don't believe so, as the final model does not actually contain its source information.
Show threadProf. Emily M. Bender(she/her)Dec 5, 2022
@rmbles @miklo Indeed not. This is part of why it's such a terrible idea to use a language model as an information access system. It cuts off any ability for the person using the system to contextualize the source of the information (when it even is legit information, as happens only by chance with these things).
Show threadMark McClellandDec 5, 2022
@miklo @emilymbender In my limited experiments, I've been able to ask for and receive links to sources, and the linked content has always been relevant but often contradictory. I have asked for additional sources, thinking GPT-3 might have "averaged" multiple sources, but have not typically gotten anything to support my hypothesis; instead, I get the same link as the first time, or a link to another contradictory source.
Show threadCian O'ConnorDec 6, 2022

@emilymbender
I continue to be baffled that AI researchers think you can separate out a tool that emerged from intelligent, situated, and social, behaviors (language) as a discrete thing. Have none of these people read Wittgenstein? or Harold Garfinkel for that matter.

We knew this was a stupid idea in the 80s. So why are people still doing this?

Apologies for the rant...

Show threadSw0rdDooMDec 9, 2022
@emilymbender try asking in it for sources, its will make links up
Show threadCsepp 🌢Dec 10, 2022
@emilymbender The section about arithmetic reminds me of something my AI course's teacher said, that the hype pendulum might soon swing closer to expert systems or proof search.