Prof. Emily M. Bender(she/her)Nov 16, 2022

Facebook (sorry: Meta) AI: Check out our "AI" that lets you access all of humanity's knowledge.

Also Facebook AI: Be careful though, it just makes shit up.

This isn't even "they were so busy asking if they could" --- but rather they failed to spend 5 minutes asking if they could.

#AL #ML #MathyMath #Bullshit #NLP #NLProc #AIhype

Show threadProf. Emily M. Bender(she/her)Nov 16, 2022

Using a large language model as a search engine was a bad idea when it was proposed by a search company. It's still a bad idea now that it's being proposed by a social media company. Fortunately, Chirag Shah and I already wrote the paper laying out all the ways in which this is a bad idea.

https://dl.acm.org/doi/10.1145/3498366.3505816

#AL #ML #MathyMath #Bullshit #NLP #NLProc #AIhype

Situating Search | Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

ACM Conferences
Show threadProf. Emily M. Bender(she/her)

In the popular press/general public-facing Q&A about our paper:

https://www.technologyreview.com/2022/03/29/1048439/chatbots-replace-search-engine-terrible-idea/

https://www.washington.edu/news/2022/03/14/qa-preserving-context-and-user-intent-in-the-future-of-web-search/

#AL #ML #MathyMath #Bullshit #NLP #NLProc #AIhype

Chatbots could one day replace search engines. Here’s why that’s a terrible idea.

Language models are mindless mimics that do not understand what they are saying—so why do we pretend they’re experts?

MIT Technology Review
Nov 16, 2022 at 9:06pmWeb
Show threadProf. Emily M. Bender(she/her)Nov 16, 2022

And let's reflect for a moment on how they phrased their disclaimer, shall we? "Hallucinate" is a terrible word choice here, suggesting as it does that the language model has *experiences* and *perceives things*. (And on top of that, it's making light of a symptom of serious mental illness.)

Likewise "LLMs are often Confident". No, they're not. That would require subjective emotion.

#AL #ML #MathyMath #AIhype #NLP #NLProc

Show threadProf. Emily M. Bender(she/her)Nov 16, 2022

I went digging in the paper to see if they cite #StochasticParrots or Bender & Koller 2020 or Shah & Bender 2022. That is, did they read about why this is misguided and just press ahead anyway? Apparently not.

#AL #ML #MathyMath #AIhype #NLP #NLProc

Show threadProf. Emily M. Bender(she/her)Nov 16, 2022

They do cite Blodgett et al 2020 (fabulous paper!)

https://aclanthology.org/2020.acl-main.485/

But in the strangest possible way. Are they reflecting on the possible harms their technology might engender? No, of course not. They're striving for TRUTH! And thus worried about "bias".

#AL #ML #MathyMath #AIhype #NLP #NLProc

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

ACL Anthology
Show threadProf. Emily M. Bender(she/her)Nov 16, 2022

Narrator voice: LMs have no access to "truth", or any kind of "information" beyond information about the distribution of word forms in their training data. And yet, here we are. Again. /fin (for now)

#AL #ML #MathyMath #AIhype #NLP #NLProc

Show threadLeon Derczynski 🏔️✍🏻🌲☀️Nov 16, 2022
@emilymbender surely you're not suggesting that calling a dataset the "ground truth" could be problematic :o
Show threadAdam CollinsNov 17, 2022
@emilymbender There are a lot of SWEs and DS engs out there who truly believe that, with enough data, the models will magically know things and be flawless.
Show threadLeon Derczynski 🏔️✍🏻🌲☀️Nov 16, 2022
@emilymbender but wait
1. social media's variation of style is a more true representation of language
2. scientific style doesn't mean higher veracity, that's an error of ethos
3. one paper showing low transfer between two styles is indicative of ..nothing.. wrt. veracity
4. science is often wrong, this is intrinsic and on balance A Good Thing
5. truth isn't quite so objective or permanent
6. getting a real "science is the only form of scholarship" vibe here, bad assumption. a False one, even
Show threadAaronNov 21, 2022
@emilymbender I definitely get the vibe that they trained this model with all these aspirations of it being awesome and helpful and useful and safe because of their choice of data source, then saw that it met none of those goals and just started making excuses. Like they decided to double down because they were heavily invested in it and just wanted it to work *so bad*.
Show threadAaronNov 21, 2022
@emilymbender "We spent so much time and money on it! It *has* to work!"
Show threadJed BrownNov 16, 2022
@emilymbender I checked this first and was disappointed. Then I tried some queries expecting to get CoPilot-style plagiarism, but almost every query returned misleading, confidently wrong, and self-contradictory babble, and I came away thinking it's perhaps not so harmful because the most use it can aspire to is as a punchline for jokes.
Show threadClive ThompsonNov 16, 2022

@emilymbender

omg I laughed so hard when I got to the "hallucinate" line

It's a terrible bit of anthropomorphization, but *damn* it's funny to see the CREATOR saying that

If I were helping them copyedit it I might have suggested the line "Language Models Can Pull Stuff Out Of Their Hats That Makes, Like, *Zero* Sense, So YMMV k thx bai"