Prof. Emily M. Bender(she/her)Nov 17, 2022

Moving a thread from Twitter over here, which starts with my reacting to Michael Black's thread on #Galatica where he tests it and concludes it's dangerous, saying:

"Why dangerous? Galactica generates text that's grammatical and feels real. This text will slip into real scientific submissions. It will be realistic but wrong or biased. It will be hard to detect. It will influence how people think. (5/9)"

https://twitter.com/Michael_J_Black/status/1593133739538022400

>>

Michael Black on Twitter

“Why dangerous? Galactica generates text that's grammatical and feels real. This text will slip into real scientific submissions. It will be realistic but wrong or biased. It will be hard to detect. It will influence how people think. (5/9)”

Twitter
Show threadProf. Emily M. Bender(she/her)Nov 17, 2022

"Generates text that's grammatical and feels real": We sounded the alarm about this in the Stochastic Parrots paper. Ordinarily, I don't expect the average AI researcher to have read my papers, but that paper (thanks to Google) can hardly be missed.

So what gives?

>>

#StochasticParrots #Galatica #AIHype

Show threadProf. Emily M. Bender(she/her)Nov 17, 2022

I looked back at Section 6 of #StochasticParrots, which begins with this paragraph.

And then we look deeply into the risks and harms that follow when that seemingly coherent text reproduces the language of systems of oppression.

>>

Show threadProf. Emily M. Bender(she/her)Nov 17, 2022

So, this looks like a case where that wasn't alarming enough. It only becomes *alarming* for some when what's threatened is ~%~science~%~.

Or maybe folks read our paper and didn't really believe us but had to wait until they could "see it with their own eyes" = see the effect from a) a live demo that b) impinges on something where they do have skin in the game.

>>

Show threadProf Austen RainerNov 18, 2022
@emilymbender, When I reflect on my interaction with Galactica, for me there’s something about the primacy of direct experience of interacting with it, and of whatever ‘expertise’ (I write that humbly) I have to personally interpret and evaluate it’s output. So as well as, or aside from, whatever threat Galactica may raise, interacting directly with Galactica was a more, hmm, ‘meaningful’ experience than ‘only’ reading of it.
Show threadProf. Emily M. Bender(she/her)Nov 18, 2022

@austenrainer
Resist the urge to be impressed.
https://medium.com/@emilymenonbender/on-nyt-magazine-on-ai-resist-the-urge-to-be-impressed-3d92fd9a0edd

And if you are impressed, please don't go around rhapsodizing about it. That only makes the #AIhype situation worse.

Show threadProf Austen Rainer
@emilymbender Indeed. And I wasn’t impressed, at all, hence ‘meaningful’ in quotes. I meant that it’s one kind of knowledge to read about something like Galactica or GPT-3, it’s another kind of knowledge to interact directly with it and to pay attention to one’s own reactions to it.
Nov 18, 2022 at 8:15pmWeb