Mastodawn

Ben May 25, 2023

@ct_bergstrom @emilymbender Interesting thought experiment very similar to Searle’s Chinese Room. As far as I understand that without studying it deeply, aren’t the “unlimited time”, our limitations wrt memory, and our own expectation of frustration doing too much work here?

Ben May 25, 2023

@ct_bergstrom @emilymbender I think I get where you, Emily, are going with this, but even the question of “understanding” in a non-functional way is anthropomorphizing LLMs. It only makes sense when we try in a second step to map our knowledge about Thai signs to our existing memory of the world. But this question actually never arises for LLMs. So the only way they can understand the world is by reacting in a “right” way to inputs. But aren’t we, too?

Frank Bennett 🇯🇵May 25, 2023

@b3n @ct_bergstrom @emilymbender That goes back to the classic problem of rhetoric: we can't access other minds directly via text (or other modes), so must rely on hints embedded in language. That opens the ever-present possibility of deception, which we come to understand from the arc of discourse and action of an individual speaker, which reveals underlying intention. That doesn't happen with LLMs. Quite the opposite, it becomes a random walk.

enmodo ⚛️ 🧬 🇺🇦 🍉May 26, 2023

@b3n @ct_bergstrom @emilymbender definitely similarities - sufficient that it was my first thought. All though this is a different angle of attack - how would you learn vs had the system learned. To some extent the Chat engines are being given external input by users as additional training but it seems to me that pales in comparison to their base model.

I'm unsure if the grand plan is to get so many people using them that their feedback becomes significant.

enmodo ⚛️ 🧬 🇺🇦 🍉May 26, 2023

@b3n @ct_bergstrom @emilymbender the real problem with that is an insufficient number of people can tell good output from hallucinogenic lies. So at best we'd be training the systems to be better liars or at least more agreeable ones.

It boggles my mind that no other than Google failed to launch their Bard system with a built in bullshit detector. They have a huge knowledge graph that could easily be analysing output for BS.

enmodo ⚛️ 🧬 🇺🇦 🍉May 26, 2023

@b3n @ct_bergstrom @emilymbender another issue seems to be that even if we could convince ourselves that these models "understand" then what use is understanding without empathy or consequences? Congratulations you just built yourself a chat psychopath.

Jon Sullivan May 25, 2023

@ct_bergstrom @emilymbender That's very good. One difference going forward between #LLMs and Emily's "stuck in a Thai library with only words", is that Bing-style ChatGPT gets to make up answers and see how real people respond. If you were smart, couldn't speak Thai, were stuck in a Thai library, *and* you could try out sentences on Thai people to see their responses, could you gradually build up some concepts of what words mean? Or, do you still need some external context to apply meaning?

@ct_bergstrom @emilymbender

Andreas May 25, 2023

@joncounts
Interesting thought experiment - but LLMs don't learn that way. They use *labeled* data:
https://labelyourdata.com/articles/data-annotation-for-training-chatgpt
In general, when a neural network is trained, you need data to process where you already know the correct "meaning", this data is called labeled data.
Labeled data would mean some books with pictures, or even instructional books.

Also, afaik LMM don't learn from peoples answers. You cannot re-configure them "on the fly".

ChatGPT and Data Annotation

Learn about the importance of data annotation for creating AI chatbots and training smart language models like ChatGPT ✅

Jon Sullivan May 25, 2023

@andreasgoebel @ct_bergstrom @emilymbender @benjamingeer Thanks for those responses. That makes sense.

Whenever machines *can* learn this way, by adjusting to human responses, I still expect it's going to be quite a challenge to figure out the meaning behind words. It feels like we're still at the very start of a long road to get to real artificial intelligence.

Andreas May 25, 2023

@joncounts
I agree that (at the moment) AI doesn't really understand language, the general conclusion is right, . the thought experiment is (sadly) not completely acurate.

But yes, we're only at the beginning.
@ct_bergstrom @emilymbender @benjamingeer

Eric Hellman May 25, 2023

@joncounts @ct_bergstrom @emilymbender That's what I was thinking. If a LLM is allowed to train on the queries it gets, it should be able to learn how to manipulate them. The "reality" of an LLM is the feedback it can get from the answers it sends out.

Darrin West May 25, 2023

@emilymbender @gluejar @ct_bergstrom @joncounts Get feedback… From humans, on the internet? That’s not going to turn out well.

chris martens May 25, 2023

@joncounts @ct_bergstrom @emilymbender is the sense in which those tools “see” how they respond similar to the sense in which you imagine “seeing” their responses in the thought experiment? in other words, do you see their bodies or just their words?

Jon Sullivan May 25, 2023

@chrisamaphone @ct_bergstrom @emilymbender To be consistent with Emily's thought experiment, I'd think you'd only get the responses in words, and not be able to see who is responding. Maybe you're stuck in the Thai library and could pass pieces of paper out through a slot and get written replies back.

I still think it would be terrifically difficult to build up meaning behind words this way. Blind babies do it, but maybe that's by using other senses to explore objects associated with words.

chris martens May 25, 2023

@joncounts @ct_bergstrom @emilymbender indeed it is. sensory information is crucial to how we experience meaning

@ct_bergstrom @emilymbender I do feel like there are some exceptions to this, for example text that is self-referential. The model is able to capture for instance that an instruction is something that manipulates how another section (a person) behaves in terms of language form, though of course it still doesn't get that it is an intentional act of communication, only that it does something.

Some other areas where I think the function is captured somewhat are basic code simulation and math.

@ct_bergstrom @emilymbender I do think this can seep into other areas slightly, but it is not the way the model deals with most data, it doesn't have an internal representation of the world when answering questions even if it's technically capable of bringing one up sometimes, it at most has a table of fact statements, relationships and knowledge of how syntax and text in general is structured so it can string them together.

@ct_bergstrom @emilymbender Still, it's quite fascinating to see how it manages to solve these problems, because it's so different from a human in so many ways, it makes completely different errors, has different limitations.

Same reason I find AI art interesting (when it's not trying to replace artists at least), it makes mistakes, it's weird, it's uncanny.

@ct_bergstrom @emilymbender

Padjo May 25, 2023

That sounds good in the abstract but I don’t think it really reflects reality. ChatGPT really does appear to be able to reason in surprising ways. Good write up with some example output here https://medium.com/@fergal.reid/why-are-so-many-giants-of-ai-getting-gpts-so-badly-wrong-f8dadfac4f61

@Padjo @ct_bergstrom @emilymbender

rastilin May 25, 2023

This is a brilliant article, and it looks like they've actually used ChatGPT. I get the sense from a lot of the AI articles that the people writing them haven't played around with ChatGPT itself beyond very surface level interactions or they've only used one of the lower powered versions.

⁂ Fish Id Wardrobe May 25, 2023

@Padjo @ct_bergstrom @emilymbender "appear to be able to" – correct. It can't reason. But it does appear to reason.

Le Néandertal se sent las, las May 25, 2023

@ct_bergstrom @emilymbender I fail to see the difference with the Chinese room argument... On which I tend to agree, at the present time, with Daniel Dennett (evocated in the WP article). https://en.m.wikipedia.org/wiki/Chinese_room

Chinese room - Wikipedia

Pete Carlton May 26, 2023

@HydrePrever @ct_bergstrom @emilymbender Yes, it's a Chinese room argument with the "dials set differently" as Dennett has put it. I agree with Dennett's refutation of Searle's argument, but this setting of the dials has made me think a lot about its application to LLMs and I'm still pondering it; I'm not sure what to conclude yet. So, a very good essay even if I may end up disagreeing.

Martin May 25, 2023

@ct_bergstrom @emilymbender The frustrating thing about the topic is that when one has understood the basic workings of #LLMs everyone is a pundit. Because at this point everything can be just hand-waving and speculating there is no basis for gaining any scientific knowledge IMO.
The only aspect that continues to be very tangible is the danger of mis- and disinformation arising from these systems and the debate around what they "are" or can "be" just distracts from this.

Carl T. Bergstrom May 25, 2023

@micron @emilymbender yes, how intensely exasperating that one of the leading experts in the field spend her time writing for the public on subjects that they want to read about. No excuse for it, really.

Martin May 25, 2023

@ct_bergstrom @emilymbender No, I read it.
From your writing I think I know that you also see the importance of the mis- disinformation aspect.
My concern is that the focus on the what it "is or is not" lets the public easily dismiss the whole debate around #LLMs as "academic" or on the other end sensationalise it.

Matunos May 25, 2023

@micron @ct_bergstrom @emilymbender I also feel that we risk falling down a rabbit hole of pondering questions like "is ChatGPT sentient?", "does it truly understand the prompts you give it, or its own responses?", etc. and it's much more imperative to understand their outward-facing and objective capabilities and limitations.

Frank Bennett 🇯🇵May 25, 2023

@matunos @micron @ct_bergstrom @emilymbender One could see the task of scuppering the notion that these systems are capable of independent intention as an essential first step in that direction.

@fgbjr @micron @ct_bergstrom @emilymbender rather than the notion, I would scupper the question as unfalsifiable, and instead ask "if the system were capable of independent intention, what do you think that would mean it could do?" and then test if a system could do those things.

Frank Bennett 🇯🇵May 26, 2023

@matunos @micron @ct_bergstrom @emilymbender I think language is too imprecise for that kind of testing, and I'm certain that if it were attempted, the testing regimen would be gameified into disutility.

@fgbjr @micron @ct_bergstrom @emilymbender how precise does the language need to be? there's not going to be a single definitive test suite; rather it will be a series of "yes but can it…?" as challenges are developed and AI systems meet them, until they either reach their limits or are indistinguishable from humans in performance

@fgbjr @micron @ct_bergstrom @emilymbender we can endlessly pose questions like "but does it *really* understand?" (something you can similarly pose to spouses— also unanswerable), but without objective criteria it just boils down to vibes. Ultimately what matters is the capabilities we can observe.

Frank Bennett 🇯🇵May 26, 2023

@matunos @micron @ct_bergstrom @emilymbender The question I would (and do) pose is, "can one of these systems exhibit underlying intention?"

And you will tell me that that is not a falsifiable assertion.

I think we've reached the limit of our debate over this.

@fgbjr @micron @ct_bergstrom @emilymbender do you disagree that it's not falsifiable? what do you mean by "underlying intention"? how would you know if a human subject exhibits underlying intention?

Frank Bennett 🇯🇵May 26, 2023

@matunos @micron @ct_bergstrom @emilymbender I thought I stated it clearly in the last post, but I will try again: There is no point in continuing this discussion further, Sydney.

@fgbjr who is Sydney?

@ct_bergstrom I find the argument frustrating as it focuses on overwhelming the reader with a big, hard task, not finding a smaller case with the same features.

If I were given a huge library of pure maths texts in an unknown language, I have no idea how I'd extract meaning from it. Yet given some unexplained maths-y puzzles, I could get the pattern, and I reckon an ML algorithm will also get the meaning in some sense as much as I would, despite the lack of other context...

@ct_bergstrom Of course, this is not the point. I think the point is "Can a system derive meaningful understanding of something fundamentally experiential from a corpus lacking good representation of that experience?"

I think the best (still weak) small-scale analogy I can think of is "Can a blind person understand colour?"

They'll never experience colour directly, but can certainly learn optics, colour theory etc. in a way that may be useful in some domains but not others...

@ct_bergstrom And I think this allows us to examine the true constraints of these models. While a GPT can absorb enough text saying "dog" to build something that looks suspiciously like a model of a dog without meeting one, it has no real representation of, say, space, and must rely on "faking" through manipulating language instead.

@ct_bergstrom It's possibly also interesting in terms of how much true understanding a human has of experiential things, and what that means. If you have an addiction expert who understands complex biochemistry, psychology etc. etc. but has never actually been addicted to something, are they a fake without true understanding who should not be listened to?

@ct_bergstrom And an alternative approach is to try "continuously deforming" the Thai library to experiential learning, and decide where the important changes lie. What if the library were given to you in a curated order? If it were a video/sensory stream, rather than text? If it were interactive? At that point, is it experiential learning?

@ct_bergstrom Ah, bummer, not a good analogy to go with - https://dair-community.social/@emilymbender/110429533429263953 . I wonder how much can be rescued with a non-problematic analogy?

Emily M. Bender (she/her) (@[email protected])

@[email protected] @[email protected] No, you really don't want to go there: https://medium.com/@emilymenonbender/no-llms-arent-like-people-with-disabilities-and-it-s-problematic-to-argue-that-they-are-a2ac0df0e435

Distributed AI Research Community

militant_dilettante May 26, 2023

@sgf @ct_bergstrom how would you distinguish between patterns arising from correlation and patterns arising from causation then?

Simon Frankau May 26, 2023

@militant_dilettante I'm not quite sure how to interpret "correlation vs. causation" since it might depend on what kind of formal lang we're talking about (vs. e.g. equivalence and implication), but...

I think that if you're reading something put together by a human with intent, that can still come through even in formal language - proofs, examples etc. are carefully chosen. Proofs run in a direction. Questions appear before answers. I think that intent can be pulled out?

militant_dilettante May 29, 2023

@sgf thank you for this comment. It shows me, where I must be more clear and precise in definitions, and prompts me to sharpen my conclusions. I will try to respond more coherently later.

ArchaeoIain May 25, 2023

@ct_bergstrom @emilymbender Steven Harnad's equivalent thought experiment was about whether it would be possible to learn Chinese, or any other non-alphabetic language, using only a dictionary of that language. The Chinese-Chinese dictionary problem

@ct_bergstrom @emilymbender

Matilda May 25, 2023

& of course this *exact* scenario has been shown - Linear A & other scripts are still un-deciphered for this VERY reason (Linear B only figured out because I believe there is overlap with early greek?)

Jörg Knappen May 25, 2023

@ct_bergstrom @emilymbender The Thai library task is maybe impossible for a single human (the task of reading everything is already impossible during a typical lifespan), but it is probably possible for a machine. For some results on supervised machine translation see https://linguistics.stackexchange.com/a/34102/9781

Could the Ancient Egyptian hieroglyphs have been deciphered without the Rosetta Stone with modern tech?

The Rosetta Stone was one of the most important documents in the history of linguistics. Discovered around 1800, it allowed Ancient Egyptian to be deciphered. Let's say that the stone didn't exist,...

Linguistics Stack Exchange

militant_dilettante May 26, 2023

@jk @ct_bergstrom @emilymbender isn't the Thai Library an example of unsupervised learning?

Lisa DeBruine 🏳️‍🌈May 25, 2023

@ct_bergstrom @emilymbender Oh, this is such a good explanation!

Mark vW May 25, 2023

@ct_bergstrom @emilymbender I think to start down the journey, you'd need multiple (non-English) libraries. But, this is hard on the brain!

Philip Dutré May 25, 2023

@ct_bergstrom @emilymbender The last paragraph relates to the old question "what's intelligence?". I then always think of Edsger Dijkstra's quote: "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."

Dave Ackley May 25, 2023

@ct_bergstrom @emilymbender I'd watch out for a headsplitting mother of all AHAs, once grounded meanings for just a few terms, inevitably, leak through the semantic cordon sanitaire.

Ryan Germann

May 25, 2023

@ct_bergstrom @emilymbender everyone should read this to put the ChatGPT hype into perspective.

sorrykb May 25, 2023

@ct_bergstrom @emilymbender Look for books that have an index. That would at least get some relational words.

sorrykb May 25, 2023

@ct_bergstrom @emilymbender Or go to the theater in the library and watch a bunch of movies. :-)

⁂ Fish Id Wardrobe May 25, 2023

@sorrykb @ct_bergstrom @emilymbender But you can't read the words in the index either? How would that help?

sorrykb May 26, 2023

@fishidwardrobe @ct_bergstrom @emilymbender Knowing the typical structure of an index, you can at least get a sense of words that have some connection to each other. Maybe.

marymessall May 25, 2023

@ct_bergstrom @emilymbender That was helpful - thanks. And it's an easy question to ask others to ask others to make my own conversations about AI more productive.