Mastodawn

Carl T. Bergstrom May 25, 2023

One of the decisive moments in my understanding of #LLMs and their limitations was when, last autumn, @emilymbender walked me through her Thai Library thought experiment.

She's now written it up as a Medium post, and you can read it here. The value comes from really pondering the question she poses, so take the time to think about it. What would YOU do in the situation she outlines?

https://medium.com/@emilymenonbender/thought-experiment-in-the-national-library-of-thailand-f2bf761a8a83

@ct_bergstrom @emilymbender I do feel like there are some exceptions to this, for example text that is self-referential. The model is able to capture for instance that an instruction is something that manipulates how another section (a person) behaves in terms of language form, though of course it still doesn't get that it is an intentional act of communication, only that it does something.

Some other areas where I think the function is captured somewhat are basic code simulation and math.

@ct_bergstrom @emilymbender I do think this can seep into other areas slightly, but it is not the way the model deals with most data, it doesn't have an internal representation of the world when answering questions even if it's technically capable of bringing one up sometimes, it at most has a table of fact statements, relationships and knowledge of how syntax and text in general is structured so it can string them together.

@ct_bergstrom @emilymbender Still, it's quite fascinating to see how it manages to solve these problems, because it's so different from a human in so many ways, it makes completely different errors, has different limitations.

Same reason I find AI art interesting (when it's not trying to replace artists at least), it makes mistakes, it's weird, it's uncanny.