One of the decisive moments in my understanding of #LLMs and their limitations was when, last autumn, @emilymbender walked me through her Thai Library thought experiment.
She's now written it up as a Medium post, and you can read it here. The value comes from really pondering the question she poses, so take the time to think about it. What would YOU do in the situation she outlines?
@ct_bergstrom I find the argument frustrating as it focuses on overwhelming the reader with a big, hard task, not finding a smaller case with the same features.
If I were given a huge library of pure maths texts in an unknown language, I have no idea how I'd extract meaning from it. Yet given some unexplained maths-y puzzles, I could get the pattern, and I reckon an ML algorithm will also get the meaning in some sense as much as I would, despite the lack of other context...
@militant_dilettante I'm not quite sure how to interpret "correlation vs. causation" since it might depend on what kind of formal lang we're talking about (vs. e.g. equivalence and implication), but...
I think that if you're reading something put together by a human with intent, that can still come through even in formal language - proofs, examples etc. are carefully chosen. Proofs run in a direction. Questions appear before answers. I think that intent can be pulled out?
Carl T. Bergstrom
Simon Frankau
militant_dilettante