Mastodawn

Taylor Webb Jan 11, 2023

Melanie Mitchell Jan 11, 2023

New blog post: @taylorwwebb, co-author of fascinating paper on GPT-3's analogy-making ability, responds to my perspective:
https://aiguide.substack.com/p/response-to-on-analogy-making-in

Response to "On Analogy-Making in Large Language Models"

My first post on this blog was a perspective on a very interesting paper by Webb et al., “Emergent Analogical Reasoning in Large Language Models”. Taylor Webb, the first author of that paper, responded by email to my perspective, and he gave me permission to post his response on this blog. Here is Taylor’s thoughtful response, which is well worth reading. I will make a few comments at the end.

AI: A Guide for Thinking Humans

Show thread

Taylor Webb Jan 6, 2023

@achterbrain @adel nothing more for now, will probably post an update to the paper soon as we’ve also run a bunch of additional tests, will post here when that’s ready.

Show thread

Taylor Webb Jan 5, 2023

@achterbrain @adel by ‘correspondence finding’, I mean the process of determining which elements go together to form a sub-problem. Our task doesn’t require object segmentation, but it arguably does still require correspondence finding.

Show thread

Taylor Webb Jan 5, 2023

@achterbrain @adel yes we found that human error rates on our digit matrix problems were extremely similar to error rates on the standard visual RPM problems (figure 4 of the paper). I like this study from Duncan et al., but I think it’s not entirely conclusive about the key source of difficulty in RPM. In particular, the separated problems remove two sources of difficulty - object segmentation and correspondence finding. I believe the latter is more important.

Taylor Webb Jan 5, 2023

Jascha Achterberg Jan 5, 2023

What are your favourite projects investigating if / how different large #foundational #models do #logical #reasoning ? Or how their "next token prediction mechanism" emulates reasoning.

Still trying to make my mind up whether the internal dynamics of these models are worth investigating.

Very curious to hear people's thoughts!

#NLProc #LLM #genAI #AI #ML #logic @cogsci @cognition @neuroscience #neuroscience #cognition

Taylor Webb Jan 4, 2023

Matt Slocombe Jan 3, 2023

📢 📢 Delighted to announce that the Analogical Minds Seminar is returning next week with a new series of talks on analogical processes in cognition and learning.

Over the spring term, we’ll be covering topics such as reasoning, language, development, conceptual blending, mathematics, design, and science education. Click on the image below for the full spring programme.

Registration and further info: http://www.analogicalminds.com

All welcome!

@cognition #psychology #development #education #AI

Analogy List - Analogical Minds Seminar

Analogical Minds Seminar Analogical Minds is a weekly online seminar dedicated to exploring the role of analogy, metaphor, and relational processes in cognition and learning. Our speakers discuss research from a wide range of disciplinary perspectives, including cognitive and developmental

Taylor Webb Jan 2, 2023

David Bau Dec 21, 2022

I want to show the NSF there would be broad support+utility for a "National Deep Inference" service for >100b LLMs.

If your research would be enabled by an inference service on open LLMs w API access+overrides to internal activations, params, gradients: please boost this thread!

(I'm also gathering feedback on twitter - more details here:)

https://twitter.com/davidbau/status/1605609105824964611

David Bau on Twitter

“I want to show the NSF there would be broad support+utility for a "National Deep Inference" service for >100b LLMs. If your research would be enabled by an inference service on open LLMs w API access+overrides to internal activations, params, gradients: Please Like this thread!”

Twitter

Show thread

Taylor Webb Dec 21, 2022

Overall, we were shocked that GPT-3 performs so well on these tasks. The question now of course is whether it's solving them in anything like the way that humans do. Does GPT-3 implement, in an emergent way, any of the features posited by cognitive theories, e.g. relational representations, variable-binding, analogical mapping, etc., or has it discovered a completely novel way of performing analogical reasoning? (or are these cognitive theories wrong?) Lots to investigate.

Show thread

Taylor Webb Dec 21, 2022

Nevertheless, the overall conclusion is that GPT-3 does appear to possess the core features that we associate with analogical reasoning -- the ability to identify complex relational patterns, zero-shot, in novel problems.

Show thread

Taylor Webb Dec 21, 2022

Finally, we also tested GPT-3 on letter string analogies. @melaniemitchell previously found that GPT-3 performed very poorly on these problems:

https://medium.com/@melaniemitchell.me/can-gpt-3-make-analogies-16436605c446

but it seems that the newest iteration of GPT-3 performs much better.