Spring Jo π₯ 
πApple did the research; LLMs cannot do formal reasoning. Results change by as much as 10% if something as basic as the names change.
https://garymarcus.substack.com/p/llms-dont-do-formal-reasoning-and
Not too surprising, but a big weight like Apple standing behind this should shift sentiment more and more
Lulu π ΧΧΧΧThis is not surprising at all and I don't understand why anyone had to waste time and resources on demonstrating a self-evident fact that was known before the research even started.
Levi@lulu @ShadowJonathan yeah I don't know anything about LLMs really but I thought this was a fundamental. There's no reasoning, they don't understand meaning, they just make sentences by jumbling words together based on probabilities.
felizYes, the problems posed to the #LLMs in this study are mathematical in nature or logic problems β why are systems that are trained to produce text expected to produce any meaningful results here?
@ShadowJonathan @feliz @lulu are they though? I don't know enough about LLMs and their implementations to refute you but it seems as though anyone who has ever entered a prompt and looked at the response could understand that there is no reasoning.