Apple did the research; LLMs cannot do formal reasoning. Results change by as much as 10% if something as basic as the names change.

https://garymarcus.substack.com/p/llms-dont-do-formal-reasoning-and

LLMs donโ€™t do formal reasoning - and that is a HUGE problem

Important new study from Apple

Marcus on AI
Not too surprising, but a big weight like Apple standing behind this should shift sentiment more and more
@ShadowJonathan we knew this just based on how they arrive at answers. how can you reason if you don't understand the words you're using?

@elexia @ShadowJonathan Because OpenAI and co. marketed them as it. And many, many people swallowed it raw.

The technological realities of LLM have always been secondary in this AI rush, it's all been about marketing, and Oh boy has it worked.

It will take a long time and a lot of demonstrations, including the simplest ones, for the world to understand their mistakes and pull out of this impasse. Or the next revolutionary, distributive technology in the Silicon Valley cycle will hijack all the fund.

Now, it's a matter of proving that the ad was misleading, and that is harder than creating a pretty beautiful lie.