Apple did the research; LLMs cannot do formal reasoning. Results change by as much as 10% if something as basic as the names change.

https://garymarcus.substack.com/p/llms-dont-do-formal-reasoning-and

LLMs don’t do formal reasoning - and that is a HUGE problem

Important new study from Apple

Marcus on AI
@ShadowJonathan not to sound antiintellectual, but isn't it kinda obvious that a *text* generator, no matter how complex, can't do abstract reasoning?
@halva @ShadowJonathan yeah, I appreciate the demonstrations, but this feels a little like, "New study confirms bicycles cannot fly."
@graue @halva @ShadowJonathan But imagine of the world's most awful people were pouring hundreds of billions of dollars into telling everyone bicycles can fly. Then you would need to spend resources refuting that. đŸ€Ź

@dalias @graue @halva @ShadowJonathan You’d think that people who own a bicycle can just check


On a tangentially related note, flying bicycles are invented by future humanity ‘The Dark Forest’ personal flying vehicles in the form of helicopter backpacks. They’re “bicycles” in the sense that they’re two counter-rotating coaxially-mounted propellers. That’s actually not a bad idea. If only we poured billions of dollars into making that work.

@enoch_exe_inc @dalias @graue @halva

> You’d think that people who own a bicycle can just check


does the emperor have no clothes? would people call him out on it?

@ShadowJonathan @enoch_exe_inc @dalias @graue @halva I would. Whether from a government or a corporation, I will not reject the evidence in front of my eyes.

@ShadowJonathan @enoch_exe_inc @dalias @graue @halva

I'm an avid cyclist. I'll be out riding most of today and tomorrow.

Technically ...
Some bicycles do fly:

https://www.youtube.com/watch?app=desktop&v=v9KJwOZ3frk

And many "mountain bikers" are known for getting a lot of "air"

https://www.youtube.com/watch?v=6s6-O054SXg&ab_channel=BrendanFairclough

But yes, generally, bicycles do *not* fly.

WITH NO ENGINE!! University Students Create a Human-Powered Aircraft

YouTube