brianpeiris3d ago

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

https://lemmy.ca/post/62420870

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%. - Lemmy.ca

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks. > ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels. > > Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations. You can try the tasks yourself here: https://arcprize.org/arc-agi/3 [https://arcprize.org/arc-agi/3] Here is the current leaderboard for ARC-AGI 3, using state of the art models - OpenAI GPT-5.4 High - 0.3% success rate at $5.2K - Google Gemini 3.1 Pro - 0.2% success rate at $2.2K - Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K - xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K. ARC-AGI 3 Leaderboard [https://lemmy.ca/pictrs/image/c7521941-7eac-46f4-98de-876bcf99220c.png] (Logarithmic cost on the horizontal axis) https://arcprize.org/leaderboard [https://arcprize.org/leaderboard]

Show threadfox2263

I can’t see AI actually being intelligent until they no longer need to send a built up prompt of guides and skills and the chat history on every submission.

It’s no different from Alexa 15 years ago with skills. Just a better protocol and interface and ability to parse the current user prompt.

In my opinion of course.

Mar 28 at 7:09amWeb
Show threadPhoenixDog1d ago

Right? I have a Google Home Mini in our kitchen and if we ask it a question it just pulls a source from a website and tells us. That’s it. Nothing intelligent about it.

AI now is no different. It’s just pulling more complex wording from more (albeit illegally) sources to give a (albeit sometimes incorrect) better description of the question asked.

AI is just as stupid as Alexa is/was 15 years ago. It just has more information to pull from and still fucks it up.

Show threadGrimtuck1d ago
LLM’s are just very well-read morons.
Show threadNotMyOldRedditName1d ago

Ya i agree. The whole infrastructure of how these work is flawed for a trur AI/AGI.

It might be able to do a lot of cool things, but its fundamental flawed at its core.

Someone will need to figure out something completely different for a true AI.

Show threadNotMyOldRedditName1d ago

Oh also, I remember Elon once talked about how the upcoming cars would get bored when they weren’t doing anything with all that compute so they could do use that compute and pay people for it.

Paying for the compute isnt a terrible idea in the future, but become bored? LOL. Fucking crazy talk.

Like even if it was a true AI that could be bored. You’re now going to enslave it to do what you want on its free time?

Show threadlordbritishbusiness1d ago

Yeah, if it’s got the capacity to be bored it’s not going to stick around waiting for you. Pets act out when bored, as will AI, better to let the ghost in the machine go have fun in an arcade or something.

Current models can pretend to be bored when directed to, but they’re only facsimiles of thought at the moment, and the current approach probably won’t change that.