brianpeiris3d ago

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

https://lemmy.ca/post/62420870

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%. - Lemmy.ca

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks. > ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels. > > Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations. You can try the tasks yourself here: https://arcprize.org/arc-agi/3 [https://arcprize.org/arc-agi/3] Here is the current leaderboard for ARC-AGI 3, using state of the art models - OpenAI GPT-5.4 High - 0.3% success rate at $5.2K - Google Gemini 3.1 Pro - 0.2% success rate at $2.2K - Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K - xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K. ARC-AGI 3 Leaderboard [https://lemmy.ca/pictrs/image/c7521941-7eac-46f4-98de-876bcf99220c.png] (Logarithmic cost on the horizontal axis) https://arcprize.org/leaderboard [https://arcprize.org/leaderboard]

Show threadRustyShackleford3d ago
As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\_(ツ)_/¯
Show threadCosmicTurtle0 [he/him]3d ago

As a technologist, I have to remind everyone that AI is not intelligence. It’s a word prediction/statistical machine. It’s guessing at a surprisingly good rate what words follow the words before it.

It’s math. All the way down.

We as humans have simply taken these words and have said that it is “intelligence”.

Show threadunpossum3d ago

As another technologist, I have to remind everyone that unless you subscribe to some rather fringe theories, humans are also based on standard physics.

Which is math. All the way down.

Show threadxep3d ago
What maths do our memories follow? What about consciousness?
Show threadstray
We’re not actually individuals; we’re massive colonies of cells that work in concert. Memories and consciousness are both products of chemical interactions that happen between the cells, and the cells themselves are conglomerates of subatomic particles. Everything about us is determined by particle physics, which can be expressed and predicted mathematically.
Mar 28 at 5:31pmWeb
Show threadxep2d ago
The hubris of modern science and medicine is thinking that we know everything about our biology. I contend that we don’t. Can you tell me what’s in my gut microbiome?
Show threadstray2d ago

No one said we know everything about our biology. But we’re made up of particles, just like everything else. We don’t fully understand those particles either, but it doesn’t make them not real or not subject to the rules reality seems to follow.

They actually make little pills you can swallow to take samples at certain locations a long your digestion, so I suppose I could, given the knowledge and resources. Surgical sampling is also possible.

But I don’t see why it matters because all of the bacteria and archaea present in the body are made up of subatomic particles.