Mastodawn

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

https://lemmy.ca/post/62420870

Announcing ARC-AGI-3 - An benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%. - Lemmy.ca

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks. > ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels. > > Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations. You can try the tasks yourself here: https://arcprize.org/arc-agi/3 [https://arcprize.org/arc-agi/3] Here is the current leaderboard for ARC-AGI 3, using state of the art models - OpenAI GPT-5.4 High - 0.3% success rate at $5.2K - Google Gemini 3.1 Pro - 0.2% success rate at $2.2K - Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K - xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K. ARC-AGI 3 Leaderboard [https://lemmy.ca/pictrs/image/c7521941-7eac-46f4-98de-876bcf99220c.png] (Logarithmic cost on the horizontal axis) https://arcprize.org/leaderboard [https://arcprize.org/leaderboard]

Multiplexer 3d ago

Link to the recent Al Explained video mainly covering ARC-AGI-3:
www.youtube.com/watch?v=s4tptozUJ8Y

Two AI Models Set to “stir government urgency”, But Will This Challenge Undo Them?

YouTube

Biased study. Take any average person off the streets and shove this thing in their face. That 100% notion will go down fast.

brianpeiris 3d ago

ARC-AGI-3 Launch event - Shared publicly live on March 25 in San Francisco at Y Combinator HQ, featuring a fireside conversation between François Chollet (creator, ARC-AGI) and Sam Altman (CEO, OpenAI) on measuring intelligence on the path to AGI.

François Chollet is a software engineer, artificial intelligence (AI) researcher, and former Senior Staff Engineer at Google. Chollet is the creator of the Keras deep-learning library released in 2015.

tomalley8342 3d ago

They didn’t say “100% of humans can solve this benchmark”, they said “humans can solve 100% of this benchmark”.

I couldn’t get past the second level :(

feelsbadman. You need more RAM!

ExLisper 3d ago

Guy, I found the bot!

MagicShel 3d ago

I see by your lack of pluralization that you’ve realized there’s only one person here and everyone else is bots. However through inference and deduction, you are therefore also a bot. I have good reason to believe I am the non-bot though I wonder if I could know for certain…

That was a lot of effort for a typo joke…

ExLisper 3d ago

My programming tells me I’m not a bot.

Aceticon 3d ago

Cogito, ergo sum

I finished one of the tasks. And, I imagine I could finish at least some of the others. But, I wasn’t being paid, and it wasn’t very entertaining, so I stopped.

They should ad a “global” and “friends-only” leaderboard (like the Zachtronics games, etc.) and really see the competition (at leat human competition) heat up.

potustheplant 3d ago

Of the first task? Yikes.

“Humans score 100%. Frontier AI scores 0.26%.”

The title deals in absolutes.

davidgro 3d ago

Those are high scores.

🤔 So this is a visual comparison between peak performance of some humans and peak performance of current LLMs in a controlled environment?

floquant 3d ago

Is this a gotcha? Not sure where you got the “visual” from, but yes it is best human performance vs best LLM performance

I don’t know why you assume there has to be a gotcha, maybe it’s the competitive background… Anyway, it’s visual because you look at it to see it. And it’s not the best human performance vs best LLM performance, it’s best controlled performance because the testing is limited to a set of parameters.

floquant 3d ago

That’s what games are? I really don’t see how it is an unfair comparison to you. How would you change it?

Stress test it. Low, average, high, impairment conditions, safeguards off, order, chaos and everything in between.

I haven’t read all of their Benchmark introduction and Technical Documentation. I assume you have and didn’t find any of the tests you’re asking for?

ARC Prize - What is ARC-AGI?

The only AI benchmark that measures AGI progress.

ARC Prize

;)

pulsewidth 3d ago

Pretty defensive there. It’s not even a study

If it studies something, it’s a study. If you feel defensiveness, you consider aggression. If you feel bias in one way, someone can feel bias in another way. If there’s an action, there’s a reaction.

If you feel defensiveness, you consider aggression.

Aggression as in calling something biased without providing evidence?

As in assuming you are starting with an unbiased point of view.

Of course we all have our biases. But what to do with that lesson? It can be a convenient response whenever someone disagrees with us. But it can also serve as a powerful motivation to find some common ground against all odds. The universe is chaotic. Language is illogical. Yet sometimes we find stuff we can agree on. Isn’t that beautiful?

pulsewidth 2d ago

If there’s an action, there’s a reaction.

Sort of like how when people outsource all their critical thinking to AI, their ability for critical thinking atrophies?

I’m studying these comments, now I am a study

I salute your dedication to science. 🫡

RustyShackleford 3d ago

As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\_(ツ)_/¯

It could also be that it lacks the machinery to feel any emotions at all. You don’t (normally) have to train people to be afraid of bears or heights or loneliness or boredom. You also don’t (normally) have to train people to have empathy or compassion.

I argue that our obsession with AI is, itself, a misalignment with our environment; it disproportionately tickles psychological reward centers which evolved under unrecognizably different circumstances.

dblsaiko 3d ago

You don’t (normally) have to train people to be afraid of bears or heights or loneliness or boredom. You also don’t (normally) have to train people to have empathy or compassion.

So what are you implying about people who don’t experience these?

What am I implying? That their machinery is abnormal and they likely need assistance to live normal, healthy lives. That’s literally why the fields of psychiatry and psychology exist: healthy people don’t need doctors and therapists. Do you disagree?

sp3ctr4l 2d ago

Introverts exist, and are… very often fine with solitude, prefer it generally over socializing.

But they are generally fine at participating in society and living normal lives.

Healthy people… do need doctors … and therapists.

A person can outwardly appear to be healthy… and actually not be.

Preventative medicine, regular checkups, your body changes as you grow, and habits you develop in your youth may need significant reworking.

Therapy can give otherwise healthy people a method of exploring their inner selves more fully or more consistently… they can teach them frameworks for understanding and dealing with other kinds of people, for being better able to deal with kinds of trauma they have not yet experienced.

Also… same with physical health… people with some nascent mental problems or patterns forming… probably won’t be obvious to a non specialist, untill it gets more severe.

Introverts exist, and are… very often fine with solitude, prefer it generally over socializing.

Definitely! I am one :) but I still desire the presence of friends from time to time (and usually in small groups).

A person can outwardly appear to be healthy… and actually not be.

Yup! There’s always a nonzero chance you’re not as healthy as you think you are (let’s call it the quantum theory of health: everyone is in a superposition of being both healthy and unhealthy at the same time), especially as we change due to age, making us unfamiliar with our own bodies… I’d tell you about my own challenges here, but that’d be TMI.

And, yes, that’s why we go to regular checkups with someone who has a better perspective to judge “healthiness” (side note: doctors aren’t perfect, so visiting them too frequently can be worse than never at all; there’s a “healthy” cadence to checkups).

Therapy can give otherwise healthy people a method of exploring their inner selves more fully or more consistently…

This boils down to the definition of “healthy”. It even becomes a philosophical question that’s really hard to answer… Is it healthy to live a sedentary lifestyle? Is it healthy to exercise too much? Is it healthy to not know TIPP, in case you (or a loved one) gets a panic attack? Is it healthy to ignore yourself? Ignore others? Is it healthy to mention quantum superposition in a conversation about health?

But, yes, I agree. Life’s as messy and diverse as as hard to sum up as everybody whose ever lived, but yet we carry on … I hope that’s healthy :)

Doctors overestimate risk leading to over-diagnosis, overtreatment, study finds

Primary care practitioners often over-estimate the likelihood of a patient having a medical condition based on reported symptoms and laboratory test results. Such overestimations can lead to overdiagnosis and overtreatment, according to a recent study conducted by researchers at the University of Maryland School of Medicine (UMSOM) published in JAMA Internal Medicine.

Medical Xpress

sp3ctr4l 2d ago

My entire point is that you are just overgeneralizing, in general, and saying rather silly things.

Fair enough; the Internet is a silly place full of distracted, armchair philosophers. However, my entire point was that an LLM doesn’t rely on machinery in the same way that a human brain does. That doesn’t make AI “worse” or “better” overall, but it does make it an awful replacement for humans.

Havoc8154 3d ago

I guess you don’t have children.

You absolutely do have to train them to be afraid of bears, heights, and every fucking thing you can imagine. You absolutely do have to teach them empathy and compassion. There may be some nugget of instinct, but without reinforcement it might as well not exist.

Hah, okay, you got me there. From my understanding, though, that’s mostly because kids are still figuring out what’s “normal”, so their fear instinct isn’t nearly as strong. I guess I should’ve stuck to the more instinctive sources of fear…

Regardless, that’s not really my point. My point is an LLM doesn’t rely on machinery in the same way that a human brain does. That doesn’t make AI “worse” or “better” overall, but it does make it an awful replacement for other humans.

Are you anthromorphizing word suggester into a being experiencing things?

RustyShackleford 3d ago

No.

CosmicTurtle0 [he/him]3d ago

As a technologist, I have to remind everyone that AI is not intelligence. It’s a word prediction/statistical machine. It’s guessing at a surprisingly good rate what words follow the words before it.

It’s math. All the way down.

We as humans have simply taken these words and have said that it is “intelligence”.

unpossum 3d ago

As another technologist, I have to remind everyone that unless you subscribe to some rather fringe theories, humans are also based on standard physics.

Which is math. All the way down.

What maths do our memories follow? What about consciousness?

Obligatory xkcd… we’re just meatbags somewhere to the left Purity

On a more serious note, there’s plenty to explore there and there are some potentially interesting links to quantum physics and stuff in our brain, as well as how certain drugs can completely disrupt our consciousness (ever had an operation?) and how it could link up. But there is obviously no definitive answer.

At best consciousness is whatever flavour of philosophical interpretation/explanation you like at any given time.

wonderingwanderer 3d ago

Philosopher: looks at the mathematician…

Iconoclast 3d ago

Consciousness (the fact of experience) doesn’t necessarily need to be linked to intelligence. It might be but it doesn’t have to. An LLM is almost definitely more intelligent than an insect but it most likely is like nothing to be an LLM but it probably is like something to be an insect.

partofthevoice 3d ago

Isn’t it kind of eery that you can only suppose it must be “like something” to be an insect, from the very precise bias of being human? We’re projecting the idea that “it’s like something to be something [as a human]” only the experience of other things.

How would we describe what it’s like? Would something poetic suffice, such as “it’s like being a leaf in the wind, and with weak preference of where you blow but no memory of where you’ve been.” … but, all of that is human concepts, human experience decomposed into a subset of more human experiences (really weird, the recursive nature of experience and concepts).

I think the idea of “what it’s like…” has some interesting flaws when applied to nonhumans. It kind of presupposes that insects are lesser, in a way. As though we can conceptualize what it’s kind to be them, merely by understanding a stricter subset of what it’s like to be human.

Iconoclast 3d ago

I can only suppose that of other people as well. There’s no way to measure consciousness. The only evidence of its existence is the fact that it feels like something to be me from my subjective perspective. Other humans behave the way I do so I assume they’re probably having similar experiences but I have no idea what it’s like to be a bat for example.

However, answering the question “what it’s like to be” is not relevant here. What’s relevant is that existence has qualia at all.

What Is It Like to Be a Bat? - Wikipedia

partofthevoice 3d ago

However, answering the question “what it’s like to be” is not relevant here. What’s relevant is that existence has qualia at all.

Does existence “have qualia?” That treats qualia almost like it’s ontological, if I’m interpreting you correctly. Yet, qualia can only exist from the perspective of a being with the capacity to model a (seemingly external) world via said qualia. There is no magic qualia sauce we can embed inside something.

Qualia, I think, is a process of information reduction… but also it’s a flavor of information interrogation. Because, reducing electromagnetic radiation to “visual perception” happens inside light sensors too — albeit without counting as “qualia.”

What would you say counts as “qualia?” Or rather, what are its dependencies?

Iconoclast 3d ago

It’s the fact of subjective experience - the warmth of a campfire, the bitterness of lemon, the greenness of green. We’re essentially talking about consciousness here. The fact that there’s something it is like to be.

While nobody knows what consciousness is or how it comes about, what I mean by it is best captured by the philosopher Thomas Nagel in his aforementioned essay “What Is It Like to Be a Bat?”

Nagel argues that consciousness has an essentially subjective character, a what-it-is-like aspect. He states that "an organism has conscious mental states if and only if there is something that it is like to be that organism – something it is like for the organism.

partofthevoice 3d ago

The premise still strikes me as odd. How can we know it’s like anything to be anything, if we can not know what it’s like to be anything else? Coming from a premise that, to truly understand anything, you must also understand what it is not.

Is it really fair to presume, from our biased perspective where “likeness” is an abstract quality of “being,” that everything ought have a manner of which it is like to be?

What about the totality of the universe, to include all its embedded agents. What would that be like? Would an ever small portion of that likeness include precisely what it’s like to be me?

Do you think it would be possible to qualitatively describe and differentiate between two distinct phenomenologies, one day? Not just behaviorally, but to actually differentiate between their internal processes — what it’s like to be them?

And what might it be like to be a whirlpool, lightning, or even an entire ecosystem? Would that strictly be as ludicrous as asking “what might it be like to be a rock,” or is there something else to be said given whirlpools, lightning, and ecosystems are more-or-less events rather than objects?

I don’t disagree with the argument you shared… I think there’s an obvious difference between what it’s like to be a bat versus a human, but I also feel like we’re missing something important that clearer terminology could work out.

Iconoclast 3d ago

The premise still strikes me as odd. How can we know it’s like anything to be anything

Because it undeniably feels like something to be in this very moment from the perspective of my subjective experience. In fact, I’d even go as far as to claim that it’s the only thing in the entire universe that cannot be an illusion. I could be a mind living in a simulated universe on an alien supercomputer, with every person I’ve ever interacted with just being a convincing AI, or I could be a Boltzmann brain - but what remains true despite all that is that something seems to be happening.

I think the closest we can get to true unconsciousness that you can still come back from is general anesthesia. It’s nothing like sleep. It’s like that period of time doesn’t even exist. It’s like the time before you were born.

partofthevoice 3d ago

In fact, I’d even go as far as to claim that it’s the only thing in the entire universe that cannot be an illusion.

Descartes would too, I doubt I think, therefore I think…

What about a third-term fetus (3tf)? To me, I think it’s obvious and intuitive that a 3tf has an experience. This is as obvious and intuitive to me as a rock not having an experience. Yet, there’s also something similar about them which isn’t made obvious by those two points; both a rock and a 3tf can (perhaps) be said to be sharing the same kind of experience.

A 3tf would have experience that doesn’t contain meta-cognitive function (e.g., self awareness). That said, the experience of a 3tf can (again, perhaps) be modeled simply as a function like experience=fn(qualia) where qualia=nervous-system-capacity + stimuli. Effectively, it’s the structure of the being (the nervous system) being exposed to the world (stimuli). Rocks can be said to be the same, with a very “poorly functioning” nervous system. You can model a rock’s experience too, given qualia=0 for the rock.

From this framing, I think it starts to become more clear that we’re discussing a kind of physical process. Qualia starts to look like a name we’ve given to that particular process, and less like it’s some elusive thing which evades scientific understanding.

I’m partially not convinced that it feels like anything to feel something, though. I mean, I do understand feeling happy, angry, sad, even sublime. But these are categories of feeling that my very own internal processes have conjured up. How can I be sure that “feeling” something isn’t similar to the kind of illusion a heap of cells can evolutionarily succumb to when it begins to regard “itself” as separate from its environment? You wouldn’t use the sense of “self” to justify “I exist as myself, for fact.” So why would our experience of phenomenology be different?

Iconoclast 2d ago

I can only assume a fetus likely has some faint level of consciousness while a rock has none, but again, this is all just speculation. I couldn’t possibly know for sure. Consciousness and qualia are entirely subjective experiences. There’s no evidence of them in the universe outside our own direct experience of it. If I wasn’t conscious myself, I wouldn’t even have any idea that such a thing exists.

What it is - I have no idea. If I had to bet, I’d say it’s an emergent feature of a sufficient level of information processing and therefore a physical process, but that’s just my speculation. Nobody actually knows.

Illusions are experiences. That’s why I say consciousness cannot be an illusion: the very fact that you’re experiencing the illusion proves that the space where that illusion appears exists - and that space is consciousness.

I’m also not talking about any human concepts we layer on top of feelings, nor the thoughts we have about those feelings. I’m only talking about the raw sensation itself that our brain then interprets as hot, wet, green, bitter, and so on.

The sense of self implies some kind of center of consciousness or thinker of thoughts. I don’t buy that. Thoughts just appear - nobody is authoring them. I speak of “me” or “I” as a being in the universe, but that’s just because it’s the only way I know how to refer to these things. I don’t know how accurate my view of the universe really is. Like I said: I could just be a mind living in a simulated universe. I don’t think I am, but it would be perfectly compatible with my experience.

partofthevoice 2d ago

Preface: I agree with pretty much all of what you said.

The other day, though, I had washed my hands. I had to be careful because one of my fingers can’t get wet due to an injury. While carefully washing my hand, I noticed that I was “experiencing” wetness all over my hand — to include on portions that were completely dry. I found this rather interesting, that I was experiencing something which I knew to be factually false. I wonder if the difference between processing and experiencing could have something to do with that.

I think a lot about this stuff.

conscious beings seem to self-produce composite models of the world, from which the world can be effectively navigated.
conscious beings seem to also model themselves. This is keenly distinct from self-awareness. I’m referring to a model that helps you balance, walk, know when you’re hot or cold, …
conscious beings can have “concepts,” which seem to be recursive and generative. You can’t describe a concept without referring to more concepts. There is no “root” concept. Also, for some reason, it’s often easier to understand what a “concept” is by investigating what it is not.
conscious beings seem to be able to compartmentalize composite “concepts” into a singular, singular irreducible concept. Like if I conceptualize a combination of “banana,” “bread,” and “pudding,” I might come up with a brand new experience of “banana bread pudding.” That new experience can be referenced in its own right, and it’s not necessarily reducible back to the concepts which birthed it in the first place.

I could go on and on. Sometimes I think it’s ridiculous that I can’t so easily find existing material on this stuff.

You seem to be well versed on this topic. Can I ask what your study materials have been?

partofthevoice 2d ago

The taxonomy/topology of concepts are especially fun to think about. I think concepts are rather interesting, alike experience, because it’s similarly something that is unique to consciousness.

Iconoclast 2d ago

There’s also the concept of consciousness without memory. What’s that like? Being able to experience the current moment but having no memory of any past experiences - including your experience one second ago.

Or here’s a scary thought: what if general anesthesia doesn’t actually switch off consciousness but simply blocks new memories from forming? You could experience the full horror of being awake during surgery but remember none of it. From the perspective of “now,” that would be functionally the same as never having experienced it at all.

Then there are those extremely weird recordings from split-brain studies. Back when grand mal seizures were treated by cutting the corpus callosum - the bridge between the two brain hemispheres - to stop the “storm” from spreading. On the surface these patients seemed completely normal after the operation, but some really strange stuff shows up when you start testing them properly.

There’s a way you can communicate with each hemisphere independently without the other one knowing. The left hemisphere controls the right side of the body, the right hemisphere the left side. You can flash text on the left side of the visual field (which only the right, non-verbal hemisphere sees), then hand them a pen and let the left hand (controlled by the right hemisphere) answer questions by writing. Turns out that the two halves often don’t agree on things. Ask the right hemisphere what it wants to do for a living and you’ll get a different answer than what the left hemisphere says out loud. Or you can give the right hemisphere a task - “go get a glass of water” - and when you ask the left hemisphere why it did that, it just makes up an explanation. “I was thirsty,” it’ll say, even though the researchers know that’s not true. It genuinely seems like there are two separate consciousnesses running in the same brain at the same time. The big question is: were they there all along, or does the second one only emerge once the connection between them is cut?

And yeah, this is all stuff I’ve absorbed from podcasts covering these topics - mostly from Sam Harris. I’m just naturally really curious about the human mind, and I’m pretty experienced with meditation as well, so I probably pay attention to my day-to-day conscious experiences about 1% more than the average person. I’m however not in any way expert on this. It’s not even remotely related to what I do for living.

We’re not actually individuals; we’re massive colonies of cells that work in concert. Memories and consciousness are both products of chemical interactions that happen between the cells, and the cells themselves are conglomerates of subatomic particles. Everything about us is determined by particle physics, which can be expressed and predicted mathematically.

The hubris of modern science and medicine is thinking that we know everything about our biology. I contend that we don’t. Can you tell me what’s in my gut microbiome?