Mastodawn

Analysis finds Google AI Overviews is wrong 10 percent of the time
Is 90 percent accuracy good enough for a search robot?
https://arstechnica.com/google/2026/04/analysis-finds-google-ai-overviews-is-wrong-10-percent-of-the-time/?utm_brand=arstechnica&utm_social-type=owned&utm_source=mastodon&utm_medium=social

Show thread

Richard Brockie 3d ago

@arstechnica @ianbetteridge

Easy answer: https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines

Betteridge's law of headlines - Wikipedia

Show thread

Joe ❌👑3d ago

@arstechnica It only does that well because most questions it is asked are very easy to answer. For the nontrivial questions it does a good deal worse, and it can't distinguish satire from reality, a feature that people are exploiting (the new version of Googlebombing).

Show thread

David Nash 3d ago

@not2b @arstechnica

That, and “90 percent accuracy” is often not the same as “90 percent utility”.

If you have a bread recipe with 10 steps, and an “AI” summary inadvertently deletes 1 of them, the recipe is “90%” correct, but if the deleted step is “add yeast”, the bread ranges from unpalatable to inedible, and if the deleted step is “add flour,” it completely fails to be bread.

Show thread

FLOWRI~1 aims to misbehave 3d ago

@arstechnica 90 % ought to be enough for anybody.

- Bill G.

Show thread

Benjamin J Gilbert 3d ago

@arstechnica hegeth would say yes

Show thread

SirWumpus 👾🍁3d ago

@arstechnica Frack no!

Show thread

Cassandrich 3d ago

@arstechnica This isn't even a meaningful claim. Overviews aren't binary "right" or "wrong". Maybe only 10% meet some extreme definition of "wrong", but none of them are accurate because they don't contain the key thing a search is looking for: provenance, a basis for knowing who the claims you're being presented with are coming from. And nearly all have some counterfactual claims.

Show thread