Google's AI overviews give wrong answers 9% of the time (1.25 billion times a day), and in over half of the cases point to websites that don't actually support the answer given.
https://futurism.com/artificial-intelligence/google-ai-overviews-misinformation
Analysis Finds That Google’s AI Overviews Are Providing Misinformation at a Scale Possibly Unprecedented in the History of Human Civilization

A new analysis commissioned by The New York Times suggests that Google's AI Overviews are wrong an astonishing percentage of the time.

Futurism
How Accurate Are Google’s A.I. Overviews?

The company’s A.I.-generated answers look authoritative, but they draw on an array of sources, from trustworthy sites to Facebook posts.

The New York Times