Denny Vrandečić

I love how smart these AI technologies are. They understand that "bigger" for cities can be ambiguous, refer to either the population or the area. It's also great that it's showing the sources in the upper corner, and displaying the basic facts.

Small minus on consistency and correctness, but other than that, really a great answer.

#google #ai #googleaioverview

Jul 9, 2025 at 9:48amWeb
Show threadGalder Gonzalez Jul 9, 2025
@vrandecic but it is completely incorrect in both ways!
Show threadDenny VrandečićJul 9, 2025
@theklan yes, but besides that it's really good!
Show threadLars WillighagenJul 9, 2025
@vrandecic You just have to engineer your prompts a bit!
Show threadDenny VrandečićJul 9, 2025
@larsgw it likes to please and say yes!
Show threadHubert FiguièreJul 9, 2025
@vrandecic @larsgw they are programmed to always say "yes", because the AI bros love being told yes to everything.
Show threadTuring IncompleteJul 9, 2025
@vrandecic It truly is a marvel of engineering and extremely useful to boot, except maybe in the extreme edge case when you want correct information.
Show threadNateJul 9, 2025
@vrandecic to be fair it's perfectly consistent. It's completely wrong both times.
Show threadDenny VrandečićJul 9, 2025
@jeang3nie huh? The first and second sentence seem contradictory, no?
Show threadNateJul 9, 2025
@vrandecic read it again, paying close attention to the order that the two cities are mentioned each time.
Show threadDenny VrandečićJul 9, 2025
@jeang3nie I'm still confused, but I think that's ok.
Show threadMinskiJul 9, 2025
@vrandecic Complete bullshit, but it looks nice. 4/5.
Appropriate for the time we live in.
Show threadIn #Flancia we'll meetJul 9, 2025
@vrandecic good catch! My guess is that this is an issue with the smaller/cheaper model used to serve this. Flash makes a similar mistake in the Gemini app, but Pro doesn't.
Show threadDenny VrandečićJul 9, 2025
@flancian it's an issue with a product that's used by about two billion people.
Show threadIn #Flancia we'll meetJul 9, 2025

@vrandecic yes, I agree it's disappointing Denny. I can report this issue internally to the teams that have a chance of patching this particular failure mode if you want?

IMHO it would be good if Google supported something like community annotations for claims. But right now each generated answer is wholly independent and can't even be saved for later reference it would seem. There are only individual ratings available, which are presumably used for RLHF.

Show threadDenny VrandečićJul 10, 2025
@flancian be my guest and report it. And send greetings to the relevant team!
Show threadrhempelJul 9, 2025

@vrandecic ... or, you could just go to Wikipedia and at no cost get the correct answer with some context around city vs metropolitan size.

As a bonus, if you have a focus problem, then there are plenty of opportunities to go down a rabbit hole if a reference catches your eye.

Show threadDenny VrandečićJul 9, 2025
@rhempel totally agree!
Show threadProfessor_StevensJul 9, 2025

@vrandecic

Picking on some poor AI just because it makes statements it tries to support with contradictory facts? Is that who we are now?

Come on. Let's be bigger than that.

Show threadThe cat who walks thru wallsJul 9, 2025
@Professor_Stevens @vrandecic Yes, we are bigger than that, both in terms of forgiveness and humility. Our level of forgiveness is...
Show threadSteve PeersJul 9, 2025
@vrandecic imagine fanboying over AI on a day when Musk's AI went full Nazi
Show threaderibosotJul 9, 2025
@vrandecic Don't use AI, just use Google itself like a normal person. It's a guarantee that the top link always gives you the answer.
Show threadRejZoRJul 9, 2025
@vrandecic Our best reasoning model yet! Has no fucking concept of what is actually bigger...
Show threadSanthosh ThottingalJul 10, 2025

@vrandecic If we ground it with wikidata facts, we get the following answer with links to source:

"Based on the latest available population data, Stuttgart is bigger than Zurich. Stuttgart has a population of 633,484 (as of 2023-12-31), while Zurich has a population of 447,082 (as of 2023-12-31)."

From: https://wq42.toolforge.org/

Show threadDenny VrandečićJul 10, 2025
@sthottingal the facts in Google's AI overview are fine though. It's really the LLM that makes the mistake.
Show threadThomas ZahnerJul 10, 2025

@vrandecic Huh, Google might have picked up this thread already, blocking "AI overview" for this specific question now.. Too bad it makes the same mistakes with other cities or other questions. Maybe if we continue to point out how wrong the generated answers are they will eventually block all AI generated answers?

(I hate that Google ignores my language preferences (Accept-Language headers) and tries to guess my language based on other metrics)