I thought this was fake or a bad result or something, but totally just duplicated it. Wow.

If you read the block of text…. It doesn’t make sense either.

I expect if you follow the references you'd find one of them to be one of those "if Earth was a grain of sand" analogies.

People like laughing at AI but usually these silly-sounding answers accurately reflect the information the search returned.

It’s in the quote that they scaled it.

The point is that the entire alleged value is the ability to parse the reading material and extract the key points, but because it doesn’t resemble intelligence in any way, it isn’t actually capable of meaningfully doing so.

Yes, not being able to distinguish between the real answer and a “banana for scale” analogy is a big problem that shows how fucking useless the technology is.

Except it is capable of meaningfully doing so, just not in 100% of every conceivable situation. And those rare flubs are the ones that get spread around and laughed at, such as this example.

There's a nice phrase I commonly use, "don't let the perfect be the enemy of the good." These AIs are good enough at this point that I find them to be very useful. Not perfect, of course, but they don't have to be as long as you're prepared for those occasions, like this one, where they give a wrong result. Like any tool you have some responsibility to know how to use it and what its capabilities are.

AIs are definitely not “good enough” to give correct answers to science questions. I’ve seen lots of other incorrect answers before seeing this one. While it was easy to spot that this answer is incorrect, how many incorrect answers are not obvious?

Then go ahead and put "science questions" into one of the areas that you don't use LLMs for. That doesn't make them useless in general.

I would say that a more precise and specific restriction would be "they're not good at questions involving numbers." That's narrower than "science questions" in general, they're still pretty good at dealing with the concepts involved. LLMs aren't good at math so don't use them for math.