@cstross When I show stuff like this to "AI" boosters, their response is, "Sure, right, that's terrible, but you're asking it to do something it's not good at! Let's focus on what it *is* good at!"

To which I respond: The thing it's "good" at is lying with confidence. Which isn't, like, actually *good*.

@orionkidder But it's not "lying". To lie you need a model of truth and falsehood. LLMs don't have an internal model of the external world to provide context, all they've got is a probability distribution of one fragment of text being followed by another, entirely divorced from context. They have no means to determine whether the text they're producing in response to a prompt is a valid answer or just an answer-shaped string.

There is no intellect here, just text.