I think I'm going to start using David Copperfield as an argument when talking with AI optimists. You see, lately I'm finding a bunch of people who acknowledges the issues and the (limited) capabilities of the current crop of LLMs, but are "optimist they can improve and fulfill their promises".
David Copperfield made the statue of liberty disappear. It was an astonishing feat, and you could argue that he's one of the most advanced 'magicians' in the history of the world. But you see, he's really an illusionist, 'magician' is just a pop term. I'm sure he's still improving his art. I'm sure what he does, besides remarkable and spectacular, could even be useful in some cases. But no matter how good he gets, no matter if he makes the entire continent of America disappear, it's still just an illusion. He's not making the statue of liberty disappear for real. And no matter how much better he gets, he won't ever be able to make real magic happen, he won't be able to make actual matter disappear and the appear back for real. Because that's not what he does, at all, and we with the current state of science, for what we know that may not even be possible in this universe.
LLMs are the same. Sure, what they do is technically impressive. But they are a statistical model: They get things from their training data and combine them to find what they compute to be the most statistically liking answer to a user input. They don't solve new problems, they don't understand what they are answering, they don't cogitate. And no matter how advanced they get, they won't ever do. Because that's not what they do. We may call them "AI", but that's a pop term, as 'magician' is for Copperfield. No matter how much better they get at what they do, what they do will always be to return the most statistically likely answer compiled from their training data. No matter how much they evolve, they won't ever cogitate, they won't ever solve new problems. Not because I'm not an optimist, but because that's not what this technology does.
David Copperfield will never make the statue of liberty disappear for real. It's not a matter of being an optimist or not.