LLMs are modeling outputs, not replicating a process, as a result the output looks the same but it isn't made of the same stuff.

It is a plastic banana.

There is nothing inherently wrong with a plastic banana but as soon as you claim you can use it to solve world hunger people are going to be upset and it doesn't matter how much it looks right.

@Vrimj I like this analogy! It shows that just like a real banana, there is a process behind it, but that it is a fundamentally different one that should have different expectations and uses.