I extremely doubt that hallucination is a limitation in final output. It may be an inevitable part of the process, but it's almost definitely a surmountable problem.

Just off the top of my head I can imagine using two separate LLMs for a final output, the first one generates an initial output, and the second one verifies whether what it says is accurate. The chance of two totally independent LLMs having the same hallucination is probably very low. And you can add as many additional separate LLMs for re-verification as you like. The chance of a hallucination making it through multiple LLM verifications probably gets close to zero.

While this would greatly multiply the resources required, it's just a simple example showing that hallucinations are not inevitable in final output

That’s not how LLMs work.

Super short version is that LLMs probabilistically determine the next word most likely to occur in a sequence. They do this using Statistical Models (like what your cell phone’s auto complete uses); Transformers (rating the importance of preceding words, so the model can “focus” on the most important words); and Relatedness (a measure of how closely linked different words/phrases are to reach other in meaning).

With increasingly large models, LLMs can build a more accurate representation of Relatedness across a wider range of topics. With enough examples, LLMs can infinitely generate content that is closely Related to a query.

So a small LLM can make sentences that follow writing conventions but are nonsense. A larger LLM can write intelligibly about topics that are frequently included in the training materials. Huge LLMs can do increasingly nuanced things like “explain” jokes.

LLMs are not capable of evaluating truth or facts. It’s not part of the algorithm. And it doesn’t matter how big they get. At best, with enough examples to build a stronger Relatedness dataset, they are more likely to “stay on topic” and return results that are actually similar to what is being asked.

This can only work in contexts where the LLM already has a strong Relatedness database for the topic. LLMs are incapable of assessing the accuracy of any information they weren’t trained on with sufficient examples to build that database.

This paper, if I’m understanding it correctly, is saying the same thing: LLMs will always hallucinate and are incapable of identifying hallucinations in certain contexts.

The problem we face is that it’s hard to identify hallucinations in the exact context when LLMs are most likely to make them: in content that’s not widely known and understood with many examples in the training data.

Or maybe I’m off base and I need to read the full study.