Mike Tyka

Hallucination is Inevitable: An Innate Limitation of Large Language Models

https://arxiv.org/abs/2401.11817

"In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. [...] By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hallucinate. "

@garymarcus

Hallucination is Inevitable: An Innate Limitation of Large Language Models

Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

arXiv.org
Feb 25, 2024 at 11:54amWeb
Show threadgjmFeb 26, 2024

@mtyka @garymarcus This paper has absolutely nothing to do with either LLMs or hallucination. What it proves is a formalization of the following truism: "No system can learn everything about an infinite world from a finite amount of data".

More precisely: suppose your "world" is specified by an arbitrary, unknown computable function F, defined on a countably infinite set, and what you're trying to do is to reconstruct _exactly F_. So you have some procedure that uses only a finite number of pieces of information of the form F(x)=y, and produces some other computable function G that agrees with F as far as that information goes. And G has to provably always yield an output for any given input. In this scenario, they prove, there will be choices of F for which the resulting G isn't identical to F.

This is of course completely unsurprising. It could be an exercise in a course on computability in undergraduate computer science.

There is nothing specific to LLMs here -- e.g., provided whatever human brains do is computable, it applies just as much to us.

It has nothing to do with LLM "hallucinations" -- except perhaps the following; in their setup, they require their learned function G always to yield a definite output and never say "I don't know" -- which means that when you ask it "what is F(x)?" for an x it doesn't learned about, it by definition has to make something up. An actual LLM could learn to Not Do That; the hallucination problem is that they tend not to; but in this paper it's "inevitable" because the formalism they chose to impose implies it.

Next up, perhaps: "Death is Inevitable", wherein they show that a provably terminating program cannot run for ever.

Show threadGary MarcusFeb 27, 2024
@gjm @mtyka @garymarcus I strongly suspect that the paper’s conclusion is right but agree the proof is not compelling.
Show threadgjmFeb 27, 2024

@garymarcus @mtyka The proof is a perfectly good proof, of a proposition that has nothing to do with the title of the paper. The claim in the title of the paper might be true, but if so it will be for reasons that have nothing to do with anything proved in the paper.

(Actually, I haven't checked their proof, but I assume that if it were actually wrong someone would have noticed by now, and the actual thing it's proving is obvious.)

For anyone hoping the technical content of the paper would justify the title, it's not that the proof is _not compelling_, it's that it's _completely absent_.

Show threadJoanna Bryson, blatheringFeb 27, 2024
@mtyka @garymarcus @UlrichJunker Forget the computable functions, there will always be an edge of knowledge, and hallucination is what it looks like when you have semantically underinformed/inadequately-contextualised text being generated with highly informed syntax (note there's actually more than two levels of learning going on here, so that sentence is also just an approximation – compressed to 2 degrees of freedom – but I hope the implications are clear.)