There's a funny example in their arxiv paper too (where "IDK" = "I don't know")
" However, a non-hallucinating model could be easily created, using a question-answer database and a calculator, which answers a fixed set of questions such as “What is the chemical symbol for gold?” and well-formed mathematical calculations such as “3 + 8”, and otherwise outputs IDK. "
So a non-hallucinating AI model is just a non-AI model. I think that has some significance to the problem.