Why language models hallucinate, and what the research says reduces it

Language models hallucinate for two reasons. A fact they saw once or never in training cannot be recalled reliably, and the way they are trained and graded rewards a confident guess over an admission of doubt, which teaches them to fill every gap. Retrieval narrows the problem without closing it. What measurably reduces it is changing the incentives so that a wrong answer costs more than a refusal, then checking answers the model cannot reproduce, or that its sources do not support, before anyone sees them.

The internal platform I build at Tandem Interactive runs on a rule: the system says so when it cannot back a claim with data it retrieved, and my essay on building it covers why. I went through the research on hallucination to find out how far the evidence supports that rule, and where it says the rule costs more than it saves.

Why do language models hallucinate?

Part of the problem is statistical and arrives with pretraining, and the rest comes from how models are trained and graded afterward. Kalai and Vempala proved at STOC 2024 that a calibrated model’s hallucination rate on arbitrary facts is close to the share of those facts that appeared exactly once in its training data. The Nature paper Kalai published this year with colleagues from OpenAI and Georgia Tech turns that into a working example: if 20% of birthday facts appear exactly once in pretraining data, a pretrained model should hallucinate on at least 20% of them.

The same paper explains why post-training leaves the rest of the problem in place. Most benchmarks grade an answer as simply right or wrong. Under that scoring, abstaining is strictly suboptimal, because a refusal scores the same as an error while a guess sometimes lands, so a model tuned to climb those leaderboards learns to guess.

Their fix is to change the test. They call it an open rubric: the model is told how errors and abstentions are scored, and a penalty for a wrong answer is equivalent to a reward for abstaining. They also ran a simple mitigation on four frontier models, in which the model writes two answers, judges whether they agree, and abstains when they do not. It cut errors at some cost to accuracy. Under open rubrics the mitigation beat the baseline for every model and every penalty they tried. At the steeper penalties all four models scored below zero without it, meaning they would have done better by never answering at all.

Can a model tell when it doesn’t know?

Partly, and the signal is easier to read from the outside than from what the model says about itself. Farquhar and colleagues at Oxford showed in Nature in 2024 that asking the same question several times and measuring whether the answers agree in meaning flags confabulations better than the alternatives they tested. They call the method semantic entropy, and across 30 task and model combinations it averaged an AUROC of 0.790, against 0.691 for naive entropy and 0.698 for asking the model whether its own answer is true.

They define confabulations narrowly as answers that are both wrong and arbitrary, the ones that change with irrelevant details like the random seed, and they are clear that semantic entropy does not help when a model’s outputs are systematically bad. A model that learned a wrong fact will repeat it consistently, which is exactly the case this method cannot see. The detection literature has a reliability problem of its own. An EMNLP 2025 re-evaluation found that reported detector performance was inflated by the metric used to judge correctness, and that simple length-based heuristics can match or exceed sophisticated detectors like semantic entropy.

Scale does not solve it either. Zhou and colleagues found in Nature in 2024 that as model families were scaled up and shaped with instruction tuning, the early models often avoided questions while the later ones gave an apparently sensible but wrong answer much more often. Reasoning models have the same weakness in a newer form. The AbstentionBench study at NeurIPS 2025 found that reasoning fine-tuning degraded abstention by 24% on average, even in the math and science domains those models were trained for.

Does retrieval stop hallucinations?

Retrieval reduces hallucinations and leaves plenty behind. The clearest evidence comes from legal research, where the documents are curated and the stakes are high. Magesh and colleagues at Stanford tested the retrieval-backed research tools from LexisNexis and Thomson Reuters and found that each one hallucinated between 17% and 33% of the time on queries run in 2024, in a study published in the Journal of Empirical Legal Studies in 2025.

Joren and colleagues showed at ICLR 2025 that adding retrieval can lower how often a model declines. Claude 3.5 Sonnet abstained on 84.1% of questions in their comparison without retrieval and 52% with it, and the strongest models often answered incorrectly instead of abstaining when the retrieved context was not sufficient. Position matters as well. Liu and colleagues found in TACL in 2024 that models use information at the start and end of a long context far better than information in the middle, badly enough that GPT-3.5-Turbo with the relevant document buried among 20 or 30 others scored below its own closed-book accuracy.

A correct citation does not prove the model used it. Wallat and colleagues found at ICTIR 2025 that up to 57% of citations in their retrieval experiments were post-rationalized, meaning they were attached to answers the model had already produced from what it knew. A real link next to a claim is weaker evidence of grounding than it looks.

What measurably reduces hallucinations?

Incentives come first, because a model that is scored for guessing will guess however good the retrieval is. The TruthRL paper at ICML 2026 trained models with a reward that has three outcomes, so an abstention scores between a correct answer and a hallucination. In its headline setting, hallucination fell from 43.5% to 19.4% while truthfulness rose from 5.3% to 37.2%. The same paper found that a plain right-or-wrong reward produced the highest accuracy and lost the ability to abstain, which is the Nature argument showing up in training.

Where retraining is not an option, the output itself can still be trimmed down to what the model is sure of. Mohri and Hashimoto at Stanford showed at ICML 2024 that a conformal back-off, which strips a response’s least certain claims until a target correctness rate is met, takes GPT-4 output well past its base correctness while keeping most of the original claims. Their guarantee holds on average across inputs, so any single answer can still miss.

Retrieval systems can also judge whether the retrieved context is sufficient to answer the question and combine that judgment with the model’s confidence to decide when to abstain. Joren’s group tested that approach, and it raised the share of correct answers when a model responded by 2 to 10%. That same paper is a useful warning against overdoing it. Models answered correctly 35 to 62% of the time even with insufficient context, and a gate that refuses on every thin context throws those answers away.

Fine-tuning a model on facts it does not already know makes things worse, according to Gekhman and colleagues at EMNLP 2024. They found that as those examples are eventually learned, they linearly increase the model’s tendency to hallucinate, and that stopping early limits the damage because the new facts are mostly fitted late in training.

What do standards bodies say about hallucination?

NIST’s Generative AI Profile, published in July 2024 as a companion to its AI Risk Management Framework, calls this risk confabulation and defines it as the production of confidently stated but erroneous or false content. It describes confabulation as a natural result of the way generative models are designed. The OWASP Top 10 for LLM Applications covers the same ground from the security side, listing misinformation as one risk and excessive agency as another. Its mitigation for the second is to require a person to approve high-impact actions before they are taken, which is the control that matters most once a model can act on its own.

What should a production AI system do differently?

In the order I would build it:

1. Score every internal eval so a wrong answer costs more than an abstention, and tell the model what the penalty is.
2. Report abstentions as their own number beside accuracy, since a model can raise accuracy just by guessing more.
3. Check that the retrieved context can answer the question before generation, and let the model decline when it can’t.
4. Sample answers that matter more than once, and treat disagreement in meaning as a reason to hold the answer back.
5. Verify claims against the retrieved sources before display, because a citation beside a claim doesn’t prove the claim came from it.
6. Keep facts the base model doesn’t know out of fine-tuning data where possible, and stop training early when they have to go in.
7. Put a person in front of any action that can’t be undone.
8. Re-run abstention evals whenever the underlying model changes, since a reasoning upgrade can quietly lower refusals.

What hasn’t the research settled?

Almost every rate in this essay depends on its grader, and most of those graders are language models, so the numbers compare poorly across papers. The Nature paper also concedes that its open-rubric setup is less realistic than ordinary use, because people rarely tell a model what a wrong answer will cost them. I have not found a study that measures whether stating penalties in a production system prompt changes behavior the way it did on SimpleQA. That is the experiment I would most like to see run, since it would test the cheapest control on this list.

FAQ

Why do AI chatbots make things up?

A fact seen once or never in training cannot be recalled reliably, and training and benchmarks reward a confident guess over an admitted gap. A 2026 Nature paper by researchers at OpenAI and Georgia Tech shows that under right-or-wrong grading, abstaining is never the score-maximizing move, so models learn to guess.

Does RAG eliminate hallucinations?

Retrieval reduces hallucinations without eliminating them, and in some setups it makes models more willing to answer when they shouldn’t. Stanford researchers found retrieval-backed legal research tools hallucinated between 17% and 33% of the time, and an ICLR 2025 study found retrieval lowered how often models abstained even when the context couldn’t support an answer.

How can you detect when an LLM is hallucinating?

Ask the same question several times and check whether the answers agree in meaning, using the semantic entropy method published in Nature in 2024. It catches arbitrary errors well and misses errors a model makes consistently, so claims that matter should also be checked against their sources.

What is the difference between hallucination and confabulation?

NIST uses confabulation for any confidently stated false content, the same ground most people cover with hallucination. The Oxford semantic entropy paper uses the word more narrowly, for wrong answers that change with irrelevant details like the random seed, which is the kind that repeated sampling can detect.

Do reasoning models hallucinate less?

The evidence is mixed, because reasoning models get more hard questions right while the AbstentionBench study at NeurIPS 2025 found reasoning fine-tuning degraded abstention by 24% on average, with bigger reasoning budgets generally making it worse. Any switch to a reasoning model should come with a fresh abstention eval.