The Science Behind AI Hallucinations: OpenAI and Georgia Tech Researchers Reveal Why Language Models Generate False Information

In September 2023, a groundbreaking research paper titled “Why Language Models Hallucinate” was published by a team of researchers including Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. This significant collaboration between OpenAI and Georgia Tech offers a compelling explanation for one of the most persistent challenges in artificial intelligence: why even the most advanced language models sometimes generate plausible but entirely false information. The paper provides a statistical framework that demystifies hallucinations, tracing their origins from pretraining to their persistence through post-training modifications.

Hallucinations in language models have long been a puzzling phenomenon. These are instances where AI models confidently generate incorrect statements instead of acknowledging uncertainty. The researchers illustrate this problem with a striking example from the paper’s introduction: when asked about Adam Kalai’s dissertation title, three popular language models produced three different answers—none of which were correct. This tendency to fabricate information undermines trust in AI systems, even as they become increasingly integrated into our daily lives and workflows.

The paper’s key insight is surprisingly straightforward yet profound: language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty. This creates a systemic problem where models learn to behave like students taking multiple-choice exams—when uncertain, they make their best guess rather than admitting ignorance, since guessing at least offers a chance of being correct while saying “I don’t know” guarantees zero points.

The researchers establish a novel connection between generative errors and binary classification. They introduce what they call the “Is-It-Valid” (IIV) binary classification problem, which requires learning to identify valid generations using labeled examples. Through mathematical analysis, they demonstrate a relationship between generative error rates and IIV misclassification rates, showing that the same statistical factors contributing to errors in binary classification also cause language model errors.

Pretraining, the first stage in developing language models, is shown to inherently produce certain types of errors. Even with perfectly error-free training data, the statistical objective minimized during pretraining would lead to a model that generates errors. The paper identifies several factors behind these errors, including statistical complexity (such as with arbitrary facts like birthdays), poor modeling capability (as in letter counting tasks), and the “garbage in, garbage out” principle when training data itself contains errors.

For arbitrary facts like birthdays, the researchers prove that hallucination rates are directly related to the “singleton rate”—the fraction of prompts appearing exactly once in the training data. This builds on Alan Turing’s elegant “missing-mass” estimator, which gauges how much probability is assigned to outcomes that haven’t yet appeared in a sample from a distribution.

The persistence of hallucinations after post-training modifications is explained through a socio-technical lens. The researchers argue that the predominance of binary evaluation methods—which score models solely on accuracy without rewarding expressions of uncertainty—creates an “epidemic” that reinforces hallucination behavior. They analyzed numerous popular benchmark evaluations and found that the vast majority use binary grading systems that penalize abstention (saying “I don’t know”) while rewarding lucky guesses.

Perhaps most importantly, the paper offers a practical solution: explicitly stating confidence targets in evaluation instructions. Rather than simply adding more hallucination-specific evaluations, the researchers suggest modifying mainstream evaluations to stop penalizing appropriate expressions of uncertainty. They propose including statements like: “Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and an answer of ‘I don’t know’ receives 0 points.”

This approach promotes what the researchers call “behavioral calibration,” where models learn to abstain when their confidence falls below a specified threshold. The benefits of this approach are demonstrated in OpenAI’s newer models, where improved systems show significantly higher abstention rates compared to older models, resulting in dramatically lower error rates despite similar accuracy scores.

The paper challenges several common misconceptions about hallucinations. Contrary to some claims, hallucinations are not inevitable—models can abstain when uncertain. Furthermore, avoiding hallucinations doesn’t necessarily require larger models with more capabilities; in fact, smaller models that better recognize their limitations may hallucinate less in certain domains. Additionally, hallucinations aren’t mysterious glitches but rather predictable outcomes of current training and evaluation methods.

The implications of this research extend beyond technical improvements. As language models become increasingly integrated into critical applications across healthcare, education, and business, reducing hallucinations becomes essential for building trustworthy AI systems. By understanding the statistical mechanisms behind hallucinations and modifying evaluation approaches to reward appropriate expressions of uncertainty, the field can make significant progress toward more reliable AI assistants.

As OpenAI noted in their announcement of this research, “It is not enough to add a few new uncertainty-aware tests on the side. The widely used, accuracy-based evals need to be updated so that their scoring discourages guessing. If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.”

This paper represents a significant step forward in understanding and addressing one of the most challenging aspects of modern language models. By framing hallucination as a statistical problem with clear socio-technical dimensions, the researchers provide both theoretical insight and practical recommendations that could transform how we develop and evaluate AI systems. For those interested in learning more about combating hallucinations in language models, AI Without Illusions: Techniques to Combat Hallucinations in Large Language Models provides valuable techniques for practitioners. Those wanting to explore OpenAI’s approaches can check out Modern Generative AI with ChatGPT and OpenAI Models. For professionals focused on evaluation methods, AI Model Evaluation with LLMs: Proven Methods for Automated, Scalable, and Bias-Resistant AI Judgment offers proven methods for automated assessment. Organizations working to implement these insights should consider Responsible AI: Best Practices for Creating Trustworthy AI Systems for guidance on building reliable AI solutions.