Why Large Language Models Hallucinate and How to Govern Them

1. What Are Large Language Model Hallucinations?

A large language model (LLM) hallucination is generally an output that appears fluent and logically reasonable but conflicts with known facts, the provided input, or verifiable evidence. Hallucinations do not necessarily contain obvious grammatical errors. In fact, their strong linguistic fluency often makes users more likely to mistake them for factual information.

From an evaluation perspective, hallucinations can be viewed across two main dimensions.

1.1 Factualness

A factual error occurs when the model’s output does not match verifiable facts in the real world. Examples include inventing nonexistent policies, personal histories, technical specifications, or phone numbers, or confusing the time, location, and outcome of two different events.

These errors are particularly risky in open-domain question answering, legal consultation, medical information, and enterprise knowledge systems.

1.2 Faithfulness

Unfaithfulness occurs when a response does not correctly follow the provided source material or conflicts with it. For example, a product document may explicitly state that a feature is not supported, while the model’s summary says that the feature is already available. Similarly, a source may describe a project as being in the planning stage, while the model presents it as already launched.

Factualness and faithfulness are not exactly the same. Information that is absent from the input does not automatically constitute a hallucination. If the additional information can be confirmed through reliable external facts, it may simply be a reasonable extension. However, when its source cannot be verified, it should be treated as a potential risk.

Therefore, in open-domain conversations, hallucination detection should not focus only on whether the answer goes beyond the input. It should also examine whether the information is true, traceable, and consistent with the surrounding context.

2. Why Large Language Models Hallucinate

2.1 Language Models Predict Text Rather Than Directly Verify Facts

The basic training objective of an LLM is usually to predict the next token based on the available context. During training, the model learns language patterns, knowledge associations, and ways of expressing information. It does not automatically create a strict truth constraint for every fact in the way a database is designed to do.

As a result, the model is better at producing expressions that sound reasonable than at determining whether those expressions have been validated by reliable evidence. When several possible answers have relatively high language probabilities, the model may choose the most natural-sounding wording rather than the most factually accurate answer.

High fluency should therefore not be treated as evidence of high reliability. The more closely an answer follows familiar language patterns, the easier it may be to overlook details that have not been verified.

2.2 Pretraining Data Contains Errors, Conflicts, and Outdated Knowledge

An LLM’s parametric knowledge comes from large-scale data, and real-world data is not perfectly accurate. Training data may contain:

• Factual errors or unverified claims
• Conflicting conclusions from different sources
• Outdated policies, software versions, or statistics
• Incorrect opinions that have been repeatedly reproduced
• One-sided descriptions caused by missing context

Models generally do not attach an explicit source, timestamp, or confidence label to every item of learned knowledge. When users ask questions, the model may activate a group of related facts without being able to determine whether they are still valid or whether they conflict with one another.

For rapidly changing domains, relying only on the model’s parameters is especially risky. Regulations, market information, organizational structures, software versions, and public events should not be handled entirely through static parametric memory.

2.3 Training Objectives May Encourage the Model to Provide an Answer

In practical conversations, users usually expect the model to solve a problem directly rather than repeatedly respond, “I don’t know.” If training places too much emphasis on completeness, politeness, and task completion, the model may develop a tendency to produce a seemingly complete answer even when sufficient evidence is unavailable.

This tendency is particularly visible when:

• The user’s question contains a false premise
• The question concerns an entity that rarely appears in the training data
• The user requests a large number of details
• The question requires precise numbers, sources, or dates
• The context is insufficient but the task requires a conclusion

Without effective mechanisms for refusal, clarification, and uncertainty expression, the model may fill information gaps with speculation.

2.4 Decoding Strategies Affect the Probability of Incorrect Content

During generation, the model typically selects the next token from multiple candidates. Higher-randomness sampling provides more expressive variety but can also make the model diverge from the most probable factual path, increasing the risk of fabricated names, numbers, citations, and event details.

Reducing randomness can lower some forms of stochastic hallucination, but it cannot solve the problem completely. The most likely output from the model may itself be wrong. In other words, deterministic generation can make the model’s internal judgment more stable, but it cannot guarantee that the judgment is correct.

Decoding strategies should therefore be selected according to task risk:

• For factual question answering and data extraction, prefer lower randomness and structured output.
• For creative writing and advertising copy, a higher level of generation diversity may be appropriate.
• For high-risk domains, prioritize evidence constraints and external retrieval rather than relying only on decoding adjustments.

2.5 Training and Generation Involve Different Distributions

During training, the model usually processes complete, human-written text. During generation, however, it must rely on its own newly generated content to continue predicting what comes next. If an incorrect entity, date, or conclusion appears early in the response, later content may be built around that error.

This accumulation of errors is more visible in long-form generation. The longer the response, the more entities, dates, causal relationships, and numerical relationships the model must maintain. A local error can therefore expand into inconsistency across an entire passage.

For example, the model may first misidentify the year in which a project started. It may then use that year to infer the project’s duration, participants, and development stage, eventually producing a narrative that is internally coherent but globally incorrect.

2.6 Long Contexts Create Information Selection and Attention Challenges

A larger context window does not mean that the model can use every piece of information with equal accuracy. In long documents, the model may:

• Ignore important constraints located in the middle of the document
• Focus too heavily on information at the beginning or end
• Confuse multiple similar entities
• Combine conclusions from different sections
• Omit negations, time qualifiers, or conditional requirements

Long-document question answering should therefore not simply provide the entire document to the model in one pass. A more reliable approach is to retrieve relevant passages first, answer based on those passages, and require the model to identify where the supporting evidence appears.

3. How to Evaluate LLM Hallucinations

3.1 Reference-Based Evaluation

Traditional text-generation tasks often use human-written reference answers or evaluate outputs based on word overlap. The generated result is compared with the standard answer at the level of words, phrases, or sequences.

This approach is useful for measuring similarity of expression, but it has significant limitations:

• The same question may have multiple correct answers.
• Similar wording does not guarantee factual correctness.
• The reference answer may be incomplete.
• The method cannot identify additional information that is true.
• Its applicability to open-domain dialogue is limited.

Reference-based evaluation is therefore more suitable for closed tasks or tasks with relatively stable formats. It should not be the only measure of factuality in open-domain conversations.

3.2 Source-Based Evaluation

For summarization, document question answering, and knowledge-base question answering, the output can be checked against the input materials. The key questions are not simply how many words the answer shares with the source, but whether:

• Every key conclusion is supported by the source
• The response changes the source’s conditions, scope, or time frame
• It introduces entities or numbers that do not appear in the source
• It omits important qualifiers that would change the conclusion
• It presents speculation as a confirmed fact

This method focuses more directly on faithfulness and is useful for detecting conflicts between the input and output. However, it cannot determine whether the source itself is true, so it cannot fully replace factual verification.

3.3 Information-Extraction-Based Evaluation

Another approach is to convert the answer into structured facts, such as subject–relation–object triples or event–time–location–participant structures, and then compare them with a source or knowledge collection.

A typical process is:

1. Extract entities, relationships, attributes, and events from the model’s answer.
2. Extract the corresponding structures from the reference source.
3. Compare the two sets of structures.
4. Calculate the proportions of missing, conflicting, and unverifiable facts.

This method makes it easier to locate specific errors, such as incorrect relationships between people, wrong dates, or inaccurate numbers. However, it depends on an additional information extraction model. If the extraction stage misses or misidentifies information, those errors may propagate into the evaluation results.

In addition, simple triple structures cannot fully represent complex conditions, causal relationships, negation, or long-distance semantic dependencies.

3.4 Question-Answer Consistency Evaluation

Question-answering evaluation can be used to test whether the claims in a response are supported by the original source. The general idea is to generate several questions from the model’s answer, answer those questions using the source material, and then compare the results.

A typical workflow includes:

1. Identify key facts in the response.
2. Generate verification questions for those facts.
3. Answer the verification questions using the original materials.
4. Compare the conclusions in the original response with the answers supported by the source.
5. Calculate the support rate, conflict rate, and unverifiable rate.

This approach can capture richer semantic relationships than simple word matching. However, it introduces dependence on two additional model-based steps: question generation may omit important facts, and the verification model may produce new errors. Important applications should therefore combine this method with manual sampling or an independent verifier.

4. Engineering Practices for Reducing Hallucinations

4.1 Add Retrieval and External Evidence

For questions that are time-sensitive, highly specialized, or high risk, the model should first access filtered external materials and then answer based on those materials. The retrieval system must consider document quality, version validity, access control, and recall accuracy.

A reliable workflow often includes:

1. Rewrite the user’s question and identify the intent.
2. Retrieve relevant content from a knowledge base or document repository.
3. Deduplicate, rank, and apply permission filters to candidate passages.
4. Provide the evidence passages and their source information to the model.
5. Require the model to answer only on the basis of the evidence.
6. Perform a second verification of key conclusions.

Retrieval augmentation does not guarantee correctness automatically. If the retrieved materials are irrelevant or outdated, the model may still produce an incorrect conclusion. Retrieval quality and generation quality must therefore be governed together.

4.2 Require Evidence and Distinguish Levels of Certainty

Prompt constraints or application-layer designs can require the model to separate its response into:

• Facts explicitly supported by the source
• Reasonable inferences based on those facts
• Information that cannot be confirmed from the available materials

The model can also be required to identify the evidence location for each key conclusion. This makes verification easier for users and reduces the likelihood that speculation will be presented as fact.

When the available information is insufficient, the model should be allowed to say that the evidence is not enough to determine the answer, that the materials do not mention the subject, or that more information is required. It should not be forced to generate a specific answer.

4.3 Use Structured Output to Limit Unconstrained Generation

For entity extraction, field completion, classification, and data transformation, fixed fields, enumerated values, and format validation should be used whenever possible. This reduces the model’s ability to add unconstrained content in natural language.

For example, an application may require the model to return:

{
  "conclusion": "...",
  "evidence": ["..."],
  "confidence_level": "high/medium/low",
  "unconfirmed_items": ["..."]
}

The application can then validate field completeness, data types, value ranges, and whether evidence is present. Structured output does not eliminate factual errors, but it makes anomalies easier to detect and helps prevent noncompliant content from entering downstream systems.

4.4 Apply Risk-Based Review to High-Risk Content

Different tasks should not use the same tolerance for errors. Organizations can establish different handling policies by risk level:

• Low-risk content: Generate automatically and perform basic format checks.
• Medium-risk content: Add factual sampling and source validation.
• High-risk content: Bind the response to authoritative materials and require human review.
• Extremely high-risk operations: Use the model only to organize information; do not allow it to make the final decision directly.

The system should also record the user’s question, retrieved evidence, model output, validation results, and human revisions. This creates a traceable chain for locating the source of errors.

4.5 Build a Continuous Evaluation Dataset

Hallucination governance is not a one-time parameter-tuning exercise. Organizations should build evaluation datasets based on real business scenarios, covering:

• Common questions
• Edge cases
• Questions with false premises
• Multi-turn conversations
• Long-document questions
• Questions that require refusal
• Time-sensitive questions

Useful metrics include factual accuracy, source support rate, conflict rate, unverifiable rate, appropriateness of refusal, and citation completeness.

In addition to overall averages, results should be reported by domain, question type, and risk level. Otherwise, average metrics may conceal serious weaknesses in specific categories.

5. How to Understand LLM Hallucinations Correctly

LLM hallucinations are not caused by a single algorithmic defect. They result from the combined effects of generation mechanisms, data quality, parametric memory, training objectives, context processing, and application workflows.

Reducing hallucinations therefore requires more than switching models or lowering sampling randomness. A reliable system should combine data governance, retrieval augmentation, prompt constraints, structured output, automated evaluation, and human review.

For developers, the most important principle is this:

Treat an LLM as a probabilistic generator with reasoning and language capabilities, not as an inherently reliable fact database.

In low-risk scenarios, the model’s flexibility can improve efficiency. In high-risk scenarios, the system must provide verifiable evidence, clear boundaries, and mechanisms for handling failure.

An LLM becomes reliable enough for production only when the surrounding system can answer three questions:

• Where did this conclusion come from?
• Which parts have not yet been confirmed?
• How can an error be traced and corrected?