LLM Hallucination Governance: Evaluation Metrics and Practical Model Selection

    This article explains why public leaderboard scores are insufficient for model selection and presents a practical framework for LLM hallucination governance. It breaks model capabilities into measurable dimensions, defines how to measure hallucination rate with confidence intervals and refusal rates, and outlines reproducible evaluation workflows. It also shows how to turn evaluation results into deployment decisions, avoid common pitfalls, and use a pre-production checklist.

    1. Why a Single Leaderboard Score Is Not Enough for Decisions

    Public leaderboards provide a rough reference, but they rarely answer the question that matters most: can this model work in my business? There are three main reasons.

    First, benchmark task distributions usually differ from business distributions. A model that performs well on code repair tasks may not reliably summarize customer support tickets or contract clauses.

    Second, score gaps may be smaller than measurement error. If two models differ by one or two percentage points and the evaluation set contains only a few hundred samples, their confidence intervals may already overlap. In that case, ranking has no statistical meaning.

    Third, evaluation settings significantly affect results. Prompt templates, sampling temperature, available tools, retry counts, and system prompt length all change outputs. If two reports use different settings, directly comparing their scores has no solid basis.

    A reasonable approach is therefore to use leaderboards as a filter for the candidate pool, while final decisions still depend on evaluation in the target business scenario.

    2. Break “Capability” into Measurable Dimensions

    Asking broadly whether a model is “good” is not actionable. A more effective approach is to break capability into several dimensions, each with a clear test set and judgment rule.

    Task completion ability: Measures whether the model can complete multi-step tasks according to instructions, such as generating structured data from documents or performing a sequence of operations within a constrained tool set. The judgment criterion is usually whether the task is achieved, not whether the answer is fluent.

    Factual accuracy: Measures the correctness of verifiable statements in the output. This is the core of hallucination evaluation.

    Long-context stability: A larger context window does not mean the model can accurately retrieve information at any position in the window. Key information should be placed at different lengths and positions to observe how recall changes with length and position.

    Multimodal understanding: If the business involves mixed image-text input, image question-answering samples should be built separately to check whether the model correctly reads chart values, text, and spatial relationships.

    Cost and latency: This dimension is often overlooked. Mixture-of-experts (MoE) models have a large total parameter count, but only a subset of parameters is activated during each inference. Therefore, total parameter count does not directly equal inference cost or the capability ceiling. Model selection should consider active parameter scale, time to first token, throughput, and cost per call together.

    3. How to Measure Hallucination Rate

    Hallucination rate is usually defined as the proportion of verifiable answers that contain statements conflicting with facts or lacking support. This definition looks simple, but several details are easy to get wrong in practice.

    1. Use a consistent denominator. It can be calculated by answer count, by statement count, or by total sample count. With different denominators, the same model may produce very different numbers. Reports must state the denominator.

    2. Handle refusals separately. When a model is uncertain and refuses to answer, this is reasonable behavior and should not be counted as hallucination. However, if the refusal rate is high, the model loses practical value. It is therefore recommended to report hallucination rate, refusal rate, and valid answer rate together.

    3. Use stratified sampling. The evaluation set should cover different domains, difficulty levels, and question forms, with samples allocated according to business traffic proportions. Conclusions from a single-domain test set cannot be extrapolated.

    4. Sample size determines the reliability of conclusions. Using a normal approximation, when the sample size is 400 and the observed hallucination rate is 50%, the 95% confidence interval is approximately plus or minus 4.9 percentage points. The smaller the sample, the wider the interval, and the less meaningful small differences become.

    5. Use double-blind annotation and calculate agreement. Two or more annotators should judge independently, disagreements should be arbitrated by a third party, and inter-annotator agreement should be recorded so that the standard can be reproduced by others.

    Below is an example that calculates the hallucination rate and provides a confidence interval. This assumes that each sample has already been judged through the annotation process.

    1. import math
    2. from dataclasses import dataclass
    3. @dataclass
    4. class Judgement:
    5. sample_id: str
    6. hallucinated: bool # The answer contains a statement that conflicts with facts or lacks support
    7. refused: bool # The model reasonably refused to answer
    8. def hallucination_stats(judgements: list[Judgement], z: float = 1.96) -> dict:
    9. """Compute the hallucination rate and an approximate confidence interval. Refused samples are excluded from the denominator, and the refusal rate is reported separately."""
    10. total = len(judgements)
    11. refused = sum(j.refused for j in judgements)
    12. answered = [j for j in judgements if not j.refused]
    13. n = len(answered)
    14. if n == 0:
    15. raise ValueError("No valid answer samples; cannot calculate hallucination rate")
    16. k = sum(j.hallucinated for j in answered)
    17. p = k / n
    18. margin = z * math.sqrt(p * (1 - p) / n)
    19. return {
    20. "hallucination_rate": round(p, 4),
    21. "ci_low": round(max(0.0, p - margin), 4),
    22. "ci_high": round(min(1.0, p + margin), 4),
    23. "answered": n,
    24. "refusal_rate": round(refused / total, 4),
    25. }

    The output should include the hallucination rate, confidence interval, number of valid answers, and refusal rate. A percentage alone, without sample size and interval, does not let readers judge how credible the conclusion is.

    4. Build a Reproducible Evaluation Workflow

    Evaluation results have decision value only if they can be reproduced. A reliable workflow usually includes the following steps.

    • Pin versions: The evaluation set, scoring rules, and judgment guidelines should all have version numbers, and the reason for each change should be recorded.
    • Pin parameters: Decoding parameters such as sampling temperature, maximum output length, and stop words should all be written into configuration files and archived with the results.
    • Archive prompts: Fully save system prompts, user prompts, and examples. Any change should be treated as a new experiment.
    • Run multiple times and take mean and variance: Run the same setting at least three times and observe result fluctuations. Samples with large fluctuations often indicate ambiguity in the question itself.
    • Archive results: Save raw outputs, judgment records, and statistical results for later traceability.

    5. Turn Evaluation Results into Decisions

    The ultimate purpose of evaluation is to support launch decisions. Here are three practical principles.

    Set thresholds by scenario. In high-risk scenarios such as healthcare, legal, and finance, a lower hallucination rate should be required, and answers should be forced to include verifiable sources. In low-risk scenarios such as creative assistance and brainstorming, a higher hallucination rate may be acceptable, but users should be clearly told that results need verification.

    Compare options instead of scoring in isolation. When comparing two options, check whether the confidence interval of the difference crosses zero. Suppose two options have hallucination rates of 8% and 10%, each using 400 valid samples. The difference is 2 percentage points, and its 95% confidence interval is approximately -2.0 to +6.0 percentage points. Because the interval includes zero, the former cannot be considered better at this point; more samples or an improved evaluation set are needed.

    Launch gradually and feed samples back. An evaluation set cannot cover all real traffic. After launch, sample and record user feedback, add newly discovered failure cases to the evaluation set, and form a continuous iteration loop.

    6. Common Pitfalls

    • Treating parameter scale as proof of capability. Parameter count only indicates the model’s capacity ceiling, not its performance on a specific task.
    • Looking only at average scores and ignoring tail failures. Average accuracy may be high, but concentrated errors in a specific user group or scenario can create greater business risk. Statistics should be calculated by slice, and the worst-performing group should be checked.
    • Overlap between evaluation set and training data. Public questions may already appear in training data, inflating scores. Building an internal evaluation set and rotating questions regularly can reduce this risk.
    • Ignoring prompt sensitivity. The same question phrased differently may produce clearly different results. During evaluation, prepare multiple equivalent prompts and check whether results remain stable.

    7. Production Checklist

    Before introducing a model into production, confirm the following items one by one: whether the evaluation set covers the main business scenarios; whether the hallucination rate is reported with sample size, confidence interval, and refusal rate; whether comparisons between two options are based on statistical tests; whether decoding parameters and prompts have been archived; and whether there is a feedback loop after launch.

    Once these checks are complete, model selection conclusions become explainable, and the team can quickly determine whether a new model version is worth replacing the old one after an update.