LLM Post-Training and Hallucination Rate Management: Engineering Practices
This article discusses how to evaluate post-training gains separately from pretraining knowledge, interpret benchmark scores with statistical caution, define and measure different types of hallucination, and apply practical engineering controls such as retrieval augmentation, refusal handling, structured output validation, executable checks, and multi-sample consistency. It also outlines how to build an internal regression evaluation process and provides a checklist for model selection and version rollout.
1. Why Post-Training Deserves Separate Attention
Large language model capabilities are usually formed in two stages. During pretraining, the model learns language patterns and general knowledge from massive text corpora, establishing its knowledge foundation. Post-training includes instruction tuning, preference alignment, reinforcement learning from feedback, and related methods. It determines how the model understands user intent, organizes responses, decides when to call tools, and decides when to refuse a request.
Adding post-training compute on top of the same base model often produces gains in three areas: task completion rate, output format stability, and tool-calling accuracy. These improvements are directly useful to developers because they can reduce parsing failures and retry counts. However, post-training does not change the knowledge ceiling of the base model. Evaluation therefore needs to distinguish between “can perform the task” and “knows the relevant facts.”
Post-training also has a side effect that is often overlooked. Overly strong alignment can make a model excessively conservative on boundary cases, while insufficient alignment can increase harmful or fabricated content. The effect of post-training must therefore be validated in specific business scenarios rather than judged only by general-purpose leaderboards.
2. Reading the Numbers in Model Evaluations
Public model evaluation reports usually provide a set of percentage scores. When reading these numbers, first confirm whether the following conditions are consistent:
- Evaluation set version and number of questions
- Prompt template and system prompt
- Sampling temperature, maximum output length, and retry count
- Tool permissions and execution environment
- Whether multiple samples are taken and the best result is selected
If any of these differ, the scores cannot be compared directly.
The size of a gap also requires a statistical perspective. For a proportion metric, the standard error can be estimated with the following formula:
SE = sqrt(p × (1 − p) / n)
Here, p is the pass rate and n is the number of samples. Suppose an evaluation set has 500 questions and the pass rate is about 65%. The standard error is approximately sqrt(0.65 × 0.35 / 500) ≈ 0.021, or about 2.1 percentage points. At this scale, a difference of less than 1 percentage point may well fall within random variation. This is an illustrative calculation; actual judgment should use the real question count and confidence interval.
3. Hallucination Rate: Define First, Then Measure
Hallucination refers to model output that is inconsistent with verifiable facts or with the given context. In engineering scenarios, common types include:
- Factual errors: incorrect times, numbers, names, locations, and similar details.
- Faithfulness errors: output that contradicts provided documents, retrieval results, or conversation history.
- Fabricated entities: invented citations, function names, parameters, or configuration items that do not exist.
- Over-inference: presenting a definitive conclusion when the evidence is insufficient.
Different types require different judgment methods. There are four common measurement approaches:
- Human annotation: accurate but costly, suitable for calibrating a baseline.
- Reference answer matching: suitable for closed questions with standard answers.
- Model judging: using another model to judge according to rules, with periodic human sample validation of its consistency.
- Executable validation: for code, query statements, and similar artifacts, running them directly and comparing results.
Below is the skeleton of an evaluation framework for calculating the hallucination rate over a batch of samples. The judge function needs to be implemented by the team according to its own tasks:
from dataclasses import dataclassfrom typing import Callable@dataclassclass Sample:question: strcontext: stranswer: strdef hallucination_rate(samples: list[Sample],judge: Callable[[Sample], bool],) -> tuple[float, int]:"""judge returns True if the answer contains a hallucination."""if not samples:raise ValueError("samples must not be empty")flagged = sum(1 for s in samples if judge(s))return flagged / len(samples), flagged
This code only defines the counting logic and does not depend on any specific service. The result should be recorded together with the sample count and judgment method; otherwise, results from different periods cannot be compared.
4. Engineering Approaches to Reduce Hallucination
Hallucination cannot be eliminated through a single method. It usually requires a combination of the following approaches:
- Retrieval augmentation: provide trusted materials as context and require answers to cite specific passages. Material quality determines the upper bound of effectiveness, and when retrieval finds nothing, the model should state that clearly.
- Allow refusal: define in the prompt how to respond when information is insufficient, and treat reasonable refusal as correct behavior. This avoids fabrication in an attempt to appear complete.
- Structured output: require the model to output a fixed JSON structure and then validate it against a schema. Format errors can be caught directly, reducing downstream misuse.
- Executable validation: for code, SQL, configuration files, and similar artifacts, run them in a sandbox or perform static checks, using execution results to judge correctness.
- Multi-sample consistency: sample multiple times for the same question, and route cases with large disagreement to human review or refusal.
The example below demonstrates structured output validation using only the standard library:
import jsonREQUIRED_KEYS = {"answer", "evidence_ids", "confidence"}def validate_output(raw: str, allowed_evidence: set[str]) -> dict:try:data = json.loads(raw)except json.JSONDecodeError as e:raise ValueError(f"Output is not valid JSON: {e}") from emissing = REQUIRED_KEYS - data.keys()if missing:raise ValueError(f"Missing fields: {sorted(missing)}")unknown = set(data["evidence_ids"]) - allowed_evidenceif unknown:raise ValueError(f"References unknown evidence: {sorted(unknown)}")return data
Samples that fail validation should enter a retry or fallback process and be logged for later analysis. Citing evidence that does not exist is a typical fabrication behavior; this kind of check is inexpensive to implement and provides clear benefits.
5. Building Internal Regression Evaluation
General-purpose leaderboards only show how a model performs on public tasks. Before a business launch, an internal regression evaluation is needed. It is recommended to build one through the following steps:
- Build an evaluation set: sample from real business logs, remove sensitive information, stratify by task type and difficulty, and keep enough samples in each stratum.
- Fix the evaluation configuration: put prompts, temperature, random seed, and maximum length under version control. Any change is treated as a new version, and results are recorded separately.
- Define metrics: in addition to hallucination rate, track task success rate, average latency, cost per call, and refusal rate. Focusing on only one metric can easily lead to unbalanced optimization.
- Report uncertainty: attach sample counts and confidence intervals to each metric, avoiding the mistake of treating random variation as progress.
- Canary rollout and rollback: run new versions with a small amount of traffic first and compare online metrics. If anomalies appear, switch back to the previous version quickly.
The evaluation set should be updated regularly. Changes in the business and shifts in the distribution of user questions will gradually make an old evaluation set less representative.
6. Checklist for Model Selection
Before introducing a new model or new version, check the following items one by one:
- Whether public evaluation conditions match your own scenario
- Whether the definition and judgment method of hallucination rate are public and reproducible
- Whether metric gaps exceed statistical error
- Whether performance on your own evaluation set is consistent in direction with public data
- Whether context length, multimodal capability, and response latency meet actual load requirements
- Whether data processing and privacy terms comply with enterprise compliance requirements
- Whether a fallback plan for failures is already in place
The answers to these questions are more informative about whether a model fits the current project than a single score.
