Demystifying LLM Evaluation Discrepancies: How to Objectively Assess Model Capabilities

I. The Phenomenon: A Typical “Caliber Divergence”

Recently, a vendor released a new open-source Large Language Model (LLM), confidently claiming it had “joined the ranks of the strongest open-source models for the first time.” However, subsequent scorecards from third-party evaluation agencies told a different story: its overall intelligence index was lower than another participating model, and its hallucination rate was flagged as “high.”

The gap between confident official claims and conservative third-party assessments is not new; almost every model release plays out a similar scenario. It is worth noting that such discrepancies do not necessarily mean one side is inflating numbers or suppressing the model. A more common reason is that official tests, third-party leaderboards, and user experiences are inherently three different measurement systems. Rather than obsessing over a single score, it is more productive to understand why they misalign—this is the first lesson for engineering teams building their own evaluation frameworks.

II. Four Technical Root Causes of Score Misalignment

1. The “Overall Intelligence Index” is an Art of Weighting

Comprehensive indices are almost always the product of “score by category, then apply weights”: dimensions like knowledge QA, mathematical reasoning, code generation, instruction following, and long-context understanding are scored individually, then synthesized into a total score using a set of weights. The problem is that adjusting a weight from 10% to 25% can completely reverse the ranking. Vendors often cite the task combinations and weighting schemes most favorable to their own models, while third-party institutions have their own fixed templates. These two weighting systems naturally yield different conclusions.

2. Evaluation Implementation Details Can “Eat Up” Several Points

It is an industry consensus that the same benchmark run on different evaluation frameworks can yield scores differing by several percentage points. Implementation details affecting results include:

• Prompt Templates: How system prompts are written, the number of few-shot examples, and how examples are selected significantly alter model performance.
• Decoding Parameters: Differences in temperature, top-p, and maximum generation length cause drift in stability and scoring.
• Answer Extraction and Scoring: Whether a math problem outputs “The answer is 42” or “\boxed{42}”, the regex used to extract multiple-choice options, or whether open-ended questions use rule-based matching or a judge model—any inconsistency here can cause “meaningless point loss.”

Therefore, two models’ scores on the “same leaderboard” might not have been measured under the “same implementation.”

3. Data Contamination and Benchmark Saturation

Public benchmark questions inevitably enter web corpora, meaning models might have “seen the exact questions” during training. In such cases, high scores reflect memorization rather than generalization. Conversely, when a benchmark is widely targeted and top models approach perfect scores, it loses its discriminative power. If everyone is scoring above 90, a difference of a few points might just be noise. Serious evaluations implement contamination detection (like n-gram overlap screening or perturbation re-testing), but not all leaderboards do this thoroughly.

4. Different Normalization and Aggregation Methods

An “80 score” could mean an “80% accuracy rate,” an “80% normalized score against a human baseline,” or an “80% normalized score against a random baseline.” Different normalization bases and aggregation sequences (averaging first then normalizing, vs. normalizing first then averaging) can lead to different rankings from the exact same raw data.

III. Correctly Understanding the “Hallucination Rate” Metric

Hallucination refers to the model generating content inconsistent with facts or provided evidence. Its measurement method inherently has a high degree of freedom: it can be based on knowledge base comparisons, retrieval evidence support levels, or manual sampling checks. Because measurement calibers differ, there is no universally recognized hard boundary between “high” and “medium.”

There are two easily overlooked technical facts:
First, a more capable generative model does not necessarily have a lower hallucination rate. Longer answers with more details and a confident tone mean there are more “checkable error points.” Measuring hallucinations with the same yardstick, a model giving conservative short answers naturally has an advantage over one providing extensive, long answers.

Second, the hallucination rate must be interpreted within its specific context. In creative writing or brainstorming scenarios, a certain degree of “divergence” can even be valuable; in medical, legal, or financial contexts, hallucinations are critical flaws. Discussing hallucination rates without context has limited engineering significance.

Standard engineering approaches to mitigate hallucinations include: Retrieval-Augmented Generation (RAG) to base answers on evidence, requiring citations and traceability, allowing the model to express uncertainty, and adding post-verification pipelines for high-risk answers. When evaluating hallucination rates, these mitigation strategies should be integrated into the overall assessment plan rather than just looking at the raw model’s score.

IV. Actionable Evaluation Methodology

1. Reverse-Engineer Capability Dimensions from Business Scenarios

Don’t ask “which model is the strongest”; ask “what capabilities does my business require.” Break down requirements into dimensions like instruction following, multi-turn dialogue, long context, code generation, structured output, and domain knowledge, defining acceptance criteria for each. Leaderboards provide preliminary screening clues but cannot replace this step.

2. Build a Private Evaluation Set

Sample, anonymize, and label data from real business logs, covering typical traffic, edge cases, and adversarial examples, while keeping the version fixed. A private evaluation set is a team’s most important “regression testing asset”: run it every time you switch models, upgrade versions, or modify prompts to prevent capability regression.

3. Multi-Signal Combination

• Automated Metrics: Objective questions look at accuracy; code looks at test case pass rates (e.g., pass@k); extraction tasks look at F1 score.
• Judge Models: Using strong models to score open-ended answers is efficient, but known issues like position bias and length bias must be controlled—for instance, by swapping answer order and re-testing.
• Manual Spot-Checks: Small-scale human evaluation calibrates automated metrics, especially before going live.

4. Statistical Rigor

Sample multiple times under the same configuration, report means and variances, and pay attention to whether differences are significant. A one or two-point difference likely falls within the confidence interval and is insufficient to support a conclusion of “who is stronger.”

Here is a minimal evaluation scaffolding illustration:

# Evaluation process illustration (pseudo-code)
CASES = [
    {"id": "001", "dim": "Math Reasoning", "prompt": "...", "ref": "42"},
    {"id": "002", "dim": "Code Generation", "prompt": "...", "tests": ["assert f(2)==4"]},
]
def extract_answer(text):
    """Unify answer extraction rules to avoid losing points due to format differences"""
    ...
def run_eval(model_fn, cases, n=3):
    scores = {}
    for case in cases:
        runs = []
        for _ in range(n):                 # Multiple samplings to estimate stability
            out = model_fn(case["prompt"], temperature=0.7)
            runs.append(score(case, extract_answer(out)))
        scores[case["id"]] = sum(runs) / n
    return report_by_dim(scores)           # Aggregate by capability dimension, not just total score

V. From Evaluation to Production: Engineering Perspectives

High offline evaluation scores do not guarantee good online performance. Latency, throughput, unit cost, context length limits, and deployment models (self-hosted vs. cloud API) will all influence the final decision. During the selection phase, it is recommended to compare all candidate models under a unified calling protocol: standardize requests, responses, and log formats through cloud-based model services (such as Baidu AI Cloud and similar platforms offering general LLM services), so switching models only requires modifying a single configuration item. Post-launch, use canary releases and A/B testing to validate real business metrics, and route online bad cases back to the evaluation set, forming a closed loop of “evaluation—deployment—feedback.”

VI. Conclusion

A single leaderboard score does not equate to model superiority; evaluation is a means, not an end. Faced with the coexistence of “official claims of top-tier performance” and “conservative third-party assessments,” the rational approach is to establish a three-tier information structure: use official data as a reference, third-party data for cross-validation, and your own private evaluation set as the final decision-making basis. Internalizing the evaluation system as a team’s engineering capability is the only way to consistently make selection decisions that withstand reproducibility tests in today’s rapidly iterating model landscape.