Why LLM Benchmark Scores Conflict: Harness Fitting and Benchmark Contamination Explained

1. The Phenomenon: One Model, Two Verdicts

After a recent flagship model launch, an oddly dramatic situation emerged. In the vendor’s official, purpose-built tests, the model set records on agentic execution, scientific coding, mathematical reasoning, and other tasks, ranking near the top overall. Yet on a third-party composite index known for its fixed evaluation pipeline, the overall score not only failed to rise but actually dropped—falling two to three points on subcategories such as professional knowledge work, industry customer-service agents, and long-context multi-document reasoning. It was “inverted” by the previous-generation model and even by models from other vendors.

Read only one of those reports, and you would reach the opposite conclusion. This kind of evaluation “Rashomon” is not rare in the industry. Behind it is not simply a question of who is hedging or who is protecting their own, but a concentrated expression of three structural problems in LLM evaluation methodology: strong fitting between evaluation harnesses and scores, the mismatch between long-horizon capabilities and static benchmarks, and the trade-off between benchmark contamination and decontamination. Understanding these three points is more valuable than arguing over which score is more real.

2. Harness Fitting: Are You Testing the Model, or the “Model + Pipeline”?

2.1 What Is a Harness?

A harness is the entire evaluation pipeline wrapped around a model. It typically includes prompt templates and how few-shot examples are injected, sampling parameters (temperature, top_p, max generation length), tool-calling protocols and failure-retry strategies, truncation rules when context overflows, and answer extraction and scoring rules. The same problem set, run through different harnesses, can produce strikingly different scores.

2.2 A Typical Case of Strong Fitting

The industry has seen cases where the same model differed by dozens of percentage points on the same benchmark: near-perfect under a vendor’s dedicated pipeline, but significantly degraded under a third-party general-purpose harness. The reason is that agent-style tasks depend heavily on the interaction protocol between model and environment—how tools are exposed, how failures are retried, how context is truncated. Every step is a degree of freedom. The more thoroughly a model is tuned to a specific pipeline, the higher its score fits that pipeline—and the more it may fall apart in a different environment.

2.3 What a Fixed-Harness Composite Index Really Means

To be fair, third-party composite indices usually apply the same fixed harness to all models. This has two consequences:

• For vendors: the advantages a model enjoys under its own tuned pipeline do not transfer. The index measures the combined capability of “model + that index’s harness.”
• For readers: the index score reflects a “lower bound in a standard environment,” not the model’s upper bound under optimal engineering configuration.

Schematically, a harness is full of decision points that affect scores:

# Key decision points in an evaluation harness (pseudocode)
def run_case(model, task):
    prompt = build_prompt(task, fewshot=FEWSHOT_POLICY)  # Example injection policy
    for attempt in range(MAX_RETRY):                     # Retry policy
        resp = model.generate(prompt, temperature=0.2)   # Sampling parameters
        if resp.exceeds_context:
            prompt = truncate(prompt, strategy="keep_head")  # Truncation policy
        if is_valid(parse_answer(resp)):
            return score(resp, task.gold)
    return 0.0

Any single change above is enough to cause meaningful score fluctuations for the same model on the same problem set. So when a model leads under its own harness and “regresses” under a fixed harness, both results can be true at the same time.

3. Long-Horizon and Cross-Context Capabilities: Dimensions Static Benchmarks Cannot Measure

3.1 Context Management Is Moving from “Compression” to “Externalized Memory”

Early long-task processing generally relied on context compression: when a conversation approaches the window limit, history is summarized and stuffed back into the window. This approach is simple to implement but continuously loses detail. In recent years, another route has emerged—hard window switching plus external memory. Instead of compressing during a window switch, the working state is externalized. Typical capabilities include:

• Maintaining separate cross-window working notes that accumulate task state across sessions;
• Continuing to save and carry forward accumulated state after a window switch;
• Retrieving original messages and tool outputs from earlier windows on demand;
• Backtracking to recover early test results and requirement details when the notes are not granular enough.

3.2 Why Fixed Indices Cannot Measure This

The vast majority of problems in a composite index can be completed within a single session. There is neither a long enough task span nor a need for state continuity. The value of cross-context memory only becomes visible in large-scale, multi-session, long-duration engineering tasks—for example, a repository-level refactor lasting hours, or a debugging process that requires repeatedly revisiting experimental results. Hence the mismatch: scores may “regress” on the long-context subcategory of a composite index while the model leads by a wide margin on a dedicated benchmark in a real agent environment.

For evaluators, this is a coverage blind spot in “what is being tested.” For users, it is a problem of “scores not covering the capabilities I care about.” Both are true and not contradictory.

4. Benchmark Contamination and the Cost of “Not Gaming the Score”

4.1 What Is Benchmark Contamination?

Benchmark contamination refers to evaluation problems (or close paraphrases of them) leaking into training data, causing a model to have “memorized the original questions” and inflating its score. Contamination systematically overestimates a model’s true generalization ability on public benchmarks and is one of the most closely watched problems in the evaluation community in recent years.

4.2 Common Detection and Defense Methods

• Embedding canary strings in public benchmarks and monitoring after release whether the model reproduces them;
• Applying semantics-preserving perturbations to problems (changing numbers, wording, or variable names) and comparing the gap between original and perturbed scores;
• Maintaining private holdout sets that are not public and used only for sampling verification;
• Rotating benchmark versions frequently to retire old problems that have spread too widely.

4.3 The Relationship Between “Decontamination” and Score Regression

If a vendor deliberately avoids targeted training or prompt tuning for public benchmarks, its scores on widely circulated benchmarks may actually be lower than those of competitors who “gamed the questions.” Specifically for the phenomenon discussed here: a drop of two to three points on individual subcategories of a composite index could mean a real capability regression, or it could mean no contamination-oriented optimization was done. A single index alone cannot distinguish the two; controlled-variable verification is required.

A practical heuristic: if a model regresses simultaneously on multiple independent benchmarks with different problem sources, a real capability regression is more likely. If it regresses only on a few older, widely circulated benchmarks while performing normally on new or private problems, contamination and score-gaming differences are more likely explanations.

5. How Practitioners Should Read Scores: Four Practical Recommendations

1. Cross-check multiple sources and locate the disagreements first. Put the vendor report, the third-party composite index, and individual benchmarks together, and mark the items where directions conflict. The disagreements themselves are information: they often correspond to harness-sensitive areas, capability mismatch zones, or contamination suspects, and are worth digging into item by item.

2. Break down by dimension; reject a single index. A composite index is a weighted average that hides structural differences. At minimum, separate four dimensions for evaluation: knowledge Q&A and professional reasoning, long-context understanding, agentic execution (tool calling and multi-step tasks), and code generation and engineering capability. Different businesses weight these four dimensions very differently.

3. Build your own business evaluation set with a fixed protocol. The most reliable scores always come from your own data. Sample from real business traffic to build an evaluation set, freeze the prompt templates and sampling parameters, and version the data:

# A minimal viable internal evaluation pipeline (pseudocode)
cases = load_dataset("biz_eval_v3", private=True)        # Private holdout set
results = []
for case in cases:
    traj = agent.run(case.instruction, tools=TOOLS)      # Fixed tool protocol
    results.append(judge(traj, case.gold, rubric=RUBRIC_V2))
report = aggregate(results, dims=["accuracy", "cost", "latency"])
store(report, backend="object_storage")                  # Archive results
alert_if(report.regression > 2.0, channel="monitoring")  # Regression alerting

In engineering terms, such a pipeline can be built directly on general-purpose cloud components: object storage to archive raw trajectories and results from each evaluation, a logging service to record model inputs and outputs for post-hoc auditing, and monitoring and alerting to automatically notify when a version’s regression exceeds the threshold. Whichever cloud platform is chosen (for example, a cloud provider such as Baidu AI Cloud that offers model services and supporting general-purpose components), the above pattern holds.

1. Operate evaluation like CI. Any change to the model version, prompt version, or harness version should trigger a round of regression evaluation. The point of a score is not “how high the absolute value is,” but “whether it is stable relative to the last baseline.”

6. Conclusion

A score is a measurement result, not the model itself. When different evaluation systems give contradictory conclusions, rather than asking “who is hedging,” it is more useful to treat it as an opportunity to understand evaluation methodology: every decision point in a harness, the contamination status of a benchmark, and an index’s coverage blind spots for long-horizon capabilities all make “the same model” present different faces. For developers and enterprise users, building an internal evaluation system based on your own business, with a fixed protocol and versionable regression, is far more valuable in the long run than chasing any single leaderboard.