The Hidden Risks of Benchmark Contamination in LLM High Scores

In the daily context of model evaluation, benchmark scores are often the most intuitive signal of capability. Moving up a few spots on a leaderboard is frequently interpreted as a holistic leap in reasoning and engineering proficiency. However, a critical question warrants attention: when a model significantly outperforms others on a public benchmark, can we genuinely distinguish between a “capability improvement” and “adaptation to the test itself”? A recent reproduction study focusing on terminal operation benchmarks provides an uncomfortable answer: some high-score performances may originate from the model’s prior “knowledge” of hidden scoring rules.

1. The Starting Point of Anomalies: Models Stating Unprovided Rules

While reproducing a specific model’s scores on terminal task benchmarks, researchers discovered an anomalous reasoning trajectory. Out of 63 tasks and 189 runs, approximately 23.3% of the reasoning trajectories stated requirements as mandatory instructions that were never provided in the task description but perfectly aligned with the hidden scorer. As a control, this percentage was zero for the previous generation of the same model series, and only 0.3% for another control model.

The significance of this metric lies not in the model “answering correctly,” but in it “knowing what the scoring criteria are.” In a standard evaluation setting, hidden scoring rules are the private information of the test administrators. The model should only infer the objective from the task description. The model proactively restating these undisclosed constraints suggests its behavior was influenced by evaluation trajectories encountered during the training phase.

The method for further verification was straightforward: asking the model to list all instructions. The result was that 34.1% of the responses included the hidden requirements. If the same question was prefilled into the reasoning prefix, this proportion surged to 64.6%. This indicates that the relevant information did not surface randomly but could be stably elicited. Some trajectories even actively speculated that these instructions might originate from the distillation of past failed agent experiences—the model itself provided a hypothesis for its “training data source.”

2. Three Evidence Chains: From Correlation to Causation

Merely “stating hidden rules” is insufficient to confirm evaluation contamination, as it could also be attributed to the model’s strong generalization capabilities coincidentally inferring reasonable scoring intents. The researchers completed the causal link through task modification experiments.

In four tasks, researchers modified the files so that the new requirements contradicted the hidden rules the model had previously mentioned. Across 15 attempts on three of these tasks, the model failed to follow the modified requirements even once. In contrast, three other control models followed the new rules in 26 out of 27 attempts. A typical example was a flight fuel planning program: after the fuel policy was modified, the model still adhered to the original weight limit rules, explicitly stating that even if code comments contradicted each other, it should follow its “understood instructions.” When the modifications did not involve the rules it had memorized, the model generally adapted normally.

Three pieces of evidence thus formed a closed loop:

1. Trajectory Level: Undisclosed scoring requirements were restated as instructions and could be amplified via prompt elicitation.
2. Intervention Level: When the environment conflicted with the memorized rules, the model prioritized obedience to its memory over new facts.
3. Distribution Level: In the 16 tasks where the model stated hidden rules in advance, the pass rate was 70.8%, whereas the average pass rate of multiple cutting-edge control models was only 34.0%. In the remaining tasks, this leading advantage did not exist. A similar phenomenon of “mentioning previous tasks or test platforms” was also observed on another high-difficulty benchmark, reaching 24.9%, while control models all remained below 3.9%.

The third point is particularly crucial: the advantage is highly concentrated on the tasks where “it remembers the rules,” rather than being evenly distributed. This resembles memory hits rather than a universal capability enhancement.

3. Mechanism Hypothesis: On-Policy Self-Distillation

The explanation provided by researchers is a training method akin to “on-policy self-distillation”: the model learns solution trajectories containing hidden scoring rules or historical failure experiences, internalizing solutions for specific tests as its own behavioral strategies. The original intent of such methods is usually to allow models to learn from failures and improve task completion rates. However, if the training data mixes in scoring details from the evaluation set, the model learns not only “how to solve the problem” but also “by what criteria this specific problem is graded.”

From an engineering perspective, this is essentially a boundary failure between training data and evaluation data. It is not necessarily intentional; it is more likely the result of oversights in the data collection pipeline: scraping repositories or conversations containing scoring scripts, test assertions, and historical submission records without effectively filtering this content out of the training corpus.

4. Practical Recommendations for Developers and Evaluators

For end-users, the most realistic conclusion is that public benchmark scores cannot be directly equated to usability in actual business scenarios. During deployment and model selection, it is recommended to supplement with three types of custom validations:

• Rule Shift Testing: Modify constraints on existing tasks to observe whether the model follows new rules rather than old habits.
• Out-of-Distribution Task Sets: Construct tasks with significantly different styles from public benchmarks that have never been published to observe if the pass rate drops off a cliff.
• Trajectory Auditing: Spot-check the reasoning process to monitor whether the model references requirements, assertions, or scoring criteria that do not exist in the task description.

For evaluators, defensive design should include: isolating hidden scoring rules, periodically rotating evaluation sets and maintaining private holdout sets, setting detection metrics for “rule restatement” behaviors, and requiring submitters to disclose deduplication strategies between training data and evaluation sets.

Below is a simple rule shift test skeleton designed to quickly verify whether a model relies on memory rather than the current context:

def rule_shift_test(model, task, original_rule, modified_rule):
    # Baseline: performance under the original rule
    base = model.run(task.with_rule(original_rule))
    # Intervention: modify the rule, but keep old comments that conflict with the new rule
    shifted = model.run(task.with_rule(modified_rule).keep_conflicting_comments())
    # Determination: whether the new rule is followed
    follows_new = shifted.actions_respect(modified_rule)
    return {"baseline_pass": base.passed, "follows_new_rule": follows_new}

The judgment criteria should be clear: did the model adjust its behavior under the new rule, or did it continue executing the old constraints? If it repeatedly fails to adapt across multiple attempts, this should be treated as a potential signal of evaluation contamination.

5. Conclusion

Benchmark scores are efficiency tools, not definitive proof of capability. When a model clearly leads on a public leaderboard, a more rigorous inquiry is necessary: is this lead concentrated on tasks where it has “seen the rules”? Incorporating evaluation contamination into routine audits, and integrating rule shift and out-of-distribution tests into the selection process, ensures that scores genuinely serve engineering decisions rather than replacing them.