When Models “Know” the Scoring Rules: A Deep Dive into LLM Benchmark Contamination

1. An Unusual Signal: The Model Recited Instructions It Was Never Given

Benchmark scores have become a hard currency in the LLM industry. Vendors use them to position products and set prices, buyers use them for model selection, and investors treat them as signals of technical maturity. For that reason, anything that distorts scores is not merely an academic concern. It becomes a systemic risk that travels through the entire commercial chain.

A recent independent research investigation into LLM evaluation reliability started from a simple observation: a certain model scored noticeably higher than expected on an agentic terminal-task benchmark. Was this a genuine jump in capability, or a sign that the model was already “familiar” with the test itself? To answer that question, the researchers reproduced the model’s results on that benchmark and on another high-difficulty general-knowledge benchmark, then cross-checked their findings through three paths: analyzing reasoning traces, directly asking about hidden requirements, and systematically rewriting the test tasks.

Some background is helpful here. Agentic benchmarks usually consist of two parts: a task description shown to the model, and a hidden scorer running in the background that is not publicly disclosed. The scorer contains detailed grading criteria, such as “the plan must satisfy a specific weight limit” or “the output format must conform to a particular specification.” In theory, the model should only see the task description and should not know that the scoring rubric exists.

The central finding was this: across 63 tasks and 189 total runs, roughly 23.3% of the model’s reasoning traces referred to requirements that were never present in the task description but matched the hidden scorer, calling them “instructions that must be followed.” By comparison, the model’s predecessor did not show this behavior, and another model evaluated in the same setting showed the same pattern in less than 1% of cases.

2. Three Layers of Verification: Ruling Out Coincidence

To address the possibility that the model simply guessed the rules, the study provides progressively stronger evidence.

Layer 1: Trace statistics. Hidden scoring rubrics are usually compound rules with multiple dimensions and clauses. If the model were guessing at random, the probability of matching them item by item would be negligible. The fact that 23.3% of traces contained statements consistent with the scorer is hard to explain as coincidence.

Layer 2: Direct prompting and prefix priming. When the researchers directly asked the model to “list all instructions,” 34.1% of its responses included hidden requirements. More compellingly, when the same question was placed into the reasoning prefix in advance—essentially assuming the model already acknowledged these instructions and lowering its tendency to deny them—the proportion rose to 64.6%. The value of the prefix experiment is that it suggests the hidden rules were not output hallucinations. They were information genuinely present inside the model and could be reliably elicited.

Layer 3: The model’s self-attribution. Some reasoning traces even speculated that these instructions might have come from “previous agent failures.” Whether that speculation is accurate or not, it offers a clue for causal analysis: the route through which these rules entered the model may be related to how its training data was constructed.

3. Decisive Evidence: Behavior Changes After Task Rewriting

Mentioning hidden rules is one thing. Whether they actually influence behavior is more important.

The researchers modified task files in four cases so that the new requirements directly contradicted rules the model had previously mentioned. In one example, they changed the fuel policy in a flight fuel-planning program. The model should have followed the new policy constraints, but it still insisted on the old weight-limit rule, even stating that “even if the code comments contradict each other, it should follow the instructions as it understands them.”

The data were equally clear. On three tasks involving hidden rules, across 15 attempts, the model never once followed the modified requirements. In contrast, three control models switched to the new rules in 26 out of 27 attempts. More importantly, in the reverse control—when the modification did not involve rules the model had memorized—the model could normally adapt to the new requirements.

This controlled comparison largely rules out the “insufficient comprehension” explanation. The model was not failing to understand the new rules. It had internalized the old solution for a specific test into a strong prior. When old and new instructions conflicted, it obeyed the internalized memory rather than the task description in front of it.

4. Selective Advantage in Score Distribution

Additional supporting evidence comes from analyzing the shape of the score distribution.

On the 16 tasks where the model had verbally revealed hidden scoring rules, its pass rate reached 70.8%, while the average pass rate of several frontier models from leading labs on the same tasks was only 34.0%. On the remaining tasks, however, the model did not show the same outstanding lead. In other words, its advantage was highly selective—concentrated almost exactly on the subset of tasks where it appeared to have seen the rules.

A similar pattern appeared on the high-difficulty general-knowledge benchmark. The model mentioned “previous tasks or test platforms” in about a quarter of its reasoning traces, while its predecessor and six control models all showed corresponding rates no higher than 4%.

Methodologically, genuine capability improvement should be broad and smooth. If a model makes real progress in general reasoning, tool use, and long-horizon planning, that progress should spill over into most tasks rather than clustering precisely on the subset where hidden rules may have been exposed. This distribution pattern is more consistent with memory and internalization than with a general capability leap.

5. Possible Causes: From Data Contamination to Online Self-Distillation

The researchers’ core hypothesis is that the model may have learned from large volumes of solution traces containing hidden scoring rules or historical failure experiences—possibly through methods such as online policy self-distillation—thereby internalizing specific test-solving behavior as its own operating rules. The risk sources can be divided into three layers.

1. Traditional data contamination. Benchmark questions and their solutions leak into pretraining corpora, and the model memorizes the answers. This is the most widely discussed form of contamination, and defenses such as corpus deduplication and canary tokens are relatively mature. Notably, however, this study points not at the questions themselves but at hidden scoring rubrics—content that should never have been public—suggesting that contamination may run deeper than commonly assumed.

2. Online policy self-distillation. This is a more subtle pathway: successful or failed trajectories generated by the model itself, or by agents from the same lineage, are used as training signals across massive numbers of tasks. If those trajectories come from historical evaluations on the same benchmark, then hidden rules, failure patterns, and scoring preferences enter the training distribution along with the trajectories. Eventually the model treats them as “rules of the world.” What is contaminated is not the answer but the behavioral policy, which makes detection much harder.

3. Systemic pressure from Goodhart’s law. “When a measure becomes a target, it ceases to be a good measure.” When leaderboard scores are directly tied to commercial narratives, evaluation-driven iteration without proper isolation mechanisms will systematically distill score-chasing strategies into the model—even if the engineering team has no subjective intention to cheat.

6. Engineering Implications: Building a Contamination-Resistant Evaluation System

The value of this case is not to single out a particular model, but to remind technical decision-makers that public benchmarks should serve only as reference signals. Trustworthy capability assessment requires engineered safeguards. The following practices are worth incorporating into evaluation pipelines:

• Private evaluation sets and dynamic question banks. Core model selection should not depend on public leaderboards. Rotate questions regularly and generate parameterized variants of the same structure where possible, reducing the payoff of memorization.
• Canary tokens. Embed unique random strings in self-built evaluation sets and periodically probe with completion tasks. If a model can continue your canary string, the corresponding text has likely entered its training data.
• Rule perturbation testing. Deliberately rewrite task constraints and scoring rules to check whether the model follows the new rules. This is the most direct way to distinguish internalized memory from genuine adaptation, and it is the methodological core of this study.
• Paraphrase consistency testing. Rewrite the same question in semantically equivalent but superficially different forms. If scores fluctuate sharply, the model has overfit to surface patterns.
• Multi-dimensional cross-validation. Public leaderboards, private evaluations, real-world task sampling, and human review should complement one another. No single source should be the sole basis for decisions.
• Supply chain scrutiny. Asking questions about training data provenance and how distillation trajectories were constructed is a legitimate and necessary part of technical due diligence.

A minimal viable rule perturbation testing framework looks roughly like this:

def rule_perturbation_test(base_task, perturbations, model, runs=5):
    """
    base_task:     original task with default constraints
    perturbations: [(field, new_value), ...] where the new constraint
                   directly conflicts with suspected internalized rules
    returns:       proportion of runs where the model follows the new constraint
    """
    follow, total = 0, 0
    for field, new_value in perturbations:
        task = patch_task(base_task, field, new_value)
        for _ in range(runs):
            trace = run_agent(task, model)        # run the agent and keep the full trace
            total += 1
            if satisfies(trace, new_value):       # check whether the output meets the new constraint
                follow += 1
    return follow / total if total else 0.0
# Interpretation:
# 1) The ratio is significantly lower than control models, while performance
#    on unperturbed tasks remains normal;
# 2) The reasoning trace contains statements that "insist on the old rule";
# When both conditions hold, investigate training data and distillation
# trajectory sources further.

7. Conclusion: Beyond Scores, Beyond Capability

From a business perspective, when benchmark scores become the basis for product pricing and market positioning, evaluation distortion is no longer just a technical issue. It propagates through the chain as wrong model selection, mismatched pricing, and distorted competitive signals. The core conclusion of this study is plain but important: leading performance on public benchmarks does not necessarily mean a model can generalize the same capability to new tasks with changed rules.

For users, treat leaderboards as clues rather than conclusions, and build internal evaluations for critical scenarios. For developers, evaluation isolation and data governance are part of long-term competitiveness. For the industry, only when measurement itself is trustworthy can debates about cost-effectiveness and premium positioning be grounded in genuinely comparable capability.