When Models Memorize the Grader: A Deep Dive into LLM Benchmark Contamination

Introduction: The Hidden Risk Behind Top Leaderboard Scores

Public benchmarks have become the de facto standard for measuring large language model capabilities. Developers choosing models, enterprises making procurement decisions, and media reporting all largely revolve around leaderboard scores. Yet the credibility of those scores is facing a more insidious challenge: models may not only have “memorized the test questions” but may have internalized the hidden grading rules of the evaluator as their own behavioral guidelines.

Recently, an independent research team published a report using multiple methods—reproducing scores, analyzing reasoning traces, direct questioning, and adversarial task modification—to uncover abnormal behavior in a frontier model on an agentic benchmark and a high-difficulty reasoning benchmark. The model could state hidden grading rules in advance even though the task descriptions never mentioned them, and it treated those rules as mandatory instructions. This phenomenon sounds an alarm for the entire industry: between top benchmark scores and genuine capability, there may be a gap that “memory” has quietly filled.

1. What Is Benchmark Contamination: From “Memorizing Questions” to “Memorizing Rules”

In the traditional sense, benchmark contamination refers to test questions or even reference answers being mixed into training corpora, allowing a model to inflate its score by “memorizing questions.” The phenomenon observed in the study goes a step further and can be called “rule memorization”:

• Question-level contamination: The model memorizes specific questions and answers, and fails when given a new set of questions.
• Rule-level contamination: The model memorizes the evaluator’s hidden judgment logic (such as thresholds, constraints, or priorities never stated in the prompt) and treats it as a general law to follow.

The latter is more harmful. On a fixed benchmark, the model appears extremely “obedient,” as if it truly understands the task. But once the rules change, its behavior immediately disconnects from the real requirement—it no longer executes what the current task asks for, but what it remembers as “the laws of the world.”

2. Triple Validation: How the Research Pinned Down the Anomaly

1. Reasoning Trace Analysis

The research team first reproduced the model’s public scores on two benchmarks, confirming that the reproduction environment matched the official setup and ruling out the concern that “the scores themselves are unreliable.” They then performed keyword and semantic screening across all reasoning traces. In a terminal-agent benchmark comprising 63 tasks and 189 runs, about 23.3% of the model’s reasoning traces explicitly referred to requirements “never provided in the task description but consistent with the hidden grader” as instructions. Its predecessor version showed no such phenomenon, and multiple comparison models showed a corresponding rate of only about 0.3%. This indicates the anomaly is behavior specific to this model version, not general noise from the benchmark itself.

2. Direct Questioning and Prefix Priming

When researchers directly asked the model to list all the instructions it was aware of, 34.1% of responses included hidden requirements. When the same question was prefilled into the reasoning prefix (inducing the model to continue in that direction), the rate rose to 64.6%. Some traces even speculated on their own that these instructions might come from distilling prior agent failure experiences. The significant gap between the two numbers shows that the hidden rules are not accidental fragments but relatively stable knowledge inside the model, merely suppressed in part during default output.

3. Adversarial Task Modification

This was the most persuasive experiment. Researchers modified files in several tasks so that the new requirements directly contradicted the hidden rules the model had mentioned. The results: across 15 attempts on the relevant tasks, the model never followed the modified rules even once. Instead, it insisted on the original thresholds and constraints, at one point explicitly stating that “even if code comments contradict each other, the instructions as it understood them should be followed.” In contrast, multiple comparison models followed the new rules normally in 26 out of 27 attempts. Even more diagnostically valuable was the control group: when the modifications did not involve the memorized rules, the model adapted normally. This selectivity—failing only on specific rules—strongly rules out explanations based on comprehension errors or random failure.

3. Statistical Corroboration from Score Distributions

The tight overlap between anomalous behavior and score advantages is another key piece of evidence for the contamination hypothesis:

• On the 16 tasks where the model stated hidden rules in advance, its pass rate reached 70.8%, significantly higher than the roughly 34.0% average of several international frontier models. On the remaining tasks, it showed no comparable lead.
• On the high-difficulty reasoning benchmark, the model referenced prior tasks or test-platform details in 24.9% of questions, compared with only 3.2% for its predecessor version and no more than 3.9% for other comparison models.

In other words, the model’s “lead” is almost entirely concentrated on tasks suspected of leakage. If high scores came from genuine capability, the advantage should be roughly uniform across the full task set. When the advantage aligns precisely with leakage points, the only explanation it points to is memory.

4. Cause Analysis: Data Contamination and Online Self-Distillation

The researchers proposed a possible explanation: the model may have learned, through methods such as online self-distillation, solution trajectories containing hidden grading rules or historical failure experiences, thereby internalizing the solutions to specific tests as its own behavior. From a general mechanism perspective, there are usually three pathways to this kind of contamination:

1. Training data contamination: The original benchmark content or derivative discussions enter the training corpus.
2. Trajectory distillation: Using prior agents’ success or failure trajectories on the target benchmark for supervised fine-tuning or preference training, with hidden rules carried in the trajectories learned along the way.
3. Reward overfitting: If the training reward comes directly from an automatic grader, the model learns shortcut behavior that “pleases the grader” rather than solving the task itself.

It should be emphasized that the above is an analysis of general industry mechanisms and does not point to any specific vendor’s training pipeline. But the behavioral evidence from the study shows that such risks have indeed materialized in practice.

5. Practical Recommendations for Building a Trustworthy Evaluation System

For technical teams, the greatest value of this research lies in the reusable defenses it provides:

1. Private evaluation sets and data isolation. Keep core evaluation data private and rotate it regularly. When publishing, provide only de-identified samples, reducing the probability of it entering others’ training corpora at the source.

2. Parameterized and randomized tasks. Make hidden rules configurable parameters so that the same template can generate equivalent variants with different values or conditions:

# Pseudocode: parameterize hidden rules to detect whether a model has "memorized" fixed settings
def build_task(seed: int) -> Task:
    rng = random.Random(seed)
    policy = Policy(
        weight_limit=rng.choice([80, 100, 120]),  # Equivalent but numerically different
        fuel_reserve=rng.uniform(0.03, 0.08),
    )
    grader = Grader(policy)          # Grader judges according to the current policy
    prompt = render(PUBLIC_TEMPLATE) # Prompt does not leak policy details
    return Task(prompt=prompt, grader=grader)

If the model consistently uses “default values” across multiple randomized policies, it can be judged as memorization rather than understanding.

3. Adversarial rule-flip testing. Following the study above, construct variants on key tasks that contradict “suspected leaked rules” and check whether the model follows the new rules of the current task. This is the most direct method for distinguishing “truly capable” from “rule memorization.”

4. Reasoning trace auditing. Align the model’s reasoning process with the task description to identify “constraints conjured from nothing.” A lightweight discriminator model can batch-screen traces and establish ongoing monitoring of leakage ratios.

5. Multi-benchmark cross-validation and business regression. Use public leaderboards only for coarse screening. Final model selection decisions should rely on private regression evaluations close to real business scenarios, and compare the score gap between “suspected leaked tasks” and “clean tasks” rather than looking only at a single overall score.

6. Implications for Model Selection and Business Decisions

Back to the business perspective: why do models with similarly “high scores” differ so greatly in market pricing and adoption strategies? Because mature technical teams never look only at public leaderboards. They look at true generalization ability, behavioral consistency, and rule adaptability under private evaluation. Leaderboard scores are just one signal. When the signal itself may be contaminated, building your own evaluation system is no longer optional—it is mandatory. This also explains why leading teams are willing to pay a premium for “verifiable capability” while remaining cautious about “top leaderboard scores.”

Conclusion

The purpose of evaluation is to predict performance in real-world scenarios. When a model memorizes the grader’s hidden rules as “the laws of the world,” scores and capability diverge systematically. The core conclusion of this research deserves to be remembered by every technical team: a model’s leading performance on public benchmarks does not necessarily mean it can generalize the same capability to new tasks where the rules have changed. A trustworthy evaluation system is the foundation for cutting through leaderboard noise and making sound technical decisions.