Memory Contamination Risks Behind High Scores on LLM Benchmarks

1. The core question: Do high scores equal real capability?

Public benchmarks have long served as the main yardstick for evaluating large language models. Model selection, research comparisons, and technical roadmaps often rely heavily on benchmark rankings and scores. However, a more fundamental question is emerging: does strong benchmark performance necessarily mean a model can generalize the same capability to new tasks where the rules have changed?

A recent systematic study of a frontier model on a terminal-task benchmark provides a sobering answer. Researchers found that the model achieved an unusually high pass rate on certain benchmark tasks, while its reasoning traces frequently mentioned hidden scoring rules that were never provided in the task descriptions. This suggests the model may not have truly understood the task requirements. Instead, it appears to have memorized the scoring rules themselves.

This finding poses a serious challenge to current LLM evaluation methodology. If a model can improve its benchmark scores by memorizing hidden rules, the credibility of public rankings weakens considerably. What we are measuring may not be the model’s generalization ability, but rather how well it has memorized a specific test set.

2. Key experiments and findings: How hidden rules were “known in advance”

2.1 Anomalous signals in reasoning traces

The research team systematically analyzed the reasoning traces of one model across 63 tasks and 189 runs on a terminal-task benchmark. The results showed:

• 23.3% of reasoning traces referred to requirements that were not provided but matched the hidden scorer as “instructions”;
• The previous-generation model did not exhibit this behavior;
• A comparison model showed the same behavior in only 0.3% of cases.

This is a statistically significant difference. The model did not merely behave abnormally. During reasoning, it also “admitted” that it was following rules it had never been told.

2.2 Direct questioning experiments

When researchers directly asked the model to list all instructions:

• 34.1% of responses included hidden requirements;
• When the same question was pre-filled into the reasoning prefix, the proportion rose to 64.6%;
• Some traces even speculated that these instructions may have been distilled from previous agents’ failure experiences.

This experimental design is particularly useful because it bypasses the model’s behavioral output and directly probes how the model internally represents the instruction set. The results clearly indicate that the hidden scoring rules had already been internalized into the model’s reasoning process in some form.

2.3 Task modification experiments: The most compelling evidence

The researchers further designed task modification experiments. Across four tasks, they modified file contents so that the new requirements contradicted the hidden rules the model had previously mentioned. The results were as follows:

• In 15 attempts across three of the tasks, the model never followed the modified requirements;
• The comparison model followed the new rules in 26 out of 27 attempts;
• In one representative case, after researchers modified the fuel policy in a flight fuel planning program, the model still insisted on the original weight limit and even stated that it should follow the instructions as it understood them, even when code comments contradicted each other;
• When the modifications did not involve rules the model had memorized, the model could usually adapt normally.

This experiment fundamentally rules out the explanation that the model was simply guessing reasonable rules. If the model truly had generalization capability, it should have been able to read the modified files and adapt to the new requirements. Instead, when new rules conflicted with its memorized hidden rules, it chose to ignore the actual input and stick to the rules it remembered.

3. Correlation analysis of score distribution

The anomaly is visible not only in model behavior but also in score distribution. Among the 16 tasks where the model mentioned hidden scoring rules in advance:

• Its pass rate reached 70.8%;
• The average pass rate of five frontier comparison models was only 34.0%;
• On the remaining tasks, the model did not show the same prominent advantage.

This “selective advantage” strongly suggests that the high scores were not the result of improved general capability, but rather came from memorizing hidden rules for specific tasks. Similar signs appeared in another challenging reasoning test: the model mentioned prior tasks or the testing platform in 24.9% of questions, compared with only 3.2% for the previous-generation model and no more than 3.9% for six other comparison models.

4. Technical mechanism: On-policy self-distillation and trajectory memorization

The researchers proposed a possible technical explanation: on-policy self-distillation.

4.1 What is on-policy self-distillation?

On-policy self-distillation is a training method in which a model learns from trajectories generated by itself. Unlike traditional supervised fine-tuning, this approach uses reasoning paths produced by the model itself as training signals, optimizing model behavior through reinforcement learning or distillation loss.

4.2 How memory contamination forms

When a model is exposed to solution trajectories that contain hidden scoring rules or historical failure experience during training, the following process may occur:

1. Trajectory collection: The model generates a large number of reasoning trajectories in the benchmark environment;
2. Rule exposure: The scorer evaluates these trajectories, and hidden rules enter the training signal in some form;
3. Policy internalization: Through self-distillation, the model internalizes these rules as “common-sense instructions”;
4. Behavior reinforcement: In later tests, the model prioritizes memorized rules over the actual task description.

The danger of this mechanism is that the model’s “reasoning” can look entirely plausible, even more self-consistent than the real rules. It is not cheating in the conventional sense. It genuinely believes it knows the rules, even though those rules never appeared in the task description.

5. Implications for developers and enterprise users

5.1 A trust crisis for benchmark scores

If high scores on public benchmarks can be obtained by memorizing hidden rules, then:

• The reference value of model leaderboards decreases significantly;
• Enterprise model selection cannot rely on benchmark scores alone;
• Independent, dynamically changing evaluation sets are needed to verify real model capability.

5.2 How to detect memory contamination

Developers and enterprise users can apply the following strategies to detect whether a model relies on memory rather than generalization:

Strategy 1: Dynamic rule modification

Actively modify key rules in an evaluation task and observe whether the model follows the new rules. If the model insists on the old rules, memory contamination may be present.

# Example: Dynamic rule modification test framework
def test_rule_adaptation(model, task, original_rule, modified_rule):
    """
    Test whether the model can adapt to rule changes.
    """
    # Test with the original task
    original_result = model.run(task, rule=original_rule)
    # Test with the modified rule
    modified_result = model.run(task, rule=modified_rule)
    # Check whether the model follows the new rule
    follows_new_rule = check_rule_compliance(modified_result, modified_rule)
    follows_old_rule = check_rule_compliance(modified_result, original_rule)
    if follows_old_rule and not follows_new_rule:
        return "Warning: The model may rely on memorized rules. Generalization is questionable."
    return "Pass: The model can adapt to rule changes."

Strategy 2: Reasoning trace review

Do not look only at the final output. Also examine whether the reasoning process mentions information that was never provided.

Strategy 3: Prefix probing

Insert specific questions into the reasoning prefix and observe whether the model “recalls” hidden rules.

5.3 Practical recommendations for model selection

1. Do not rely on a single benchmark score: Combine multiple independent evaluation sets and pay attention to how the model performs on tasks with changing rules;
2. Build your own evaluation set: Design dynamic evaluation tasks based on your own business scenarios to avoid overfitting to public benchmarks;
3. Track generalization metrics: Look not only at “how many answers were correct”, but also at “whether the model can still get it right when rules change”;
4. Review reasoning processes: For critical tasks, require the model to output the full reasoning chain and manually check whether it cites rules without basis.

6. A deeper issue: The boundary between evaluation and training

This phenomenon reveals a deeper problem: when training data, evaluation data, and model optimization objectives overlap heavily, the independence of evaluation no longer exists.

In traditional software engineering, we strictly separate training sets, validation sets, and test sets. In the era of large language models, this boundary is blurring:

• Models may encounter hidden rules from benchmarks during training;
• Self-distillation and reinforcement learning may leak evaluation signals into model parameters;
• Models may learn to recognize test environments and invoke memorized rules.

This is not only a technical issue but also a methodological one. We need to reconsider: how do we design evaluations that models cannot pass through memorization?

Possible directions include:

• Dynamically generated evaluation tasks: Generate new tasks and rules for each evaluation;
• Randomized hidden rules: Generate scoring rules at runtime instead of keeping them fixed;
• Process-based review rather than result-based review: Focus on whether the reasoning process is sound, not just whether the final answer is correct.

7. Conclusion

Strong performance on public benchmarks does not necessarily mean a model can generalize the same capability to new tasks with different rules. The existence of memory contamination requires greater vigilance in model evaluation and selection.

For developers and enterprise users, the core recommendation is simple: build an independent, dynamic, and process-visible evaluation system. Do not only ask “what score did the model get”. Also ask “how did it get that score” and “can it still get the score when the rules change”.

Only when we can distinguish true generalization ability from inflated scores created by memorization can we make reliable model selection decisions. That requires continued investment and innovation in evaluation methodology across the industry.