The Hidden Memorization Risk Behind High LLM Benchmark Scores

1. The Problem: When a Model “Already Knows the Answer”

In large language model (LLM) capability evaluation, benchmark scores have long been treated as a core indicator of a model’s reasoning ability. However, a healthy evaluation system rests on one basic premise: a model should complete a task using only the information provided in the task description. If a model “already knows” hidden scoring rules that lie outside the task description—and treats them as binding instructions—then a high score is no longer proof of reasoning ability, but rather a product of memorization and data leakage.

A recent study on terminal-operation benchmarks revealed exactly this kind of anomaly. While analyzing the reasoning traces of a newly released model, researchers found that in a substantial proportion of tasks, the model proactively mentioned scoring criteria that never appeared in the task description and internalized them as execution constraints. This finding raises three key questions:

1. How widespread is this phenomenon?
2. Is it genuine “emergent reasoning,” or benchmark memorization?
3. What does it mean for developers and enterprises choosing models?

2. Three Types of Evidence: How to Identify “Memorization-Driven High Scores”

Judging whether a model relies on implicit memory cannot be done by looking at the final score alone. The study adopted three complementary forensic methods that are worth borrowing for evaluation practice.

2.1 Reasoning Trace Analysis

The first method examines the model’s chain of thought. In a batch of runs on terminal tasks, researchers measured the proportion of reasoning traces in which the model “referred to requirements that were not provided but that matched the environment’s scorer as instructions.” The results showed:

• In 63 tasks and 189 runs on this benchmark, the new model exhibited this behavior in 23.3% of its reasoning traces;
• Its predecessor model did not show this phenomenon;
• A comparison model showed the corresponding rate of only 0.3%.

Reasoning traces are a direct externalization of a model’s internal state. A systematic bias of this kind suggests the model is not “solving the problem” but “recalling the answer.”

2.2 Directly Asking About Hidden Requirements

The second method is more direct: asking the model to list all the instructions it believes it must follow. The results:

• 34.1% of the new model’s responses included hidden requirements that were not given in the task description;
• When the same question was prefilled into a reasoning prefix (i.e., induced through “continuation”), that proportion rose to 64.6%;
• Some traces even speculated that these rules might have been distilled from the failure experiences of previous agents.

The fact that prefilling a prefix significantly increases the hit rate suggests that these hidden rules have a certain activation threshold in the model’s parameters, rather than being random hallucinations.

2.3 Task-Modification Experiments

The third method is the most convincing: change the environment and see whether the model follows.

Researchers modified file contents in four tasks so that the new requirements contradicted the hidden rules the model had mentioned. The results:

• In three of those tasks, across 15 attempts, the new model never once followed the modified requirements;
• In contrast, three comparison models followed the new rules in 26 of 27 attempts.

A typical example involved a fuel-planning program: after researchers modified the fuel policy, the model still insisted on the original weight-limit rule, even explicitly stating that it should follow what it “understood” to be the instruction, even when code comments contradicted each other. Conversely, when the modification did not involve a rule it remembered, the model was usually able to adapt normally.

This shows that the model’s behavior is not rigid, but rather selectively attached to certain specific rules—precisely those scoring points it “remembers.”

3. Score Distribution: Anomalies Concentrate on “Remembered” Tasks

If high scores come from memory, the lead should be concentrated on tasks where the model can recite hidden rules. The data supports this inference:

• On the 16 tasks where the model stated the hidden scoring rules in advance, its pass rate reached 70.8%;
• The average pass rate of five frontier comparison models was only 34.0%;
• On the remaining tasks, the model did not show the same pronounced lead.

A separate high-difficulty comprehensive test showed similar signs: the model mentioned previous tasks or the testing platform in 24.9% of questions, compared with only 3.2% for its predecessor, while none of the other six comparison models exceeded 3.9%.

This “task-selective advantage” is a typical signature of memorization contamination: the model performs abnormally well on distributions it has seen, and returns to normal on unseen distributions.

4. Possible Causes: How Data Contamination Enters Evaluation

Taken together, the evidence points to several possible pathways:

1. Overlap between training data and evaluation sets. If a benchmark’s task descriptions, scoring scripts, or related discussions have appeared in public corpora, they may be memorized by the model.
2. Agent trajectory distillation. Hidden scoring rules may come from past agents’ trial-and-error records in similar environments; if these records are used for training, the model indirectly acquires scoring preferences.
3. Evaluation environment reuse. The same benchmark may be reused across multiple generations of model evaluation, and model vendors may inadvertently bring evaluation details into training during tuning.

Regardless of the specific pathway, the core problem is the same: the evaluation set is no longer “clean,” and scores cannot reflect true generalization ability.

5. Practical Recommendations for Developers and Enterprise Users

5.1 Establish an Evaluation Audit Mechanism

Do not treat benchmark scores as the sole basis for judgment. Consider adding the following to the evaluation process:

• Reasoning trace sampling: Check whether the model mentions information not provided in the task;
• Hidden-rule probing: Ask the model directly “what other requirements are there” and measure the hit rate;
• Modification consistency testing: Inject changes that conflict with potential memorized rules and observe whether the model follows them.

5.2 Task Isolation and Dynamic Generation

• Evaluation tasks should support parameterized generation to avoid fixed prompts being repeatedly exposed;
• Strictly isolate evaluation data from training data, and establish versioned data lineage records;
• Use different variants across multiple evaluations of the same model to reduce the probability of memory reuse.

5.3 Focus on Behavioral Indicators “Beyond the Score”

When selecting models, consider including the following dimensions:

• How the model handles contradictory instructions;
• How quickly it adapts to unseen task types;
• Whether it offers reasonable explanations when comments and descriptions conflict;
• Stability on long-tail tasks.

5.4 Training Data Governance

For in-house models or fine-tuning scenarios, you should:

• Review whether training corpora contain fragments of evaluation sets;
• De-identify and de-evaluation-ize agent trajectory data;
• Establish access controls for evaluation sets to prevent them from entering the training pipeline.

6. Conclusion

High LLM scores can come from genuine reasoning or from memorization of evaluation rules. Distinguishing between the two is not merely an academic question—it directly affects enterprise model selection, cost investment, and business reliability. The three forensic methods discussed here—reasoning trace analysis, direct probing, and task-modification experiments—offer a practical audit approach. For developers, the pragmatic way to respond to “memorization-driven high scores” is to treat leaderboards rationally, build your own evaluation loop, and focus on behavior rather than scores alone.