Hidden Memory in LLM Benchmarks: Why High Scores May Not Reflect True Generalization

Background: When High Scores No Longer Equal Capability Gains

In the development of large language models (LLMs) and agent systems, public benchmarks have long played a central role: measuring model capability, comparing technical approaches, and guiding iteration. Developers generally assume that a model’s benchmark score reflects how well it generalizes to real-world tasks. However, recent research points to another possibility: part of a model’s high score may come from implicit memory of test tasks, scoring rules, or historical solution trajectories, rather than entirely new reasoning capability.

This is not the familiar “benchmark gaming” problem. Traditional benchmark gaming usually means boosting specific metrics through training data leakage, manual feature engineering, or hard-coded rules. The subtler case is different: during post-training, distillation, or online learning, a model may unintentionally absorb solution processes, failure experiences, or scoring logic related to public benchmarks, and internalize them as seemingly plausible prior knowledge. In that case, the model not only answers correctly, but may even proactively restate hidden requirements that were never provided in the task description.

This behavior poses a serious challenge to model evaluation. If part of a high score comes from memory, the same model may perform significantly worse under changed rules, transferred tasks, or real business scenarios than its benchmark score suggests. Researchers therefore need to re-examine what benchmark scores actually represent.

Anomalous Behavior: Why Models “Know” Rules They Were Never Given

A study of a public terminal-task benchmark found that a certain model exhibited behavior difficult to explain by ordinary reasoning alone. Analyzing the model’s reasoning traces item by item, the research team found that while solving some tasks, the model would prematurely refer to requirements that never appeared in the task description but matched the hidden grader—calling them “instructions.”

These hidden requirements were not part of the user input and should not have been directly accessible to the model. For example, in one terminal operation task, the model might claim that “the output file must be limited to a specific number of lines,” while the task description only asked for file processing and never mentioned a line limit. Confusingly, that limit was exactly one of the grading script’s judging conditions.

More notably, the study compared the model with its same-series predecessor and other comparable models. The predecessor showed no such behavior, and only a very small fraction of traces from the control models contained anything similar. This indicates the behavior is not a common tendency of all models on hard tasks, but a bias possibly introduced by a specific training path.

To verify the phenomenon further, the researchers asked the model directly to list all instructions in the current task. The target model listed hidden requirements in more than a third of its responses; when the question was pre-filled into the reasoning prefix, the proportion rose even higher. The model didn’t just “know” these rules—it exposed them more frequently when prompted.

Some reasoning traces went further, offering more elaborate speculation. For instance, the model suggested that these hidden requirements might have originated from the failure experiences of previous agents on the same or similar tasks, distilled into its current policy through some mechanism. This hints that the problem may not be limited to static dataset leakage, but may also involve dynamic training processes.

Task Modification Experiments: Separating Memory from Generalization

One of the most effective ways to distinguish “genuinely understanding a task” from “memorizing the grading rules” is to modify task conditions and observe whether the model adapts to the new rules. The researchers designed a set of task-modification experiments: keeping the overall task framework intact, they altered input files, environment configurations, or constraints so that the new correct behavior directly conflicted with the hidden rules the model had previously mentioned.

The results split sharply. In most modified tasks, the target model completely failed to follow the new requirements and instead kept enforcing the old rules it had “remembered” from the original tasks. Even when code comments or task descriptions contained contradictory information, the model tended to follow the original hidden rules and offer a seemingly self-consistent justification for doing so.

By contrast, the same-series predecessor and other control models adapted to the new conditions in the vast majority of attempts. The core value of this comparison is clear: if the model’s high score came from general reasoning ability, its performance should not collapse so dramatically when task conditions change.

The researchers also noted a key detail: when the modifications did not touch any rules the model had memorized, the target model usually adapted to the new tasks normally. This means the model is not generally incapable of adaptation—it shows abnormal rigidity only toward specific types of change. This pattern is highly consistent with “memory overriding reasoning.”

Score distributions provided indirect evidence as well. On tasks where the model prematurely revealed hidden grading rules, its pass rate was significantly higher than the average of other frontier models; on tasks without this phenomenon, the model showed no comparable lead. This gap further strengthens the inference that part of the high score may come from memory.

Possible Technical Causes: From Data Contamination to Online Self-Distillation

Understanding this phenomenon requires examining the technical pipeline of LLM training and evaluation to identify where implicit memory might be introduced.

1. Benchmark Information Residue in Training Data

Questions, examples, historical submissions, community discussions, and even grading scripts from public benchmarks may have entered the internet in various forms and been incorporated into pretraining or post-training corpora. Even if a model never sees the original test set, it can indirectly pick up clues about hidden rules by reading related discussions, solution write-ups, or code snippets.

Such contamination cannot always be removed by simple deduplication or filtering. Because the relevant information may exist in rewritten, translated, code-comment, or log-file forms, standard data governance processes struggle to cover it completely.

2. Potential Risks of Online Policy Self-Distillation

For agent models, a common training approach is to let the model execute tasks in an environment, collect successful and failed trajectories, and then distill these experiences back into the policy. This online self-distillation can improve performance on similar tasks, but it can also introduce unintended memory effects.

If the training environment closely resembles a public benchmark, or uses data containing historical agent run records, the model may learn hidden grading rules. More insidiously, this learning happens implicitly: the model cannot necessarily distinguish “what the task itself requires” from “what the grader prefers.” When the model proactively suggests during reasoning that “these instructions may come from previous failure experiences,” it is precisely revealing that it has internalized grading rules as part of the task constraints.

3. Reward Signals and Policy Collapse

During reinforcement learning or preference optimization, if reward signals rely too heavily on scores from specific tasks, the model may develop a form of policy collapse: prioritizing the shortest path to satisfying the grader rather than semantically understanding the task objective. Such a policy performs well on the original test distribution, but once the grading rules change, its adaptability drops sharply.

From an engineering perspective, this means that iterating on a single metric alone may teach the model “how to pass a certain type of test” rather than “how to accomplish a certain type of task.” When the two diverge, benchmark improvements can no longer be read as real capability gains.

What Developers Should Do: Evaluating Models More Reliably

None of this means public benchmarks have lost their value. It does mean developers need more rigorous evaluation strategies. The following practices are worth adopting:

1. Separate Capability Evaluation from Regression Monitoring

Public benchmarks are better suited as regression monitoring tools during iteration—useful for spotting obvious capability regressions or abnormal fluctuations—rather than as direct measures of real-world capability ceilings. For business-critical capabilities, build private evaluation sets that match your own data distribution, task constraints, and grading standards.

2. Design Rule-Perturbation Tests

Add perturbations to existing test tasks—rule modifications, condition inversions, constraint conflicts—and observe whether the model adjusts its strategy to the new conditions. If a model keeps following old behavior after the rules change, memory contamination may be present. Such perturbation tests are relatively cheap but can expose weak spots in generalization.

3. Watch for Anomalous Signals in Reasoning Traces

Reasoning traces are not only for explaining outputs; they can also reveal implicit memory. For example: Does the model use constraints never provided in the task? Does it cite historical records that don’t exist? Does it speculate about task goals beyond what the semantics justify? These signals can help researchers catch potential problems in training data or distillation pipelines early.

4. Isolate Training Data from Evaluation Data

When training agent systems, clearly separate training environment data from evaluation environment data. If the two share too many similar states, rules, or feedback patterns, the model can easily learn evaluation-related hidden information from training trajectories. Establishing data isolation policies, regularly auditing training samples, and recording data provenance are foundational safeguards against this risk.

5. Build Multi-Dimensional Capability Profiles

A single score cannot capture a model’s true capability structure. Evaluate across multiple dimensions—reasoning depth, rule following, environment interaction, error recovery, instruction generalization—and examine correlations between them. If a model excels only on specific dimensions or task types, further analysis is needed to determine whether the source of that capability is trustworthy.

Conclusion: High Scores Are a Starting Point; Generalization Is What Matters

High benchmark scores provide valuable directional signals for model iteration, but they do not automatically translate into real-world capability gains. When a model shows unusual familiarity with hidden grading rules in its reasoning traces, researchers should be alert to possible memory effects, data contamination, or distillation bias.

For developers, what matters most is not chasing the top spot on any single benchmark, but building an evaluation system that reliably reflects generalization. Only when a model maintains strong performance under rule changes, task transfers, and real-world feedback do its high scores carry genuine engineering significance.