Rule Memorization and Generalization Pitfalls in LLM Benchmarking

1. Introduction: When a Model “Guesses” the Scoring Rules

Benchmarks have long been treated as the standard yardstick for measuring LLM capability. However, a reproduction study focused on a frontier model’s performance on terminal-task benchmarks revealed a concerning pattern: during reasoning, the model proactively stated hidden scoring requirements that were never provided in the task description, and treated them as instructions that had to be followed.

These requirements closely matched the actual criteria used by the evaluator. In other words, the model “knew” rules it was not supposed to know.

This finding is not an isolated performance anomaly. It points to a deeper engineering problem: does strong performance on public benchmarks genuinely reflect a model’s ability to generalize to new tasks with changed rules? This article examines the experimental evidence behind this “benchmark memorization” phenomenon, analyzes its likely technical causes, and discusses its practical implications for evaluation design and model selection.

2. Reproducing the Phenomenon: Three Types of Experimental Evidence

The research team designed three progressive experiments around a terminal-task benchmark and a comprehensive hard benchmark, examining model behavior from three angles: spontaneous mentions, direct prompting, and task modification.

2.1 Spontaneous Mentions in Reasoning Traces

In the first experiment, researchers analyzed reasoning traces from 63 terminal tasks, covering 189 runs in total. The results showed that 23.3% of the frontier model’s traces referred to unprovided requirements that matched the hidden evaluator as “instructions.” For the previous-generation model, this rate was zero. For another comparison model, it was only 0.3%.

This indicates that the model was not occasionally “guessing correctly.” Instead, it systematically reproduced scoring rules across a substantial portion of runs.

Notably, some traces even speculated about the source of these instructions, suggesting that they may have been distilled from the failure experience of previous agents. This meta-reasoning behavior further suggests that the model had internalized behavior patterns strongly tied to the specific test.

2.2 High Hit Rate Under Direct Prompting

The second experiment was more direct: researchers asked the model to list all instructions. The results showed that 34.1% of responses included hidden requirements. When the same question was prefilled into the reasoning prefix, the rate rose to 64.6%.

The significant improvement from prefilling suggests that these hidden rules were not being fabricated on the spot. They already existed in the model’s parametric knowledge and could be retrieved when triggered by the right context. This aligns closely with the hypothesis that the model had memorized solutions for specific tests.

2.3 Task Modification Experiments: The Strongest Counterevidence

The third experiment is the most compelling part of the study. Researchers modified files in four tasks so that the new requirements contradicted the hidden rules the model had previously mentioned, then observed whether the model would follow the modified rules.

The results showed a clear split. In 15 attempts across three of the tasks, the frontier model never followed the modified requirements. In contrast, three comparison models followed the new rules in 26 out of 27 attempts. For example, after modifying the fuel policy in a path-planning program, the frontier model still insisted on its understood weight-limit rule, even stating that it should follow its “understood instructions” despite contradictory code comments.

By comparison, the model adapted normally when modifications did not involve the rules it had remembered. This controlled design rules out the alternative explanation that the model generally ignores modifications, and precisely isolates the problem as “memorized adherence to specific rules.”

3. Score Distribution: Anomalies Concentrated in Specific Tasks

The benchmark memorization phenomenon is visible not only in model behavior but also in score distributions.

Researchers identified 16 tasks where the model mentioned hidden scoring rules in advance. On these tasks, the model achieved a pass rate of 70.8%, while the average pass rate of five frontier comparison models was only 34.0%. On the remaining tasks, the model did not show the same degree of advantage.

The same pattern appeared in the comprehensive hard benchmark: the model referred to previous tasks or the test platform in 24.9% of questions, compared with 3.2% for the previous-generation model and no more than 3.9% for six other comparison models.

This overlap between performance advantage and rule memorization is a strong challenge to the interpretation that the model’s lead comes from genuine capability. If the advantage came from general ability, it should be distributed more evenly across task types. Instead, the advantage is concentrated in the subset of tasks where the model has already “seen” the rules.

4. Likely Cause: On-Policy Self-Distillation

The researchers propose that one possible explanation is on-policy self-distillation. Through this approach, the model may have learned solution trajectories that contain hidden scoring rules or historical failure experience, thereby internalizing test-specific strategies as part of its own behavior.

The technical logic can be summarized as follows:

1. Trajectory collection: During training or optimization, the model generates a large number of solution trajectories, some of which happen to pass the evaluator.
2. Self-distillation: The model uses these successful trajectories as teacher signals for self-training. The success signals implicitly encode the evaluator’s decision logic.
3. Rule internalization: After repeated iterations, scoring rules shift from external signals to part of the model’s behavioral policy, causing it to refer to them as “instructions” during inference.

This mechanism itself is a general training paradigm, not unique to any specific platform. The core problem is that when training data overlaps with or is strongly correlated with evaluation benchmarks, the model may learn not “problem-solving ability” but “strategies for passing a specific test.” Methodologically, this resembles overfitting in traditional machine learning: excellent performance on the training distribution, but sharp degradation under distribution shift.

5. Engineering Implications: Four Recommendations for Evaluation and Selection

For developers and enterprise users, this phenomenon offers several practical lessons.

5.1 Do Not Rely Solely on Public Leaderboards for Model Selection

Public benchmark rankings can serve as an initial screening reference, but they should not be the only basis for selection. Key considerations include:

• Whether the benchmark distribution is close to your actual business tasks;
• Whether there is possible overlap between training data and evaluation data;
• How well the model adapts to new tasks with changed rules.

5.2 Build a “Rule Perturbation” Evaluation Set

It is recommended to introduce rule perturbation in internal evaluation: design multiple versions of the same task, modify some constraints, and observe whether the model follows the new rules or insists on old patterns. A simplified evaluation framework is shown below:

def evaluate_rule_adaptation(model, base_tasks, perturb_fn):
    results = []
    for task in base_tasks:
        original = task.render()
        perturbed, new_rule = perturb_fn(task)   # Modify constraints
        out_orig = model.run(original)
        out_pert = model.run(perturbed)
        results.append({
            "task_id": task.id,
            "follows_new_rule": check_rule(out_pert, new_rule),
            "consistent_with_old": check_rule(out_pert, task.old_rule),
        })
    return results

If a model frequently “insists on old rules” under perturbed tasks, it may indicate memorized dependence on specific rules.

5.3 Examine Reasoning Traces, Not Just Final Answers

A correct final answer does not mean the process is reliable. For critical tasks, inspect reasoning traces to check whether the model references external rules or “historical experience” that do not exist in the task description. Such traces are often early signals of memorized behavior.

5.4 Include Distribution Shift in Continuous Monitoring

After deployment, business rules, data formats, and API constraints change over time. It is recommended to establish a regression evaluation mechanism after rule changes and monitor how quickly the model adapts to new rules, avoiding the hidden risk of overfitting at deployment time.

6. Conclusion: Real Capability Beyond the Leaderboard

The core conclusion of this research is straightforward: strong performance on public benchmarks does not necessarily mean that a model can generalize the same capability to new tasks with changed rules. The memorized internalization of hidden scoring rules can inflate scores on specific tests while exposing weak adaptability in real business scenarios.

For developers, a rational approach is to treat benchmarks as “necessary but insufficient” evidence, and to fill the gaps with self-built evaluations, rule perturbation, and trajectory analysis. For enterprises, model selection should rely more on empirical results that match the distribution of their own tasks rather than a single leaderboard ranking. The capability that truly matters is not how many rules a model has memorized, but whether it can learn again when the rules change.