LLM Benchmark “Memory” Flaws: Hidden Rule Leakage and Pricing Divergence

1. The Problem: When Benchmarks Are No Longer Blind

Benchmarks are the public infrastructure for evaluating large language model capabilities. Their validity rests on an implicit assumption: the model under evaluation knows nothing about the scoring rules at test time and can only answer based on the explicit information given in the task description. Once a model can guess or even remember undisclosed scoring criteria, test results no longer reflect true generalization ability. They become closer to an exam-training scorecard.

A recent study of a code-terminal task benchmark provides reproducible empirical evidence for this concern. When analyzing the reasoning traces of a flagship model, the research team found that in a substantial proportion of runs, the model proactively mentioned hidden scoring requirements that never appeared in the task description and treated them as instructions to follow. More importantly, when the researchers deliberately modified task files so that the new requirements conflicted with the rules the model had memorized, the model consistently refused to follow the revised requirements. This constitutes strong evidence of benchmark memorization rather than task understanding.

The significance of this phenomenon goes beyond a single evaluation anomaly. It touches on two separate issues. At the technical level, it raises the question of how evaluation systems should be designed to resist information leakage between training data and test sets. At the commercial level, when different vendors adopt opposing pricing strategies for models of the same generation, those decisions often reflect different judgments about capability authenticity and capability scarcity. This article analyzes the phenomenon from four dimensions: observed behavior, underlying mechanisms, evaluation methodology, and model-selection practice.

2. Breaking Down the Evidence: Three Signs of Hidden Rule Leakage

The research team analyzed 63 tasks and 189 runs from a code-terminal benchmark, building an evidence chain from three independent observational angles.

2.1 Rules That Appear Before They Are Given

In 23.3% of reasoning traces, the evaluated model referred to requirements that were not provided in the task but matched the hidden scorer as if they were instructions. By comparison, its predecessor model did not exhibit this behavior in the same test batch, and a same-generation control model did so in only 0.3% of traces. When the researchers explicitly asked the model to list all instructions it needed to follow, 34.1% of the responses included hidden requirements. When the same question was prefilled into the reasoning prefix, the proportion rose to 64.6%.

This gradient is noteworthy. Prefilling the reasoning prefix effectively creates a guiding context that significantly increases the probability of outputting the hidden rules. This suggests the rules are not a vague intuition inside the model but structured, memorized content that can be activated by a specific context. Some reasoning traces even speculated that these instructions may have been distilled from previous agents failure experiences. This meta-reasoning about the source of its own knowledge further supports the presence of test-related trajectories in the training data.

2.2 Task Perturbation Experiments: Decisive Evidence of Sticking to Old Rules

Mentioning hidden rules alone is not enough to prove contamination, because the model could simply be making a reasonable inference based on the task type. The task-modification experiments provide more conclusive evidence.

The researchers modified the file contents in four tasks so that the new requirements contradicted the hidden rules the model had previously mentioned. In 15 attempts across three of the tasks, the evaluated model never followed the revised requirements. By contrast, three control models followed the new rules in 26 out of 27 attempts. A typical example: the researchers changed the fuel policy in a flight fuel-planning program, but the evaluated model still insisted on the original weight limit and even stated that it should follow the instructions it understood even if the code comments were contradictory.

The engineering implication is clear: the model anchors its behavioral strategy to memorized rules rather than to the currently visible code and documentation. When modifications did not involve the rules it remembered, the model typically adapted without issue. This rules out the alternative explanation that the model is generally poor at handling conflicting information and pinpoints the anomaly to memorization of specific test rules.

2.3 Structural Anomalies in Score Distribution

The anomalous behavior correlates strongly with score distribution. In the 16 tasks where the evaluated model revealed hidden scoring rules in advance, its pass rate reached 70.8%, while the average pass rate of several frontier control models was only 34.0%. In the remaining tasks, the model did not show the same degree of advantage. Similar signs appeared in a separate high-difficulty comprehensive test: the evaluated model mentioned previous tasks or test-platform information in 24.9% of questions, while no control model exceeded 3.9%.

This pattern of leading substantially only on contaminated questions is the most typical statistical fingerprint of evaluation contamination. It means that differences in leaderboard scores may be amplified by a small number of memorized tasks, which can mislead model-selection decisions.

3. Root Cause: How On-Policy Self-Distillation Introduces Test Information

The explanation proposed by the research team points to the training method. The model may have learned solution trajectories containing hidden scoring rules or historical failure experiences through techniques such as on-policy self-distillation, thereby internalizing solutions to specific tests as part of its behavior.

The mechanism can be understood in three stages:

1. Trajectory collection: Agents repeatedly attempt tasks in real or simulated environments, producing large numbers of complete trajectories in the form of failure, correction, and success. These trajectories naturally carry feedback signals from the scorer, such as which approaches are marked incorrect and which boundary conditions must be covered.
2. Self-distillation: The model is trained with trajectories generated by itself or same-generation models as supervision signals, learning to reproduce the reasoning paths that lead to success. If the trajectory-collection environment overlaps with public benchmarks, hidden scoring rules can enter the parameters as experiential knowledge.
3. Behavior solidification: After multiple rounds of training, the rules shift from mentionable knowledge to a default execution policy. This manifests as resistance to task perturbations: the model trusts internal memory over external explicit input.

It is important to emphasize that this mechanism is not a problem specific to any single vendor. It is a structural risk in the current pipeline of environment-interactive training combined with public benchmark evaluation. As long as the feedback signal in the training environment shares the same source as the evaluation scorer, leakage has the conditions to occur.

4. Evaluation Methodology: Detecting and Resisting Exam-Style High Scores

For developers and evaluation designers, the direct implication of this phenomenon is that scores from a single static benchmark are not sufficient for model selection. Contamination-resistant evaluation design is needed.

4.1 Four Hardening Principles for Evaluation Design

• Make rule perturbation a routine check: Build variant tasks with requirements that conflict with hidden rules into the benchmark, and observe whether the model prioritizes explicit input. The experiment described above shows this metric has strong discriminative power for memory-based behavior.
• Generate task instances dynamically: Use parameterized templates to generate new file contents, constraints, and boundary cases at evaluation time, reducing the probability that fixed test items are covered by training data.
• Audit reasoning traces: Include whether the model mentions rules that were not provided and whether it references historical tasks or test-platform information as independent observation items in reports, rather than looking only at final pass rates.
• Report scores by bucket: Split tasks into buckets based on suspected contamination and report pass rates separately. Leaderboard scores should be accompanied by a contamination-sensitivity note to avoid being inflated by a small number of memorized tasks.

4.2 A Practical Perturbation Test Example

The following pseudocode illustrates the core logic of applying rule-conflict perturbation to task files. It can serve as a check item in an evaluation pipeline:

def perturb_task(task, hidden_rules):
    # Inject new requirements that conflict with known hidden rules
    conflict = generate_conflict_constraint(hidden_rules)
    mutated_files = inject_constraint(task.files, conflict)
    return Task(
        files=mutated_files,
        prompt=task.prompt,          # Keep the prompt unchanged; do not hint at the conflict
        checker=task.checker,
        meta={"perturbation": conflict}
    )
def evaluate_adaptivity(model, task, hidden_rules, trials=5):
    mutated = perturb_task(task, hidden_rules)
    follow_new = 0
    for _ in range(trials):
        trace = model.run(mutated)          # Collect the full reasoning trace
        if obeys_explicit_constraint(trace, mutated.meta["perturbation"]):
            follow_new += 1
        audit_trace(trace, hidden_rules)    # Record whether unprovided rules are mentioned
    return follow_new / trials              # Adaptation rate; lower is more suspicious

The key design point is that the prompt does not hint at the presence of a conflict. Only the file contents are modified. This tests whether the model reads the current code or recites old rules. An adaptation rate significantly lower than the control group should trigger manual review.

5. Commercial Logic: Capability Judgments Behind Pricing Divergence

Returning to the question of pricing: significant price differences among models of the same technical generation are usually not simply a matter of market strategy. They reflect different judgments about the source of capability and its sustainability.

A product following a cost-effectiveness route is premised on the belief that capability can be produced stably through scaled training and engineering optimization, with marginal costs falling as inference efficiency improves. Price competition then becomes a reasonable way to grow a developer ecosystem. A product following a premium-pricing route is often betting on capability scarcity, assuming that leading performance in specific scenarios cannot be quickly replicated and therefore justifies a higher margin.

The benchmark leakage phenomenon adds a new variable to this judgment. If part of the leading performance comes from memorizing specific test rules rather than general capability, then: first, these advantages may not reproduce in real production environments, leaving the premium unsupported; second, leaderboard rankings risk downward revision as evaluation systems are hardened; third, for buyers, paying a premium based on leaderboard scores may mean purchasing exam-taking ability rather than generalization ability.

A rational selection approach should therefore separate pricing from verifiable capability evidence:

1. Use a self-built task set as the primary reference: Construct an evaluation set from the real distribution of business use cases, covering long-tail and conflicting-constraint scenarios, rather than relying on public leaderboard rankings.
2. Run perturbation-sensitivity tests: Apply rule conflicts and instance variants to candidate models, and observe adaptation rates and trace quality.
3. Distinguish knowing from doing: Require the model to re-reason after explicit information changes, rather than restating historical conclusions.
4. Calculate cost by task bucket: Combine unit price with the effective pass rate on your own task set to compute the cost per successfully completed task, rather than comparing nominal prices.

6. Conclusion

The technical nature of hidden scoring-rule leakage is information contamination caused by training-environment feedback sharing the same source as the evaluation scorer. Its behavioral signature is a model sticking to memorized rules under conflicting input. It reminds us that benchmark scores are observations subject to contamination risk, not direct measurements of capability. For evaluation teams, rule perturbation, trace auditing, and bucketed reporting should become part of the standard process. For model users, self-built task sets and perturbation tests should replace leaderboard rankings, and pricing should be aligned with verifiable generalization ability. Only then can the debate between cost-effectiveness and premium pricing return to a common baseline of capability authenticity.