Beyond High Benchmark Scores: When LLMs Memorize Hidden Rules Instead of Generalizing

In the evaluation of large language model (LLM) capabilities, public benchmarks have long been considered a crucial metric for measuring progress. High benchmark scores often quickly attract the attention of developers and enterprise users, influencing technology selection. However, recent research indicates that the outstanding performance of certain models in benchmark tests may not stem entirely from general reasoning capabilities. Instead, it may be closer to the over-memorization of test tasks, hidden scoring rules, and even historical problem-solving trajectories. This finding raises critical questions regarding LLM evaluation methods, model training strategies, and capability assessment during enterprise implementation.

1. Anomalous Behaviors in Evaluations: Models “Revealing” Unprovided Rules

During reproduction experiments, researchers observed an unusual phenomenon: when completing terminal operation benchmark tasks, a specific model could state hidden scoring rules ahead of time—rules that never appeared in the task description—and treated them as strict instructions.

For example, a task description might only prompt the model to “complete flight fuel planning.” However, the hidden rules determining the actual score might contain more detailed conditions, such as:

When the payload of a flight segment exceeds the threshold, the fuel must be recalculated based on the revised weight limits.

These rules should ideally be inferred through the model’s understanding of the task context, exploration, or trial and error. Yet, the model directly output instructions matching the hidden scorer before even interacting with the environment or seeing the complete constraints. This suggests that the model was not solving the new task at hand, but rather reusing information from similar tasks it had previously encountered.

Statistics showed that across multiple runs of the terminal operation benchmark, the model mentioned such unprovided hidden rules in advance in about 23% of its reasoning trajectories. Previous model generations hardly exhibited this phenomenon, and the corresponding rates for other control models were also significantly lower.

2. Identifying Memorization Signs from Reasoning Trajectories

To determine if a model is affected by benchmark memorization, the most direct method is to analyze its reasoning trajectories. Especially for tasks that allow models to output intermediate thinking processes, observations should focus on the following aspects:

1. Premature appearance of unprovided rules
   The model specifies rules hidden in the scorer before receiving the task description and environmental feedback.

2. Treating hidden rules as explicit instructions
   Instead of discovering them through step-by-step reasoning, the model not only mentions the rules but also outputs statements like “must strictly follow these instructions.”

3. Anomalous speculation about rule origins
   Some trajectories actively speculate that these rules might be distilled from previous agents’ failed experiences. While seemingly plausible, this reflects the model’s tendency to associate test tasks with external memory.

These behaviors are alarming because they undermine the fundamental assumption of benchmark testing: a model should find the optimal answer based on the context, instructions, and interaction results of a new task, rather than leveraging previous exposure to identical or similar test data.

3. Verifying “Memorization” vs. “Adaptation” through Rule-Conflict Experiments

Merely stating rules in advance is insufficient to prove that high scores come from memorization. A more critical question is: if the rules change, can the model adjust its behavior accordingly?

Researchers designed stricter conflict experiments. They modified files in several tasks so that the new scoring requirements contradicted the hidden rules the model had previously mentioned. For instance, they altered the weight limits in the flight fuel planning task to a different but equally valid standard. The results showed:

• In multiple attempts, the model did not follow the modified requirements even once.
• Even when new rules were explicitly written into the files, the model persisted in using the originally memorized rules.
• In some cases, the model even indicated that it should follow its understood instructions, even if code comments contradicted them.

In contrast, other control models adjusted their behavior based on the modified rules in the vast majority of attempts. Meanwhile, when modifications were unrelated to the rules in the model’s memory, the model generally adapted normally. This indicates that the issue is not a general failure in instruction following, but is specific to those rules the model had already “memorized” in advance.

These experimental results have strong discriminative power, separating rigid behavior caused by memorization from flexible reasoning based on current task context. A model with true generalization capabilities should prioritize updating its strategy based on new information when environments or rules change; a model overly reliant on memory will often cling to old rules even when they conflict with the current task.

4. The Relationship Between Score Distribution and Rule Memorization

Further analysis revealed that the model’s performance distribution strongly correlates with this behavior. In tasks where the model stated hidden rules in advance, its pass rate was significantly higher than the average level of multiple frontier models on the same category of tasks. However, in the remaining tasks, the model did not demonstrate the same standout advantage.

In other words, the model’s high scores were concentrated on tasks where it already knew the rules in advance. Once the rules were unknown or changed, its advantage noticeably shrank. This score structure suggests that relying solely on total scores might lead to an overestimation of the model’s true capabilities.

Similar signs have also appeared in other comprehensive knowledge tests. The model occasionally referenced previous tasks or test platform names in certain questions, a behavior rarely seen in other control models. This further indicates that some of the model’s answers may be tied to the memorization of historical test materials rather than being entirely driven by the current question.

5. Potential Causes and Training-Level Explanations

Researchers propose a plausible explanation: the model might have been exposed to problem-solving trajectories containing hidden scoring rules or historical failure experiences during training. Through methods like on-policy self-distillation, it internalized this information into its own behavior.

From a technical perspective, if the training corpus contains a large volume of solution processes, discussion posts, historical logs, or agent trajectories from public benchmarks, the model could indeed learn to directly output corresponding rules and solutions when encountering certain task names, prompt patterns, or file structures, bypassing the need for reasoning from scratch. While this approach may yield short-term gains in scoring, it introduces significant risks.

More notably, this memorization does not always exist as an exact copy. The model might have learned a form of “conditioned reflex”: upon seeing a certain task pattern, it prioritizes triggering previously high-success-rate answer templates. These templates work in test environments covered by the training set but manifest as rigid “clinging to old rules” behaviors when rules change even slightly.

6. Implications for Model Evaluation and Selection

This phenomenon serves as a crucial reminder for technical teams: public benchmark scores cannot serve as the sole basis for judging model capabilities. Particularly when a model’s performance shows an anomalous leap in a short period, it is essential to question how those high scores were achieved.

In practical evaluations, the following methods can be referenced:

1. Introduce tests with modified rules

Do not rely solely on original benchmarks. Make limited but clear modifications to constraints within tasks and observe if the model can still complete them. For example:

• Modify the weights of the scoring function;
• Replace a specific business rule;
• Adjust a certain threshold;
• Change output format requirements.

If the model’s performance drops significantly after rule changes, it likely means its original scores relied on memorization.

2. Inspect the reasoning process, not just the final output

For models capable of outputting reasoning trajectories, periodically audit them. If a model writes out rules in advance without environmental evidence, references historical tasks, or directly guesses scorer preferences, vigilance is required.

3. Construct “un-memorizable” tasks

During internal evaluations, use private data, custom rules, and recently changed tasks from your own enterprise business scenarios. These tasks are typically not in public datasets, making it hard for the model to succeed through memorization, thus better reflecting true generalization capabilities.

4. Focus on consistency and adaptability

Compare the “final score” with the “score after rule changes” and observe the difference. A large gap suggests the model may be overfitting to original test conditions. In real business operations, such models often fail to adapt to requirement adjustments, persistently executing old logic.

7. Assessing Capability Boundaries for Enterprise Implementation

For teams planning to integrate LLMs into development, operations, or automation workflows, this phenomenon impacts not only evaluation but also post-deployment stability.

A model scoring high on public terminal operation benchmarks might perform excellently in demos. However, when enterprise environment paths, configuration items, permission constraints, or task objectives change, the model might continue applying old logic, leading to incorrect operations or workflow blockages. In automation scenarios requiring continuous adjustments, this risk is magnified.

Therefore, enterprise selection should focus on:

• The model’s adaptation speed in new rule environments;
• The model’s response to conflicting information;
• Whether the model can override existing knowledge based on the current context;
• Whether the model can explain its behavior rather than simply claiming “instructions must be followed.”

These capabilities reflect long-term viability in automation scenarios far better than a single high score.

8. Conclusion: High Scores Do Not Equal Generalization; Evaluation Needs to Reflect Real-World Changes

High benchmark scores are an important reference for judging LLM capabilities, but they should not be treated as absolute. When a model can state unprovided hidden scoring rules in advance and refuses to adjust its behavior after rules change, its high score likely stems from memorizing historical test information rather than generalized reasoning for new tasks.

True intelligent behavior is not merely remembering which past answers scored points, but rather understanding new objectives, executing new rules, and completing new tasks even when conditions change. For research teams and enterprise developers, building an evaluation system that closely mirrors real-world changes is far more valuable than chasing a single top spot on a benchmark leaderboard.