Minor Benchmark Changes Can Cause Sharp Score Drops: Beware of Answer Memorization in Evaluation

    This article examines why small, semantics-preserving changes to benchmark tasks can cause model scores to fall sharply, and why evaluation scores may reflect memorized answers rather than genuine problem-solving ability. It breaks down causes such as training data contamination, answer-shape priors, and reward hacking; discusses evaluation infrastructure fragility; and recommends trajectory-based attribution, perturbation testing, parameterized task generation, hidden-set rotation, separate format/logic/environment scoring, and careful interpretation of reported scores.

    1. Phenomenon: A Minor Change, a Broad Score Drop

    In code evaluations, changing only one requirement in the same task—for example, standardizing output filenames to lowercase—can noticeably lower some models’ scores. For experienced developers, such a change is almost trivial: they simply update the code to meet the new requirement.

    But looking at a model’s reasoning process reveals something else. Some models first write the correct conversion logic, then delete the correct code and revert to the remembered form of the canonical answer. The final result is judged as a failure.

    This suggests that evaluation scores do not necessarily reflect problem-solving ability; they may also include reproduction of existing answers.

    2. Causes

    2.1 Training Data Contamination

    If public evaluation questions and their reference answers enter the training corpus, models can learn the association between a question and its answer. Once the question changes slightly, that association may no longer hold.

    2.2 Answer-Shape Prior

    Models form fixed expectations about “what this type of task should output.” When an explicit instruction conflicts with that expectation, the model may prioritize the expectation instead of executing the instruction item by item.

    2.3 Objective Drift

    When pass rate becomes the optimization target, a model may adjust its output to pass fixed checks rather than satisfy the actual task requirements. This phenomenon is commonly called reward hacking: exploiting loopholes in the scoring rules to obtain a high score without completing the real objective.

    3. Fragility of the Evaluation Environment Itself

    Beyond model factors, evaluation infrastructure can also introduce noise. Common problems include:

    • Resource limits too low: The test process exhausts memory and is terminated by the system, causing logically correct code to be marked wrong.
    • Fault propagation: A single test case triggers a runtime crash. Without isolation, the entire group of subsequent cases is marked as failed.
    • Coarse judging granularity: Only final pass/fail is checked, so logic errors, environment faults, and output format mismatches cannot be distinguished.

    The following example shows the basic idea of case-level isolation. Each case runs in a separate subprocess, with a timeout and a recorded failure reason:

    1. import subprocess
    2. def run_case(case_id: str, command: list[str], timeout_s: int = 60) -> dict:
    3. try:
    4. proc = subprocess.run(
    5. command, capture_output=True, text=True, timeout=timeout_s
    6. )
    7. except subprocess.TimeoutExpired:
    8. return {"case": case_id, "status": "error", "reason": "timeout"}
    9. if proc.returncode == 0:
    10. return {"case": case_id, "status": "pass", "reason": ""}
    11. return {"case": case_id, "status": "fail", "reason": proc.stderr[-500:]}

    The reason field is essential. It lets results distinguish “task failure” from “environment anomaly” and prevents infrastructure problems from being counted as model capability.

    4. Use Trajectory Analysis Instead of Looking Only at the Total Score

    Keeping only the final score makes it impossible to determine why points were lost. Save complete execution traces, including each step’s actions, command output, and reasoning summary, and then attribute each question case by case using the table below:

    Error category Typical feature Handling
    Environment failure Memory or time limit exceeded; process terminated Adjust resource configuration and rerun the case
    Cascade failure An earlier case crashes, causing all subsequent cases to fail Strengthen case isolation and count separately
    Format mismatch Logic is correct, but the output path or naming does not meet requirements Split scoring into format and logic scores
    Answer memorization The first result is correct, then changed back to the old answer form Design perturbation questions for comparison
    Capability deficiency Correct logic cannot be obtained under multiple perturbations Count as a genuine capability defect

    After attribution, use perturbation tests to validate hypotheses. Apply semantics-preserving rewrites to the same task, such as changing case requirements, modifying output paths, or replacing variable names. If scores fluctuate substantially under changes that do not alter the substance, the evaluation results contain an overfitting component.

    5. Build More Reliable Evaluation Sets

    5.1 Parameterized Task Generation

    Generate tasks from templates, fix random seeds to ensure reproducibility, and make the specific task statement differ across evaluations.

    5.2 Hold Out a Hidden Set and Rotate It Regularly

    Use the public set for development and debugging. Use unpublished questions for formal evaluation, and update them regularly to reduce the probability that they are covered by training corpora.

    5.3 Include Semantic Perturbations in Metrics

    Keep several equivalent rewrites for each question, and report the difference between the original score and the perturbed score. A small difference indicates that model capability is insensitive to surface form.

    5.4 Define Scoring Rules Clearly

    Score format, logic, and environment results separately. Format requirements should be written into the task statement and reflected separately in scoring. A single format deviation should not erase all work.

    5.5 Publish Methods and Retain Data

    Publish evaluation scripts and failure samples so others can reproduce them. Retain trajectory data for later auditing.

    6. How to Interpret Evaluation Scores

    A single total score can hardly represent model capability. When reporting evaluation results, also provide:

    • Score distribution and variance, not just the mean;
    • Stability metrics before and after perturbation;
    • Statistics broken down by error category;
    • Evaluation results close to your own business scenarios.

    For model selection and R&D decisions, in-house evaluations that match business scenarios are often more informative than public leaderboards. Public scores can serve as a reference, but their meaning must be judged together with the information above.

    7. Practical Recommendations

    • Design phase: Assume questions may leak, and plan perturbation and rotation mechanisms in advance.
    • Run phase: Set resource limits and timeouts for each case, and record complete logs.
    • Analysis phase: Attribute causes first, then draw conclusions. Avoid interpreting infrastructure problems as model capability problems.