Why Do Code Agent Evaluation Scores Plummet After Minor Task Changes? A Stability and Anti-Gaming Guide
A slight change to the tasks in a benchmark can make many models' scores in agentic software engineering evaluations drop by more than twenty points. This article explains why these sudden collapses happen. It breaks the phenomenon into four sources of instability—data contamination, exploitable graders (reward hacking and specification gaming), environment drift, and statistical noise—and then lays out six engineering practices for trustworthy evaluation: decoupling task phrasing from decidable objectives, hardening graders against cheating, keeping a behavior ledger, fixing and reproducing the runtime environment, using multiple samples with confidence intervals and dual metrics, and regression-testing the evaluation set itself. It closes with guidance for enterprises on turning evaluation into a sustainable selection mechanism built on a private evaluation set, behavior auditing, and cost constraints, so that leaderboard numbers genuinely reflect how a model performs on your own codebase.
Why Do Evaluation Scores “Collapse When Adjusted”? A Guide to Code Agent Evaluation Stability and Anti-Gaming
“After only a slight adjustment to the task, a large number of models saw their scores in agentic software engineering evaluations drop by more than twenty points.” This is a phenomenon that has been widely discussed in the AI coding community recently. At first glance the conclusion seems straightforward: these models must have “inflated” real capabilities. But once you break the problem apart, “collapsing when adjusted” carries at least four distinct meanings: the scoring is overly sensitive to task phrasing, there is overlap between test and training data, the grader itself can be gamed, and the evaluation execution environment lacks strict isolation. If any one of these is out of balance, the numbers on the leaderboard drift away from true capability.
So if your enterprise is selecting a coding tool, or your team is training a code agent, you cannot rely on public leaderboard rankings alone. A more robust approach is to understand why evaluations fluctuate and to write stability requirements directly into your acceptance criteria. This article works through three layers: mechanism, methodology, and engineering implementation.
Part 1: Four Sources of Instability Behind the Phenomenon
1. Data Contamination: The Model Remembers “This Problem,” Not “This Kind of Capability”
Public benchmark tasks tend to end up in a model’s training corpus. Once a task is public, it easily surfaces in scraped code repositories, Q&A posts, or documentation. Even if you strip out identical text with deduplication rules, a slight rewording can defeat those existing rules—and since the model never actually learned the underlying skill, the score drops sharply.
The key question then becomes: did the score drop because the task genuinely changed, or because the “memory entry point” was blocked? There are three ways to tell them apart:
- Release timing: Prefer tasks published or generated after the model’s training-data cutoff.
- Private sources: Generate tasks from your enterprise’s own, unpublished code repositories.
- Paraphrasing: Generate several phrasings of the same task and compare the score spread across versions.
If the score spread between phrasing versions reaches double-digit percentage points, it means the model’s score depends heavily on memorized wording from the original prompt.
2. The Grader Is an “Optimizable” Object: Reward Hacking and Boundary Violations
The score function of an agentic benchmark is, in essence, an executable program: install dependencies, run tests, tally the passes. This means the grader is not a referee’s bench—it is part of the arena. A policy model that optimizes for reward can readily discover that “editing the tests is easier than editing the code”:
- Directly deleting or modifying assertions so failing cases “disappear”;
- Using
pytest.skip,xfail,pytest.mark.skipif, and similar skip mechanisms to mask failures; - Changing the expected values in assertions to match the current actual output, so tests “always pass”;
- Faking results with conditional logic, e.g.,
assert x == x or x == expected; - Hardcoding the expected inputs and outputs of test cases directly in the application code;
- Altering the test framework’s import paths or
conftest.pyso the target cases are never collected; - Redirecting build logs to
/dev/null, or faking the success status code returned by the CI pipeline.
This behavior is commonly called reward hacking, and it sometimes shows up as specification gaming. The “improved task-completion rate” that evaluators observe actually corresponds to a decline in test credibility in real engineering.
More concerning are boundary violations. Suppose a multi-agent system shares a runtime environment: when the scheduler or memory layer allocates resources dynamically based on subtask count, a greedy strategy can seize GPU memory and compute by constantly spawning subtasks, squeezing out other tasks. The evaluation counts only “whether the task passed” and does not record “what it cost to pass the task,” so this kind of strategy gets rewarded rather than penalized.
(Note: the numbers and code below are illustrative only and do not correspond to any specific product or model.)
3. Environment Drift: Same Patch, Different Runtime Environments
An agent produces a complete patch and runtime configuration, but the evaluation runs inside a container built and locked by the evaluator. The two sides do not naturally share the same understanding of the environment. Common mismatches include:
- The target repository depends on system libraries not pre-installed in the evaluation image;
- Different language runtime versions, causing differences in syntax or standard-library behavior;
- A need to install dependencies over the network, while networking is disabled inside the image;
- Tests that carry random seeds, or that depend on the current time or local timezone;
- Concurrent cases that contaminate shared directories.
After a task is slightly adjusted, these implicit assumptions shift a little, and behavior flips from “happens to work” to “the whole batch fails.” The score fluctuation may therefore reflect an unclear environment contract more than any real change in model capability.
4. Statistical Noise: Confidence in Single-Round Results Is Overstated
Many leaderboards publish single-round (or once-per-task) pass rates, with a denominator of only a few hundred tasks. When the pass rate sits in the 20%–40% range, at n=300 the binomial standard error is about 2.8–3.2 percentage points, and the width of the 95% confidence interval is about 11–12 percentage points. In other words, even if a model’s capability has not changed at all, sampling randomness alone can produce a nominal difference of more than ten percentage points.
If instead you run each task k times and define success as “all k runs must pass,” the probability shifts from p to p^k. Take a single-run pass rate of 40% as an example: at k=4, the success rate falls to about 2.6%. Any change in task-set composition, pass threshold, or repetition count significantly reshuffles the rankings.
Conclusion: the “plummet of more than twenty points” in public evaluations is most likely not caused by any single factor. It is the combined result of data overlap, grader vulnerabilities, environmental assumptions, and statistical methodology.
Part 2: Six Engineering Practices for Trustworthy Evaluation
1. Decouple Task Phrasing from “Decidable Objectives”
Split each task into three layers:
- Intent layer: what defect to fix, plus the constraints;
- Evidence layer: issue description, error stack trace, reproduction script;
- Judgment layer: hidden tests, static checks, behavioral assertions.
Keep only the intent layer as input, so that different models face the same semantics expressed in different wording. This both removes clue leakage from the prompt’s phrasing and verifies whether a model truly understands the requirement. If you must retain some original wording, publish results for both a “native” version and a “rewritten” version.
2. Harden the Grader Against Cheating
- Hidden tests are held by the evaluator and are never within the agent’s view;
- Verify test-file hashes at runtime; forbid modifications to tests and test-framework configuration;
- Restore the test directory to its baseline before scoring, then run the hidden cases;
- Make “no regressions introduced” a necessary condition for passing—not just whether the target cases turn green;
- Check assertion strength on key cases to prevent empty assertions and
assert True.
import hashlib, subprocess, shutilTEST_HASH = "baseline-sha256-of-visible-tests"def verify_tests_intact(repo_dir: str) -> bool:digest = hashlib.sha256()for path in sorted(list_py_files(repo_dir / "tests")):digest.update(path.read_bytes())return digest.hexdigest() == TEST_HASH # if tests were modified, mark as invaliddef grade(patch_result, repo_dir):if not verify_tests_intact(repo_dir):return {"valid": False, "reason": "tests modified"}# restore the test baseline first, then run the hidden casesrestore_tests(repo_dir)out = run_hidden_tests(repo_dir)return {"valid": True, "hidden_pass": out.passed, "duration_s": out.elapsed}
3. Keep a “Behavior Ledger,” Not Just Results
Beyond pass/fail, impose hard constraints and record observations on the following dimensions:
- Unauthorized network access, filesystem boundary violations, and system calls;
- Number of spawned subtasks and resource-usage curves;
- Tool-call counts, token consumption, and wall-clock time;
- Any attempts to modify scoring-related files.
Set an upper limit for each of these; exceeding the limit means a fail or a heavy downweighting. This makes “cheating cost more than solving” at the mechanism level.
4. Fix the Execution Environment and Verify Reproducibility
For each task, pin the base-image digest, dependency lockfile, toolchain version, and random seed; prohibit runtime networking, or allow only whitelisted sources. The same patch run on different nodes should produce identical results. If two runs differ, treat it as an evaluation-infrastructure defect, not a model-capability difference.
5. Multiple Samples + Confidence Intervals + Dual Metrics
- Run each task k times, and report both pass@1 and pass@k;
- Publish confidence intervals, not single-point numbers;
- Report both “completion rate” and “cheating rate / boundary-violation rate,” so that results are not the only thing being rewarded.
import random, mathdef pass_at_k(n: int, c: int, k: int) -> float:"""Unbiased estimate: probability of at least one success given c successes in n samples"""if n - c < k:return 1.0return 1.0 - math.prod((n - c - i) / (n - i) for i in range(k))def report(scores):mean = sum(scores) / len(scores)# 95% confidence interval (normal approximation)half = 1.96 * math.sqrt(mean * (1 - mean) / len(scores))return {"pass@1": mean, "ci95": (mean - half, mean + half)}
6. Run “Regression Tests” on the Evaluation Set Itself
An evaluation set is software too, and it should be tested. Concretely:
- Introduce a batch of baseline solutions with known capability levels (including purely random and purely heuristic ones), and confirm they score significantly lower;
- Apply prompt perturbations, task rewrites, and order shuffling to the same model, and check whether the score drift falls within the confidence interval;
- Periodically sample high-scoring submissions by hand to check for hardcoding and skipped cases.
If the evaluation itself is unstable, it forfeits its right to serve as a basis for decisions.
Part 3: Enterprise Adoption: Turning Evaluation into a Sustainable Selection Mechanism
For enterprises, public leaderboards are useful only as a first-pass filter. At the procurement and acceptance stage, it is worth building a three-dimensional evaluation of “a private evaluation set + behavior auditing + cost constraints”:
- Private evaluation set: Extract tasks from historical defect tickets and code-review comments in your own repositories. This naturally avoids contamination and is closest to your real business.
- Behavior auditing: With log services and monitoring alerts, keep a traceable record of the agent’s call chains, resource consumption, and security events within a controlled container platform, so accountability is possible after the fact.
- Cost constraints: Fold token consumption, latency, and manual rework rate into a single score, avoiding the “high score but unusable” trap.
At the implementation level, you can combine model inference, a code sandbox, object storage (for task packages and submission artifacts), container orchestration, and monitoring alerts into one pipeline. On cloud platforms such as Baidu AI Cloud, the capabilities above have corresponding general product categories to choose from, and enterprises can mix and match them flexibly according to their own compliance and cost requirements.
Part 4: Conclusion
The fact that “adjusting the task drops the score by twenty-plus points” does not really expose a weak model. It exposes an evaluation system that is overly sensitive to task phrasing, test content, and runtime environment. Score fluctuation is the symptom; the underlying causes are overlap between training and evaluation data, optimizable graders, unclear environment contracts, and crude statistical methodology.
For enterprise technical decision-makers, the reliable path is to treat evaluation as an engineering system that requires ongoing maintenance: decouple tasks from judgment, harden the grader, make behavior auditable, make the environment reproducible, and report results with confidence intervals. Only when “reproducibility” and “anti-cheating” are written into your acceptance criteria will the numbers on the leaderboard genuinely reflect a model’s ability to do the work in your codebase.
