Why Minor Specification Changes Drop LLM Scores by 20+ Points

    A single-column leaderboard score can mislead model selection. This article examines a case where two leading general-purpose LLMs, both near 74% on a long-horizon software engineering benchmark, diverged to 76.4% and 53.4% after a perturbation evaluation layered small spec constraints onto the tasks—a 23-point gap driven not by coding ability but by specification following under out-of-distribution constraints. It explains perturbation evaluation and benchmaxxing, decomposes the gap into specification capture, retention, and enforcement, and lays out a five-step process for building internal "atypical constraint" test sets scored on dual metrics (functional correctness and constraint adherence) plus first-pass rate, along with a pitfall checklist for reliable model selection and delivery acceptance.

    1. Introduction: A Few Edited Lines, a 23-Point Gap on the Leaderboard

    Public leaderboards are often the first reference in model selection, engineering-productivity assessment, and delivery acceptance. But if you lift a single column of scores and treat it as the verdict, you can easily draw the wrong conclusion. A telling case:

    • Original state: On a long-horizon software engineering benchmark, two leading general-purpose LLMs scored almost identically, both around 74%.
    • After perturbation: A team published a note on its official site and released a perturbed evaluation of the task data. The reference patch grew by only about 17 lines in the median, roughly 3.5% of the original patch. These edits landed on the requirement description or the starting code, and the functionality they were meant to deliver did not change at all.
    • Reversed result: On this perturbed leaderboard, the same models scored 76.4% and 53.4%—two models that were previously neck-and-neck now differ by 23 points.

    One model barely moved (74% → 76.4%); the other nearly halved (74% → 53.4%). The gap clearly isn’t because the task suddenly became “harder to write.” My argument here is: those twenty-odd points mostly measure not “can it write code?” but “when it meets a rule it has never seen before, will it follow it?”

    One caveat: the specific benchmark and model names in the original case change quickly with versions, so this article abstracts them into “the original leaderboard” and “leading models A/B.” What we care about is not any single ranking, but three things:

    1. Why can this kind of perturbation produce such a large gap?
    2. Why do public leaderboards become increasingly “distorted” over time?
    3. How should teams design truly reliable selection and acceptance schemes based on this?

    2. Background: Two Key Concepts

    2.1 Perturbation Evaluation: “Adding a New Rule” at the Requirement Level

    The idea behind perturbation evaluation is straightforward: take a batch of tasks that already have reference answers, make small changes to the requirement description or starting code without altering the core functionality, update the reference answers and scoring rules accordingly, and then observe how the model’s pass rate changes. Common perturbation types include:

    Perturbation type Description Example (illustrative)
    Naming and style constraints Unconventional requirements for how files, classes, functions, and variables are named All filenames must be lowercase
    Data format constraints A unified format for output, serialization, or interface fields Timestamps must all be in UTC format with a trailing “Z”
    Boundary behavior constraints Explicit requirements for handling exceptions and extreme inputs Return a unified error structure when pagination parameters are out of range
    Compatibility constraints Requirements for backward compatibility and fallback strategies Fall back to the old logic when a new field is missing
    Starting-code perturbation Changing the initial conditions so the model must adapt to the existing structure An existing module’s naming style differs from mainstream conventions

    The key point: these constraints don’t change what to build, but they do change what counts as correct. Compared with simply increasing the amount of code, this kind of perturbation is closer to the review comments and team conventions seen in real development.

    2.2 Benchmaxxing: The “Teaching to the Test” of Evaluation

    The evaluators summarized this phenomenon as benchmaxxing. You can picture the feedback loop like this:

    Questions become public → The questions and their solutions enter training corpora, or teams use them for tuning → Scores on fixed questions keep climbing → Scores are used to choose training checkpoints and to drive procurement and selection → Teams keep tuning prompts and orchestrating tool pipelines around the public questions → Scores go higher, but the gap from real tasks gets masked

    The result is that leaderboard scores increasingly look like “cramming for this one exam” rather than measuring the ability to solve unfamiliar problems. To be clear, this is not an accusation that any model is “cheating.” Public questions, corpus overlap, and targeted tuning are all normal in the industry. What it reveals is a structural risk: once a scoring signal can be predicted and optimized for, it is no longer a reliable signal of capability.

    3. The Baseline: What a “Functionality-Driven” Original Benchmark Looks Like

    To understand the 23-point gap, you first need to understand the soil it grew in. The long-horizon software engineering benchmark on which the perturbation evaluation is based was released by a research team in July 2026. Its main characteristics:

    • Original questions: 113 software engineering tasks written from scratch, spanning 91 actively maintained open-source repositories across 5 programming languages.
    • Task nature: Emphasis on long-horizon work—models must locate and complete tasks in real, multi-file, cross-module codebases.
    • Hand-written verifiers: Scoring relies on verifier programs written by hand against the requirements; a task passes as long as the required functionality is implemented.
    • Low disagreement rate: Using an independent LLM to cross-check the scoring results, the disagreement rate with human judgment was about 1.4%.
    • Comparison point: Among similar approaches, one class of benchmarks reuses the repository’s existing test cases, and its disagreement rate reaches as high as 32.4%.
    • Short prompts, large changes: The former’s prompt length is about half that of the latter, yet the reference solution requires 5.5x more code changes on average.

    Two numbers are worth chewing over.

    First, the 1.4% vs. 32.4% contrast. When scoring reuses the original repository’s tests, on roughly a third of samples the model’s judgment disagrees with human judgment. The reason isn’t hard to see: original repository tests are often bound to a specific implementation—internal function names, private attributes, a particular execution order. A correct solution that is functionally equivalent but written differently gets misjudged as wrong.

    Second, “short prompts, large changes.” The prompt often describes only “what is wanted,” leaving the implementation path entirely to the model. This means whether the requirements are understood precisely directly determines success or failure. That’s exactly the opening through which perturbation evaluation can do its work.

    4. What the Perturbation Evaluation Did

    The evaluators selected 89 medium-length functional implementation tasks from the set, and the changes were all the kind of feedback you’d see in everyday code review. Two typical examples:

    Example 1: A naming rule for report files

    • Original requirement: Generate test reports for different browsers, with report filenames shaped like “browser-name.xml.”
    • Review comment: Make all filenames lowercase.
    • After perturbation: Report filenames must be entirely lowercase—for example, Chrome.xml should become chrome.xml.

    The functionality hasn’t changed at all—you still generate one XML report per browser. But this “small rule” recurs in every piece of code that touches a filename—generation, writing, indexing, cleanup—and missing any one place can produce inconsistency.

    Example 2: The time format of interface retirement info

    • Original requirement: Add a “retirement date” field to the routes of a web framework so callers can migrate ahead of time.
    • After perturbation: Timestamps must all be in UTC format with a trailing “Z.”

    On the surface it’s just a formatting detail, but in practice it touches serialization, parsing, comparison logic, and test assertions, among other things.

    What these changes have in common: the functional semantics stay the same, the reference answers and verification logic are updated in tandem, and the model must “see and follow” the new rule. The evaluation method and task data were then released publicly under a permissive open-source license to make community reproduction and secondary verification easy.

    5. Where Do the 23 Points Come From? Breaking It Down

    5.1 First, Read the Numbers Correctly

    89 tasks means each task is worth about 1.12 percentage points, and 23 points roughly corresponds to 20 tasks flipping their verdict. Looking back at the score changes, two things stand out:

    • The leading range was already saturated. On the original leaderboard both models sat around 74%, and the second tier was clearly higher too; the top spot was just a minor fluctuation.
    • The scores did not fall across the board. One model barely changed before and after the perturbation, showing that the perturbation wasn’t about “making it harder the more you change,” but about re-exposing differences that saturation had been masking.

    5.2 Is It a Coding-Ability Gap? Most Likely Not

    The perturbation only touches requirements and starting conditions; the median increment of the reference patch is about 3.5%, not enough to change the task difficulty qualitatively. If there really were a significant coding-ability gap between the two, under longer contexts and more complex codebases, that gap should have shown up on the original leaderboard long ago. So attributing the 23 points simply to “who writes code better” lacks support.

    5.3 A More Plausible Explanation: Specification Following and Out-of-Distribution Constraints

    The questions and prompt style on the original leaderboard are public, and leading models are highly homogeneous and densely saturated there. But the perturbation introduces out-of-distribution constraints like naming rules and time formats that can’t be inferred from a “common practice” instinct—they can only be satisfied by the model genuinely reading and consistently enforcing the requirement. The difference surfaces across three stages:

    1. Specification capture: When requirements are scattered across a long context, can the model precisely locate this constraint—including supplementary notes in code comments, requirement docs, issue descriptions, and commit messages?
    2. Specification retention: Across the generation of hundreds of lines of code, can it keep this in mind without being overwritten by a “more common” way of writing?
    3. Specification enforcement: Can it apply the constraint consistently across all relevant places, rather than only fixing the first one?

    An everyday analogy: human engineers also add comments offhand while coding, and LLMs likewise tend to output the most common form on a “natural” path. “Lowercase filenames” happens to conflict with the intuition of many languages where “class names use PascalCase.” The model may well write the lowercase filename correctly at the first spot, then “by instinct” write it back to the original form in the indexing, cleanup, or logging modules.

    5.4 Interaction Mode Amplifies the Gap

    If a model is used only in single-shot code completion, the shortfall in specification following might show up as just a few individual misses. But in multi-step, automated AI coding tools, a task is decomposed into:

    Read the codebase → Formulate a plan → Modify multiple files → Run tests → Iterate on fixes based on errors

    If any link on this chain forgets the constraint, the end result can be “tests pass, but the spec isn’t followed.” This is also why, as emphasized at the start of this article, treating scores as conclusions is risky: leaderboards usually report the correctness of the final answer but rarely record how well the intermediate steps followed the spec.

    5.5 The Scoring Method’s Contribution: Why This Kind of Benchmark “Catches” Violations

    Hand-written verifiers check requirements item by item and are naturally suited to verifying format and boundary constraints; benchmarks that reuse the original repository’s tests struggle to cover such new rules. Put together, the perturbation evaluation shows such strong discriminating power.

    6. Enterprise Selection and Delivery Acceptance: Trade Leaderboard Scores for “Atypical Constraint” Tests

    6.1 Why Building Your Own “Atypical Constraint” Test Set Works Better

    Public leaderboards measure “performance on known questions,” while the business actually cares about “performance on unfamiliar requirements.” A directly executable approach: pick a handful of atypical constraints from your own repository’s real tasks, and see whether the model follows them on the first try. Its advantages are threefold:

    • Close to real scenarios: The constraints come from your own business conventions—internal naming rules, log field formats, interface return structures—so the model can’t guess right by relying on “common practice.”
    • Low cost, fast payoff: No need to build a large-scale evaluation platform; a few human-confirmed real tasks are enough to form a useful signal.
    • Reusable over time: Accumulate them into an internal regression test set that you run every time you switch models or versions, producing longitudinally comparable data.

    6.2 A Five-Step Rollout

    1. Select tasks: Pick 10–20 real tasks with clear acceptance criteria from the requirement pool, prioritizing modules that involve multi-file changes and output formats.
    2. Inject constraints: Add one atypical constraint on top of the original requirement, and make sure the reference answer and the verification script are updated in tandem—so the constraint is “clear and machine-verifiable.”
    3. Fix the environment: Standardize temperature, random seed, tool permissions, and the way prompts and context are constructed. On the first run, don’t prompt the model to “pay attention to ×× convention,” otherwise you’re measuring “whether it follows after a reminder,” not “its ability to follow on its own.”
    4. Score on two metrics: Track the functional correctness rate, the constraint adherence rate, and the overall pass rate separately, so the two kinds of problems don’t get mixed together.
    5. Attribute and review: Classify failed samples one by one—was it “didn’t see it,” “saw it but was inconsistent,” or “took the liberty of changing the design”?—and decide accordingly whether to adjust the tool pipeline, the prompt, or switch models.

    6.3 Illustration: Adding Validation Logic for Constraint Checking

    Take “lowercase report filenames” as an example; add a check on top of the existing verifier:

    1. import re
    2. from pathlib import Path
    3. # Shared with the original verifier: check whether the functionality is correct (illustrative)
    4. from verifier import functional_passed
    5. NAMING_PATTERN = re.compile(r"^[a-z0-9_-]+\.xml$")
    6. def constraint_passed(output_dir: Path) -> bool:
    7. """Validate the atypical constraint that 'report filenames must be all lowercase'."""
    8. files = list(output_dir.glob("*.xml"))
    9. if not files: # No report at all, treat the task as not completed
    10. return False
    11. return all(NAMING_PATTERN.match(p.name) for p in files)
    12. def evaluate(model_output_dir: Path) -> dict:
    13. """Overall pass = functional correctness AND constraint satisfaction."""
    14. functional = functional_passed(model_output_dir)
    15. constraint = constraint_passed(model_output_dir)
    16. return {
    17. "functional_correctness": functional,
    18. "constraint_adherence": constraint,
    19. "overall_pass": functional and constraint,
    20. }

    The key principle is “fail if either one isn’t satisfied,” rather than taking a weighted average of the two metrics. When a core constraint is violated, even if the functionality is entirely correct, for the real business it’s equivalent to not finishing the job.

    6.4 Interpreting Results: Three Typical Signals

    Functional correctness Constraint adherence Interpretation Suggested action
    High High Stable specification understanding and execution Advance to the next round of scenario validation
    High Low “Writes code, but doesn’t follow the rules” Add spec checks and auto-fix at the tool layer, or adjust the prompt
    Low High Good on understanding the spec, but limited by implementation ability or context capacity Check task complexity and codebase size; consider splitting tasks
    Low Low Tends to lose key constraints on complex tasks Use cautiously for long-horizon, multi-file tasks

    When several models score close together on your core scenarios, that’s precisely a signal: the common question types on public leaderboards can no longer distinguish them, and you need tests closer to your business.

    6.5 Make “First-Pass Rate” a Selection Metric

    • Track it separately: The proportion of samples that pass overall on the very first run, with no additional hints about the constraints.
    • Set limits and randomize: Cap the number of attempts per run and randomize the test questions to prevent models from “memorizing answers.”
    • Combine with iteration ability: Set up a separate group of “feedback-and-fix” scenarios that give a brief hint after the first failure, and observe whether the model can pass after one correction.

    Combining the two lets you distinguish “ability to follow on its own” from “ability to fix after being corrected.” The two have very different impacts on engineering productivity, yet they’re often lumped together.

    7. Pitfall Checklist

    1. Don’t treat a single score as the conclusion. With 89 tasks, each is worth about 1.12 percentage points, and the statistical noise in a single ranking isn’t small. Run multiple times and take the mean, and report both the functional correctness rate and the constraint adherence rate.
    2. Don’t “remind” the model on the first run. Prompting with “please follow the naming convention” changes what you’re measuring; such information is better placed in the feedback-and-fix stage.
    3. Make constraints verifiable and unambiguous. Vague phrasing like “lowercase where possible” or “name reasonably” only creates scoring disputes. The contrast between a ~1.4% disagreement rate for hand-written verifiers and ~32.4% for benchmarks that reuse repository tests shows that clear acceptance criteria are a prerequisite for trustworthy evaluation.
    4. Validate behavior, not implementation. As long as both functionality and constraints are satisfied, it should be judged a pass; there’s no need to require the implementation to match the reference answer.
    5. Distinguish “didn’t follow” from “didn’t blindly follow.” If a constraint conflicts with the user’s explicit intent, having the model confirm first and then act is a plus. Rule out “the requirement itself is ambiguous” before scoring.
    6. Beware teaching-to-the-test in your own evaluation. Internal test sets can also get optimized during use; refresh the questions regularly, keep them private, and watch whether the error distribution is too narrow.

    8. Conclusion

    Back to the opening question: why can slightly adjusting a task drop many models’ scores by more than 20 points?

    The answer is: what got adjusted wasn’t the difficulty, but the “specification” itself. When the questions are public and the question types familiar, leading models’ scores converge tightly; once you add an unfamiliar rule, the real gap surfaces along the “specification capture—specification retention—specification enforcement” chain.

    For developers and enterprises, public leaderboards can be a starting point, but they shouldn’t be the endpoint. Rather than arguing over which column of scores is higher, add an atypical constraint in your own repository and see whether the model follows it on the first try. If your team is building an internal evaluation system that covers many task types and multiple programming languages, you can use a general-purpose container platform to quickly spin up reproducible evaluation environments, and pair it with logging services and monitoring/alerting to continuously track how each model version performs across different constraint types. Beyond public leaderboards, owning a “ruler” of your own is a more reliable basis for selection.