Small Changes, 20-Point Score Gaps: The Hidden Pitfalls of Code Evaluation for LLMs

    An experiment that reran 89 coding tasks with tiny modifications to the instructions and starter code—such as requiring lowercase filenames or UTC timestamps—caused several leading models to drop nearly 20 points. This article explains why small constraints amplify differences in code benchmarks, breaks down the failure modes behind the score drops, and offers practical takeaways for building more discriminating evaluations and writing clearer engineering specifications.

    1. First, the Numbers: Only the Problems Changed, Not the Models

    On November 7, a group of researchers published a controlled experiment with “the same set of problems, written two ways.” The method was straightforward: they picked 89 “medium-duration” feature-implementation tasks from a mainstream software engineering benchmark, changed only a small part of each task’s instructions or starter code, and then had 15 model configurations run every task in both its original and modified forms.

    How small were the changes? Two examples show the scale.

    In the first, the original task asked for a separate test report for each browser; the modified version added a single sentence: “filenames must all be lowercase.” So a name like Chrome__Headless_.xml had to become chrome__headless_.xml, with the casing rule carried all the way through the generation logic, the references, and the downstream validation.

    In the second, the original task added a “retirement date” field to a web framework’s route; the modified version required timestamps to use UTC with a Z suffix, no longer preserving the timezone offset passed in by the caller.

    These read like two quick comments a senior engineer might scribble after reviewing a first draft: no architecture change, no language switch, no new business requirement.

    2. Did the Stance Become the Answer? Two Sets of Numbers Say It All

    (Unit: %, original data)

    Model configuration (alias used in this article) Original Modified Difference
    Flagship A 73.8 53.4 -20.4
    Lightweight fast B 74.2 55.6 -18.6
    Flagship C 74.2 76.4 +2.2
    Mid-size D 69.0 56.7 -12.3
    Flagship E (previous generation of the same series) 73.6 63.5 -10.1
    General-purpose F 72.7 69.1 -3.6
    General-purpose G 74.1 71.9 -2.2

    The two configurations that fell the hardest were originally just a fraction of a point behind the top score, essentially neck and neck. After the modified version, each dropped nearly 20 points. The two that dropped the least fell only 2 to 4 points and moved clearly up the ranking. The single “rise against the trend” among all 15 configurations came from the latest version of a flagship series.

    The overall statistics are even more telling than any single configuration:

    • The top packed tightly together: In the original version, 6 configurations clustered around 74%, within 1.5 points of each other. At conventional evaluation confidence levels, this gap is essentially indistinguishable. After the modification, they spread out across 23.0 points.
    • The total span nearly doubled: The gap between the highest and lowest scores across the 15 configurations widened from 20.9 points to 48.3 points.

    This comparison suggests that the so-called “same tier” in the original version was largely an illusion created by problems that were too easy and lacked discriminating power. Once the constraints were tightened slightly, the differences that had been hidden came into sharp focus, and who ranked where was almost entirely rewritten.

    3. Why Constraints Amplify Differences: Three Mechanisms

    The patch added only 17 lines, yet its impact was enormous, because these constraints hit several systemic weaknesses of coding models.

    1. Pattern matching is good at “roughly,” not “exactly.” When large models write code, they are essentially learning from and reusing common patterns found in vast amounts of code. A common practice in mainstream web frameworks is to preserve the caller’s timezone offset when serializing time, producing a string like 2026-03-18T09:30:00+08:00. “The timestamp must use UTC plus a Z suffix” looks almost identical but means the opposite. Here the common practice is actually wrong, and a model that follows its training distribution will land exactly on the constraint it should have avoided.

    2. Attention is pulled by “big words,” and detail constraints get diluted. In a long requirement, “implement some feature” and “make the tests pass” are prominent, headline-level terms, while “lowercase filenames” is just an inconspicuous footnote. The longer the instructions, the more easily such details get compressed away. This mirrors the implicit “everyone already knows” assumptions people make when writing requirement documents: what feels obvious to the author may not be visible to the reader.

    3. Cross-stage transmission amplifies a single oversight. Naming and format constraints rarely land in just one place. Take “lowercase filenames”: it can involve

    • the report generation logic;
    • the reading and assertions of files in the tests;
    • examples in the documentation;
    • error messages.

    If any one place keeps the old convention, it causes self-test failures or mismatched results. The more downstream steps there are, the higher the chance that a single oversight gets amplified stage by stage.

    Coding tasks are usually scored on “whether all reference tests pass,” so one unimplemented constraint can zero out an entire task. This explains why a “small change” buys a “big gap.”

    4. The One Configuration That Rose: Can We Rule Out “Behind-the-Scenes Scoring”?

    The easiest thing to suspect is whether “someone is giving points to one particular vendor.” The most powerful rebuttal comes from two products in the same family.

    In the original version, the previous generation and the latest version of a flagship series scored 73.6 and 74.2, just 0.6 apart, within the benchmark’s own confidence interval, with no clear winner. On the modified version, they became 63.5 and 76.4, a 12.9-point gap. Two models from the same lineage went in opposite directions on a stricter test; if someone really were “giving points to a vendor,” it would be hard to explain this split within a single family.

    A more reasonable explanation is that this threshold tests not “whether it can write code” but “whether it can implement each constraint exactly as stated.” This ability improved markedly across version iterations but was precisely masked by the easier problems in the original version. Of course, a single experiment cannot prove that a benchmark is absolutely neutral, but it reminds us: when judging a benchmark’s credibility, first check whether it can distinguish real differences within the same family.

    5. Trajectories Are More Worth Reading Than Scores: How the Model Actually Broke Down

    Scores can only tell you that “it broke”; trajectories tell you “why it broke.” Reviewing the full interaction, the following failure modes are most worth watching (generalized from the common patterns of such tasks).

    • Write first, patch later: The first half smoothly builds the main functionality and passes self-tests, and only at the final stage does the model remember the filename or timezone constraint, forcing large-scale rework that quickly eats up the time budget.
    • Fix one thing, break another: The model modifies only the most visible generation logic and misses old conventions in test assertions or documentation examples. It looks complete but still fails to meet the requirements.
    • Reverse alignment: Aiming to “pass the original tests,” the model actively works around the new constraints—for example, by continuing to preserve the timezone offset to stay compatible with old assertions.
    • Old-experience interference: Following common framework conventions or the previous problem’s approach, the model directly reuses old templates at the point where the constraint was rewritten.
    • Accumulated multi-step deviation: An incorrect assumption made midway goes uncorrected, and every subsequent step is built on top of it, drifting further off with each fix.

    These patterns point to the same root cause: models already have considerable “can produce code” ability; the weakness lies in “producing it accurately”—precisely following constraints, maintaining consistency across stages, and detecting and correcting deviations in time. This is also why replacing subjective scores with automatically verifiable hard rules widened the gap so quickly.

    6. Takeaways for Engineering Teams: Both Evaluation and Development Must “Read the Fine Print”

    On the evaluation side: turn “modifying problems” into a routine capability.

    • Use dual-track scoring: Report both the core-capability score and the constraint-perturbation score, and additionally give the difference between the two, i.e., “constraint robustness.” A configuration that scored near-perfect on the original version is not necessarily as reliable in the face of real requirements.
    • Follow the single-variable principle: Change only one dimension at a time—for example, naming, timezone, error codes, or log format—so results remain attributable.
    • Report discrimination and confidence intervals: When top configurations cluster within 1.5 points, treat their ranking as a tie rather than as distinct positions.
    • Fully preserve trajectories: Log the entire process from requirement understanding and code generation to test execution, so you can pinpoint whether the failure was a misunderstanding, an execution slip, or insufficient verification.
    • Introduce same-family comparisons: Watch how scores change across version iterations to check whether the evaluation is genuinely effective.

    On the development side: write implicit conventions as explicit rules.

    In requirements, API documentation, and code-review checklists, explicitly specify naming conventions, time and timezone formats, character encoding, error-code structure, and log fields, and back them with unit tests and static checks, so that requirements like “lowercase filenames” and “UTC timestamps” are validated automatically at commit time rather than surfacing only after release.

    For putting this into practice, teams can use a cloud platform’s standardized capabilities to build an evaluation pipeline:

    • use a container platform to fix the runtime environment;
    • use object storage to archive trajectories;
    • use a message queue to schedule tasks;
    • use log and monitoring services to track anomalies.

    On cloud platforms such as Baidu AI Cloud, these capabilities can be combined on demand, so teams do not have to build the infrastructure from scratch.

    7. Conclusion

    A 17-line reference patch, a single lowercase rule, and one timezone format are enough to push many models down by more than 20 points, turning “neck and neck” into “clearly behind.” This exposes a common weakness of current coding models in following constraints, and it also reminds us that a few points on a leaderboard may not reflect the full picture of real capability.

    For developers, instead of asking “which model ranks first,” it is better to ask “which kinds of requirements is it most likely to get wrong.” The answer often hides in those two comments that look so trivial.