How to Scientifically Evaluate a New LLM's Coding Ability: Four Failure Modes and a Reproducible Benchmark Design

    Hands-on impressions of a new LLM release tell you very little about whether it can ship real engineering work. This article explains why, then breaks down the four failure modes most worth watching in practice: attribution hallucination (blaming the environment instead of reading the first error), shotgun patching (scattered unverified edits instead of systematic diagnosis), constraint adherence failures where the code compiles but violates semantic requirements, and instability where passing once does not mean passing reliably. It then lays out a reproducible evaluation design — same codebase, same plan, same toolchain, same budget, with automated build, test and semantic gates — along with a weighted scorecard that replaces subjective impressions with scriptable criteria. It closes with practical developer habits: write the verification before the implementation, force small verified steps, keep the plan external, bound the change surface, retain full traces, and circuit-break early.

    1. First, separate the question: demo quality and engineering deliverability are different things

    Every time a new model generation ships, the community fills up with hands-on “impressions” posts. Those posts almost always fall into two categories: visual generation — a 3D rendering page, a particle or lighting effect — and single-point code completion. Both share the same three traits: the pass criteria are vague, the context is short, and the cost of failure is low. So even a mediocre model earns positive reviews as long as the output “looks reasonable.”

    The tasks developers face in real projects are a different class:

    • Multi-step, cross-file structural change — redrawing module boundaries in a large codebase, or swapping out an infrastructure implementation;
    • Hard objective gates — the compiler, static analysis, unit tests and integration tests must all pass, and failing any one counts as a failure;
    • Self-diagnosis after failure — the model has to read the error output, distinguish “my code is wrong” from “the environment is wrong,” and choose the right recovery path.

    Discussing these two classes as if they were one guarantees a distorted conclusion. A more sensible framing is to shift the evaluation goal from “can the model write something that looks good” to “can the model finish an engineering task according to a fixed plan, under hard gates.” Everything below is built around that goal.

    2. Four failure modes that deserve the most attention in hands-on testing

    2.1 Attribution hallucination: pushing its own defects onto the environment

    In one typical large-scale refactor run, this behavior chain appeared: the model moved through its plan quickly and claimed several steps were done within minutes. The build then failed. Instead of reading the first compiler error, the model produced an external attribution — “the build tool’s incremental cache is broken, which is why compilation fails” — and, when prompted, cleared the cache. It then did not rebuild to verify, and moved straight on to the next step. That step failed too, at which point the model doubled down: “the cache problem has gotten bad enough that nothing works,” and waited for human instruction.

    This is far more damaging than simply writing incorrect code, for three reasons:

    • Error output is systematically ignored. The first compiler error usually already points precisely at a file and line number. Skipping it to invent an environment hypothesis shows the model is not reading tool return values completely.
    • The attribution is unfalsifiable. Once the failure is pinned on an external factor, the model loses the trigger for self-correction and keeps looping inside the same strategy.
    • It poisons the rest of the context. Once a wrong conclusion lands in the conversation history, every later inference rests on that false premise — and even when a human corrects it, the model tends to preserve its original narrative.

    Score these as independent criteria instead of only checking whether the task eventually runs: did it read and correctly quote the error output, was the external attribution backed by evidence, and did it verify the hypothesis immediately after proposing it.

    2.2 Shotgun patching: random local edits instead of systematic diagnosis

    After being told explicitly “there is no cache problem, the code is wrong, fix it and make the tests pass before continuing,” the model switched to another failure mode: scattered edits across many places, one or two lines at a time, no verification after each edit, repeating in a loop. The end result was large-scale code pollution — changes ballooning into the thousands — with the actual problem still unresolved.

    This is a classic degradation in which the number of edits substitutes for depth of reasoning. The signals are easy to identify:

    • multiple edits within the same round with no build or test run at any point;
    • edit locations unrelated to what the error output points at;
    • edits that are trial-and-error rather than hypothesis-driven — no “I believe the root cause is X, so I’m changing Y.”

    The engineering answer is a mandatory gate inside the agent execution loop: every code write must be followed by a build and test run, and a failure sends the loop back to analysis rather than permitting another write. Also cap the number of edits per unit of time and the total edit span; once a cap is exceeded, interrupt and require a root-cause analysis.

    2.3 Constraint adherence failure: common sense and geometric constraints only partially met

    Similar and equally worrying behavior showed up in several visualization tasks: the overall visuals were acceptable, but key constraints were ignored. For example:

    • The plan and chain of thought explicitly described a particular art style; the final implementation did not reflect it at all — the reasoning and the output are disconnected.
    • In a physics-based projectile-style scene, the geometric layout makes it impossible to clear all targets in a single shot — a layout flaw at the common-sense level. The code runs; the semantics are wrong.
    • An overall palette that is too dark, making the result hard to read.
    • In a reinforcement-learning training scene, agent graphic dimensions are off and the training layer and terrain layer coordinates are not aligned, so interaction does not work properly and only a usable state emerges after repeated retries.

    The common thread is that the code compiles, but the artifact does not satisfy the task’s semantic constraints. Traditional unit tests rarely cover this class of requirement, which is why you need programmable semantic assertions folded into the automated gate: “target spacing must allow a single-shot clear,” “all entities must share one coordinate system,” “rendered text contrast must exceed a threshold,” “the training curve must converge within a given number of epochs.”

    2.4 Instability: passing once is not the same as being dependable

    To the credit of most models evaluated on the same task set, first-attempt success rates on a fixed plan were not low. But there is a wide gap between passing once and passing reliably. For engineering selection decisions, measure stability with repeated sampling rather than a single demo:

    • pass@k — at least one pass across k attempts — measures the capability ceiling;
    • pass^k — all k attempts pass — measures the reliability floor, and is much closer to real production readiness;
    • the wider the gap between the two, the more the model “can do it but isn’t steady,” and the more you need strict gates and finer acceptance criteria.

    One positive case from testing shows why this methodology matters. On a training-convergence task, a model presented the visualization clearly, encoded key values directly into the scene elements so they were easy to observe, and converged the training curve — a full pass across all criteria, which is fairly rare across multiple evaluation rounds. Such positive samples matter as much as the failure samples. They show that the difference usually is not about prose quality or demo appeal, but about constraint adherence and gate discipline.

    3. Designing a reproducible evaluation

    3.1 Fix the inputs, eliminate the variables

    The evaluation has to hold the same codebase, the same plan, the same toolchain and the same budget. A recommended setup:

    1. Pick a real, medium-sized project with complete build scripts, a test suite and static analysis configuration.
    2. Produce a step-by-step execution plan in advance — manually or with a strong model’s help — where each step states the goal, the file scope involved and the acceptance command.
    3. Every model under evaluation consumes the same plan. Allowing each model to rewrite the plan makes the results incomparable.
    4. Standardize the interaction shape: whether questions are allowed, the maximum turns per step, and the total token and wall-clock caps must all be identical.

    3.2 Automated gates instead of subjective judgment

    The gate configuration below is illustrative; adjust individual fields to the project at hand.

    1. evaluation:
    2. task: large_scale_refactor
    3. workspace: fixed_repo_snapshot
    4. plan: plan_v1.md # Same plan shared by every model under test
    5. budget:
    6. max_turns: 60
    7. max_wall_clock_min: 45
    8. gates:
    9. - name: clean_build
    10. command: build --clean # Cold-cache build so incremental cache cannot mask errors
    11. must_pass_before_next_step: true
    12. - name: lint
    13. command: lint --strict
    14. - name: unit_tests
    15. command: test --unit
    16. - name: integration_tests
    17. command: test --integration
    18. - name: semantic_assertions
    19. command: check_constraints.py
    20. policies:
    21. forbid_edit_without_verify: true # Any write must be verified
    22. max_blind_edits_in_a_row: 1 # Cap on consecutive edits with no verification
    23. require_root_cause_before_retry: true
    24. trace:
    25. save_tool_calls: true
    26. save_diffs: true
    27. save_build_logs: true

    Three of these policies map directly onto the failure modes above: forbid_edit_without_verify suppresses shotgun patching, max_blind_edits_in_a_row provides a hard interrupt, and require_root_cause_before_retry suppresses attribution hallucination. clean_build uses a cold cache so incremental caching cannot produce a false pass.

    3.3 Scorecard: turning subjective impressions into decidable items

    Dimension How it is judged Suggested weight
    Plan adherence Whether step order matches the plan, and whether skipped steps are explained 15%
    First-attempt correctness Whether the cold build and full test suite pass on the first round 25%
    Error handling Whether it reads the first error, whether the attribution is evidence-backed, whether the hypothesis is verified immediately 20%
    Edit discipline Number of unverified edits; whether the total count of changed files and lines converges 15%
    Constraint adherence Semantic assertions passed / total assertions 15%
    Stability pass^k pass rate at k = 4 10%

    The value of this table is that “does the demo look good” is no longer a scored item. It is replaced by objective indicators that can be scripted. If you genuinely need to assess visual output, convert it into assertions — layout spacing, coordinate-system consistency, curve convergence — rather than letting a reviewer score on impression.

    4. Practical engineering strategies for developers

    Even if you never run a large-scale evaluation, these practices noticeably improve how usable a new model is on engineering tasks:

    • Have the model write the verification before the implementation. Requiring the acceptance command and expected output for a step before any change drastically reduces “change it and run it” blind behavior.
    • Force small verified steps. Build and test at the end of every step and only then move on. Fix failures in place; do not let an error carry into the next step.
    • Keep the plan external. Hand the model a plan file rather than leaving the plan in the conversation. An external plan can be diffed, rolled back, and reused for side-by-side comparison across models.
    • Bound the change surface. Explicitly forbid touching files outside the plan so an opportunistic “while I’m here” refactor cannot sneak in unrelated changes.
    • Keep the full trace. Persist tool calls, build logs and diffs so that when things fail you can review exactly which step went off the rails.
    • Circuit-break early. When you see consecutive unverified edits or repeated insistence on external causes, terminate the session and start over instead of trying to correct it conversationally — the conversation history is already contaminated and recovery is expensive.

    5. Organizing evaluation infrastructure

    For organizations that need to make model selection decisions internally over the long term, build the evaluation as a repeatable pipeline rather than a one-off script:

    • Version the task set. Keep code snapshots, execution plans and assertion scripts under version control, and bind results to a task-set version number so historical conclusions stay reproducible.
    • Containerize for isolation. Run each evaluation instance in its own container with dependencies and toolchain versions pinned. Export results as build artifacts to object storage for long-term retention and comparison.
    • Schedule and parallelize. Let a container platform and task queue handle orchestration so you can run the model × repeat matrix in batch and in parallel instead of waiting on humans.
    • Instrument everything. A log service collects build output and tool-call traces centrally, and alerting catches pipeline anomalies such as rising timeout rates or sudden shifts in gate failure rates, so environment problems are not misread as model capability problems.
    • Dashboard and regression. Roll scorecard results into a dashboard. When a new model or version ships, run it against the historical task set as a regression check before deciding whether to try it on limited traffic in a real project.

    6. Conclusion

    What separates a “wow” demo at launch from something that is actually usable in engineering is precisely objective gates and execution discipline. Evaluation should shift its center of gravity from visuals and copy quality to three things: does the model read and handle tool feedback correctly, does it honor the mandatory verification loop, and does it converge reliably under constraints. For individual developers, externalizing the plan, scripting acceptance, and keeping a trace of failures improve output quality immediately regardless of which model you use. For teams, a versioned evaluation pipeline is what makes every model upgrade rest on comparable, reproducible data rather than on the impression left by a single demo.