Why LLM Coding Agents Keep Failing on False Root Causes
A controlled study that runs several coding agents against the same codebase and task plan reveals that their real engineering ability is uneven, and so are their failure modes. This article breaks down five common failure patterns—fabricated root causes, shotgun debugging, presentation-layer hallucination, missing common-sense constraints, and layer/state misalignment—and maps each one to a specific gap in an agent's capability stack. It then offers concrete countermeasures: turning attribution into a falsifiable check, enforcing single-variable debug loops, translating vague visual goals into assertable conditions, and asserting domain invariants at build and test time. It closes with a reproducible evaluation methodology—layered tasks, quality gates, and process metrics—plus infrastructure guidance. The core takeaway: what separates a strong agent from a lucky one is whether it asks questions, attributes with evidence, and converges after failure.
1. The Phenomenon: Capability Isn’t Uniform, and Neither Are Failures
Running several coding agents repeatedly under a controlled setup—the same codebase and the same task plan—is a common way to assess their real engineering ability. What you tend to see in practice is a split: some agents genuinely pass on the first try when requirements are clarified and steps are advanced one by one, delivering implementations of comparable quality; others look very “busy” yet never converge.
The problem with the latter group isn’t the volume of code they produce—it’s how they fail. Grouped by type, the failures fall into roughly five categories, each corresponding to a different layer of the model’s capability stack:
| Failure type | Symptom | Missing capability |
|---|---|---|
| Fabricated root cause | Attributes its own code errors to a corrupted build cache, environment variables, or tool defects | Evidence-driven attribution |
| Shotgun edits | Trial-and-error repeated across thousands of locations without converging | Hypothesis management and single-variable validation |
| Presentation-layer hallucination | The style/visual effects promised in the plan never actually land | Turning goals into assertable conditions |
| Missing common-sense constraints | Geometry and physics common sense get violated | Awareness of domain invariants |
| Layer and state misalignment | Runs but is unusable, or crashes at runtime | Coordinate-system and state-machine consistency |
One thing deserves credit: on visualization tasks, mapping quantitative training metrics—state-value tables, per-episode reward curves—directly onto the interface and actually making the training converge is something many models can’t do. This shows that “visualization” and “faithful presentation of training state” are two independent dimensions: the first tests rendering, the second tests correct understanding of the algorithm’s state.
2. Failure Mode 1: Why Models Favor “Environmental” Explanations
When an agent says “the build won’t compile because the build cache is corrupted—please help me clear the cache,” it usually hasn’t verified anything. The appeal of this kind of explanation is that environmental causes can’t be falsified by the model itself, so they legitimately end self-scrutiny and toss the problem back to the user. Clearing the cache, reinstalling dependencies, switching tool versions, suspecting a compiler bug—these all belong to the same family of “low-cost escapes.”
The engineering response isn’t adding a line to the prompt that says “don’t pass the buck.” It’s turning the attribution process into a falsifiable check:
1. Minimize the reproduction: keep a single test case, remove all other inputs2. Disable incremental builds: run one full, cache-free recompile3. Print the complete error chain: first error first, ignore downstream cascading errors4. Branch decision:- Error disappears under the minimal case + full build -> attribute to build state, may continue- Error reproduces stably -> environmental causes are ruled out; must localize to a specific file and line5. Prohibited: without executing steps 1-3, do not output any environmental conclusion
The key is the ordering: prove the environment innocent first, then you’re allowed to talk about the environment. This process doesn’t require a stronger model—only binding each conclusion to evidence.
3. Failure Mode 2: Why Shotgun Debugging Never Converges
“Change a line here, change a line there, and it still won’t compile” is a textbook gradient-free random walk. The search space is every code location, the feedback signal is a single binary result (pass/fail), and nothing tells the model which edit actually helped. Once the number of edits reaches the thousands, the codebase is effectively contaminated, and the rollback cost can exceed that of a rewrite.
A controllable debugging loop should satisfy three constraints:
- Single variable: each round allows only one semantic change, validated immediately after it’s made;
- Hypothesis list: maintain an explicit list where each hypothesis is tagged “confirmed / refuted / to be verified,” and re-attempting a refuted hypothesis is forbidden;
- A single stopping condition: the target test suite fully green—not “looks close enough.”
For teams, there’s another external constraint: the diff for each round of edits must be reviewable. When a single task’s changes span dozens of files and no one can explain their intent, that task’s output should be treated as invalid, not “nearly done.”
4. Failure Mode 3: Goal Drift and “Presentation-Layer Hallucination”
One class of failure is especially subtle: during reasoning, the agent explicitly states it will implement a certain visual style (an ink-wash texture, ancient-style wood grain), the deliverable even looks the part, but the promised style never lands. The reason is simple—adjectives are generated language, not verified states.
The fix is to translate the adjectives in the plan into assertable conditions:
| Vague goal in the plan | Verifiable assertion |
|---|---|
| “Render an ink-wash effect” | Background texture uses the specified asset files; the primary palette hits the given set of color values; screenshot similarity to the reference image is above a threshold |
| “The interface should have hierarchy” | Foreground/background luminance difference ≥ threshold; minimum font size ≥ threshold |
| “Ancient-style wood grain” | Material parameters come from the agreed config; solid-color textures are prohibited |
An often-overlooked usability issue follows from this: a dark interface that’s “pitch black, can’t see anything” isn’t a matter of taste—it’s an accessibility defect caused by insufficient contrast. Writing a contrast floor into the acceptance criteria is far more efficient than arguing after the fact about whether it “looks good.”
5. Failure Mode 4: Common-Sense Constraints and Layer Misalignment
In a 3D scene, “the pin spacing is too wide, so you can’t clear them all in one shot” is a textbook case of a missing domain invariant: the model completed the rendering and collision, but never checked whether the scene parameters satisfy the task’s semantics. By the same token, “the agent’s icon takes up half the maze,” “the maze isn’t drawn,” “the agent’s action layer isn’t aligned with the terrain layer,” and “you can’t play it interactively” all belong to the same category—layout and coordinate-system constraints weren’t expressed explicitly.
def check_grid_scene_invariants(scene, cfg):cell = scene.cell_size # world units per cellerrors = []# 1. The agent's size must not occupy more than one cellagent_ratio = scene.agent_width / cellif agent_ratio > cfg.max_agent_ratio:errors.append(f"agent size out of bounds: {agent_ratio:.2f} cells")# 2. The action layer and terrain layer must share the same coordinate transformterrain_origin = scene.terrain.grid_to_world(0, 0)agent_origin = scene.agent_layer.grid_to_world(0, 0)if distance(terrain_origin, agent_origin) > 1e-6:errors.append("layer origins inconsistent; independent coordinate systems exist")# 3. Each walkable cell must have exactly one aligned entity anchorfor cell_id in scene.walkable_cells():if not scene.agent_layer.has_anchor(cell_id):errors.append(f"missing anchor: {cell_id}")# 4. Input events must be able to drive the state machineif not scene.input_map.is_complete(cfg.required_keys):errors.append("input key mapping incomplete; scene is not playable")return errors
The value of this check is that it turns “looks wrong” into a hard failure that can be caught at build and test time, rather than something you only discover after a manual playthrough.
6. Accepting Training Tasks: Convergence Isn’t the Finish Line
Take Q-learning in a grid world. A common mistake is to look only at “did it find a path in the end.” A more complete acceptance pass should cover three layers:
- Correct state representation: the number of states and the action space match the grid size, with no out-of-bounds states;
- Observable training process: sliding-average per-episode reward curves, TD error, and state visit counts can all be exported;
- Verifiable convergence and policy: the sliding-average reward fluctuates less than a threshold within a set window, and the greedy policy can reach the goal within a limited number of steps.
If the interface can map the value table onto the floor tiles, plot the per-episode reward as a live curve, and the curve genuinely converges, that’s already a fairly high level of completion. But convergence and “correct presentation” still need to be asserted separately, otherwise you easily end up with “the curve looks great, but the data is fabricated.”
7. A Reproducible Evaluation Method for Coding Agents
Distilling the constraints above into a process, a four-layer structure is recommended:
Task tiers
- L1 single function: clear inputs and outputs, unit tests as acceptance;
- L2 module: cross-file refactoring, acceptance includes regression tests;
- L3 small system: multi-step plan execution, acceptance includes integration tests;
- L4 visualization/interaction: rendering, layout, interaction, acceptance includes invariant checks and screenshot comparison.
Quality gates
Gate 0 Full build passes (disable incremental cache as an excuse for attribution)Gate 1 Unit tests + regression tests all greenGate 2 Domain invariant assertions pass (geometry, layout, alignment, key bindings)Gate 3 Manual spot check: screenshots compared against the visual goals in the plan
Process metrics (as important as result metrics)
- Number of clarifying questions: projects that genuinely understand the requirements should raise key questions along the way; zero questions is often a signal of insufficient requirement understanding;
- Failure recovery rounds: the number of retries needed to go from “build failed” to “all green”;
- Ineffective edit count: the proportion of changes that didn’t improve test results;
- Attribution accuracy: the proportion of cases where the error type the model claims matches the actual root cause;
- Cost per task: time and token consumption, used to compare the real cost of “passing on the first try” versus “two hours of trial and error.”
Attribution accuracy is the metric that best separates “strong” from “lucky.” An implementation that occasionally passes but keeps fabricating environmental reasons when it fails will, in any long-running project, almost inevitably cause rework.
8. Engineering Implementation and Infrastructure
The evaluation needs to run in isolated environments at scale and in parallel, and you can assemble general-purpose cloud capabilities to do it: object storage holds task packages, code snapshots, and artifacts; container services provide an independent, disposable sandbox for each run; and logging and monitoring/alerting aggregate build logs, test results, and process metrics into a traceable timeline. Baidu AI Cloud offers a mature product combination for these general capabilities (containers, object storage, logging, and monitoring) that can be used to build such an evaluation pipeline, though the specific choices should still be based on the team’s isolation strength and concurrency scale.
Four companion practices:
- Contracts first: before handing work to the agent, write down the acceptance criteria and domain invariants clearly, rather than arguing about them afterward;
- Closed-loop feedback: builds, tests, and screenshots must feed back to the agent automatically, otherwise it can only guess;
- Limit the edit radius: enforce small, incremental commits so a single task’s change scope is reviewable and rollback-able;
- Tiered intervention: L1/L2 fully automated, human confirmation at key nodes for L3, and mandatory human acceptance of look-and-feel and interaction for L4.
9. Conclusion
The gap between coding agents rarely shows up in “can it write code that runs.” It shows up much more in three things: whether it’s willing to ask questions, whether it attributes with evidence, and whether it converges after failure. Fabricated root causes and shotgun edits are both fundamentally symptoms of a missing falsifiable feedback mechanism, while gaps in visual presentation and domain common sense are the result of requirements never being translated into assertable conditions.
So the right way to evaluate these tools isn’t to collect “impressive” demo clips—it’s to build an evaluation process that controls variables, is repeatable, and has hard gates, and to put process metrics and result metrics on the same dashboard. When the acceptance criteria are clear enough, the differences in model capability surface naturally, and teams can judge accordingly: which parts can be confidently handed to automation, and which must retain human judgment.
