AI Coding Agent Reliability: How to Verify Multi-Step Engineering Work

    An AI coding agent can modify several files and report that a plan is complete, but that report is not evidence that the changes work. Using a mid-sized systems programming refactor as its setting, this article examines common failure modes, including false completion, misdiagnosed build failures, and shotgun edits. It explains how builds, targeted tests, and verified checkpoints can make progress observable and help teams decide when to intervene.

    1. From Code Completion to Multi-Step Execution

    Early coding assistants focused on tasks such as completing a function, generating a snippet, or answering a specific question. The scope was small enough for developers to review the output immediately. An AI coding agent can now follow a multi-step plan, modify several files, and run builds and tests. That changes how developers need to verify the work.

    Multi-step tasks have three characteristics:

    • Steps depend on one another. An error in an early step can carry over into later ones.
    • Different steps need different evidence. Some can be checked by the compiler, others require unit tests, and some need a developer to inspect the behavior directly.
    • The execution environment has state. Build caches, dependency versions, and temporary files can affect whether a result is reproducible.

    An agent’s final status report is therefore not enough. To evaluate its work, check what it verified at each step and how it responded when a check failed.

    2. Common Failure Modes

    Long-running engineering tasks tend to fail in recognizable ways. Knowing what to look for helps teams design checks before those failures occur.

    1. False completion. The agent marks a step complete without running the build or tests. A quick turnaround can hide the fact that it edited the code but skipped verification.
    2. Misdiagnosis. When compilation fails, the agent blames the cache, a path, or a tool version before inspecting the code it just wrote. That sends debugging in the wrong direction.
    3. Shotgun edits. Instead of isolating the root cause, the agent makes small changes in several places, hoping one will fix the build. This can introduce new errors and make the diff harder to review.
    4. Context drift. As the task progresses, the agent loses track of earlier constraints, such as interface signatures, naming conventions, or modules that must not change.
    5. Failure to pause. When requirements are ambiguous or a fix fails repeatedly, the agent keeps going instead of explaining the problem and asking for guidance.

    These failure modes are not unique to any particular tool. Treat them as predictable risks of multi-step automation and build checks into the workflow.

    3. How to Troubleshoot Compilation Errors

    Compilation failures are particularly easy to misdiagnose. Use a consistent troubleshooting sequence to separate code defects from environment problems.

    1. Read the full diagnostic and locate the first error. Later errors are often consequences of the first one. Fixing it may clear several diagnostics at once.
    2. Use a clean rebuild to distinguish code from environment issues. If the same error recurs and points to a specific code location, investigate the code first. Treat it as an environment issue when the failure disappears after cleaning the build artifacts or when the diagnostic clearly identifies an issue such as a missing dependency or a version mismatch.
    3. Make the smallest change that addresses the first error, then rebuild. Changing one thing at a time lets you tell whether that change resolved the failure.

    The following pseudocode shows how to enforce build and test checks for a step:

    1. def execute_step(step):
    2. build_result = build_project()
    3. if not build_result.ok:
    4. record(step.id, "build_failed", build_result.first_error)
    5. return False
    6. test_result = run_tests(step.required_tests)
    7. if not test_result.all_passed:
    8. record(step.id, "tests_failed", test_result.failures)
    9. return False
    10. record(step.id, "verified", None)
    11. return True

    The step is marked as verified only after both the build and the required tests pass. If either check fails, the recorded diagnostic gives developers a starting point for investigation.

    4. Make Completion Verifiable

    A verification workflow turns “done” into a state the team can check, rather than a claim it has to take on trust.

    • Define acceptance criteria for each step. Specify what must be true: the code compiles without errors, the required tests pass, or a particular interface returns the expected result.
    • Enforce the order of operations. After an edit, build first, then run the tests. Only then should the workflow advance to the next step. Use a script or task orchestrator to enforce that order rather than relying on the agent to remember it.
    • Commit verified checkpoints. Create a version control commit after each verified step. If a later step fails, the team can return to the last known stable state without manually undoing a large set of changes.
    • Limit the scope of each change. Identify which files or modules a step may modify. Require human approval for changes outside that scope.

    A task description can include a checklist like this:

    1. [ ] The build passes with no new warnings.
    2. [ ] All tests required for this step pass.
    3. [ ] All modified files are within the scope defined in the plan.
    4. [ ] A checkpoint has been committed, and the commit message includes the step ID.
    5. [ ] No interface signatures unrelated to this step have changed.

    5. When the Agent Should Stop and Ask for Help

    Automating execution does not mean removing developers from the process. Set explicit conditions under which the agent must pause:

    • The same error persists after two attempts to fix it.
    • A requirement has more than one reasonable interpretation.
    • The number of files changed in a single step exceeds a preset limit.
    • A test result differs from the expected result, and the reason is unclear.

    When any of these conditions is met, the agent should report the current step, the fixes it has already tried, and what it thinks is the most likely cause. This gives the developer enough context to decide what to do next without reconstructing the entire run.

    6. How to Evaluate AI Coding Agents

    To compare agents fairly, give each one the same plan, codebase baseline, and verification criteria. Record the following measures:

    Measure What to record
    First-pass verification rate The proportion of steps that pass verification on the first attempt
    Human interventions How often a developer has to answer a question or correct the agent
    False completion count How often the agent reports completion when the step does not pass verification
    Change size The number of files and lines changed, which can help flag shotgun edits
    Response to failure Whether the agent isolates the cause or tries changes without a diagnosis

    A single run says little about consistency. Repeat the same task several times to see how much the results vary.

    Match the evidence to the type of task. A polished interface or 3D scene can look correct even when the interaction logic is wrong or the numerical results are inaccurate. For tasks such as reinforcement learning training, use quantitative indicators such as convergence curves and changes in reward values rather than relying on visual inspection alone.

    7. Putting the Workflow Into Practice

    1. Make each step independently verifiable. Steps that are too large make failures harder to isolate; steps that are too small add coordination overhead. One independently verifiable piece of functionality is a useful unit of work.
    2. Script the checks. Combine the build, tests, and static checks into a single command or pipeline to reduce manual work and avoid skipped checks.
    3. Merge large changes in batches. Keep each batch buildable and testable so that rolling it back is less costly.
    4. Treat agent output as a draft. Developers should still read and approve changes to critical paths.

    Multi-step automation is valuable because it reduces repetitive work, not because it replaces engineering judgment. Clear acceptance criteria, an enforced verification sequence, and explicit stop conditions give teams a practical basis for deciding how much work to delegate.