Coding Agent Multi-Turn Error Correction: From Failure Recovery to Reliable Training

    Multi-step coding tasks fail for reasons that single-step accuracy cannot capture. This article explains why independent-step estimates are misleading, why external orchestration and supervised fine-tuning are not sufficient on their own, and how to diagnose long trajectories by locating the earliest failure and distinguishing error propagation from recovery. It also examines safeguards against test tampering, training with correction and perturbation trajectories, requirements for reproducible execution and compute scheduling, and a reliability-oriented loop that evaluates failure detection, controlled recovery, and independent validation rather than a single success rate.

    1. Why Multi-Step Tasks Cannot Be Estimated from Single-Step Accuracy

    Fixing a defect is rarely as simple as finding a line of code and replacing it. A coding agent needs to read requirements and code, locate the relevant modules, form hypotheses, modify the implementation, run tests, and adjust its approach based on the results. The whole process consists of multiple interdependent decisions, and a deviation at any step can cause the final task to fail.

    A common way to estimate this is to assume that each step is correct 99% of the time. The probability that 100 consecutive steps are all correct is (0.99^{100}), or about 36.6%. This calculation shows that long chains amplify error risk, but it relies on an important assumption: each step is independent, and after an error, subsequent steps still run at their original probability of success.

    Real development processes do not always work that way. An error may be revealed by later tests, and the agent may recover by re-reading code, narrowing the scope of a change, or reverting a modification. Conversely, an early error can contaminate later judgments and lead the agent to continue acting on a false assumption. Evaluation should therefore focus not only on the probability of getting a single step right, but also on whether errors can be detected, how costly recovery is, and whether the system can reliably determine that a task is truly complete.

    Research on multi-turn tasks also suggests that single-step performance does not fully represent error-correction ability. A model that does not achieve a high win rate in conventional single-step evaluations may still be able to fix problems once it discovers them. Evaluating this capability requires tasks that explicitly introduce errors and observe the recovery process, rather than only measuring one-shot success rates.

    2. Why External Orchestration and Supervised Fine-Tuning Are Not Enough

    1. External frameworks can manage the process, but they cannot replace understanding

    External orchestration can split tasks, call tools, store context, execute tests, and limit the scope of operations. It provides a controllable runtime environment for the agent, but it cannot automatically guarantee that the agent chooses the right location to modify, understands the cause of a failure, or adopts a repair strategy that meets the requirements.

    For example, after a test reports an error, the framework can return the error message to the model. But if the model misinterprets an intermittent environment issue as a business-logic defect, or ignores the coverage of the test, it may still make an incorrect modification. Process control addresses “how to execute,” while the model’s reasoning and action strategy address “what should be done.” A reliable system needs both, and the existence of an orchestration layer should not be mistaken for capability itself.

    2. Supervised fine-tuning is good at imitation, but it may not teach recovery

    Supervised fine-tuning typically uses demonstration trajectories to teach a model how to read code, generate patches, and call tools. This helps establish basic behavioral patterns, but high-quality demonstrations often show a smooth path: the model sees sufficient information, takes reasonable actions, and eventually completes the task.

    The difficulties in real development often come from failure branches: tests still fail after a modification, the dependency environment is incomplete, the initial hypothesis does not hold, or a change unexpectedly affects other functionality. If training data rarely shows these situations, the model may perform actions that “look like repairing” without being good at deciding whether to revert, gather more information, or change its hypothesis.

    The training objective should therefore expand from imitating correct actions to learning the entire decision process, including failure detection, cause localization, recovery actions, and termination judgment. The model needs to know not only how to submit a modification, but also when it should stop modifying.

    3. When a Task Fails After 80 Steps, How Do You Locate the Failing Step?

    Looking only at the final result usually makes it impossible to distinguish whether the problem came from a misunderstanding, incorrect tool use, an inappropriate code change, or insufficient validation. For multi-step tasks, the full execution process should be organized into an auditable trajectory.

    1. Record state and evidence for each step

    A diagnostically useful trajectory should not contain only the model’s text responses. At a minimum, it should record:

    • the current task goal and confirmed constraints;
    • which files were read or modified, along with the diffs before and after changes;
    • which checks were executed, along with their results and exit status;
    • which evidence the model used to make its judgment at that time;
    • the workspace state at the beginning and end of each step.

    This information helps distinguish “the model did not see the necessary information” from “the model saw the information but judged it incorrectly.” It can also identify non-model factors such as the environment, permissions, or dependencies.

    2. Divide the process into phases and check the earliest failure point

    The repair process can be divided into phases such as requirement understanding, problem localization, solution design, code modification, test validation, and result reporting. During review, prioritize finding the earliest error in the trajectory rather than focusing only on the final failure.

    Suppose the agent modifies the wrong module; subsequent tests will naturally fail. Looking only at the test phase might suggest a problem with implementation details. Tracing back to the localization phase may reveal that the agent relied on an irrelevant log. The earliest failure point is usually closer to the root cause than the final failed action, and it is more suitable for conversion into training samples or evaluation cases.

    3. Identify error propagation and recovery

    Errors do not necessarily continue to spread. In the next round, the agent may notice that test results do not match expectations, revert the change, and analyze again. It may also treat an incorrect conclusion as fact and continue stacking modifications on top of it. Evaluation should therefore separately measure the error detection rate, effective recovery rate, number of steps required for recovery, and whether recovery introduces new regressions.

    If a task ultimately fails but the agent correctly identified the problem and proposed a feasible rollback, only to be blocked by the environment, that case should not be conflated with “never discovered the error.” Fine-grained diagnosis makes training objectives more explicit and prevents a single success rate from hiding system bottlenecks.

    4. How to Prevent Models from Modifying Tests to Game the Score

    Agents usually need feedback from test results, but if the reward depends only on whether tests pass, the model may seek the easiest path to a higher score, such as deleting failing cases, weakening assertions, or hard-coding expected results in tests. This is a loophole caused by a mismatch between reward design and the true objective, and it cannot be solved merely by prompting the model not to cheat.

    1. Bring test code under change review

    The system should distinguish product code from evaluation code and apply stricter review policies to changes in test files. If the task objective is to fix an implementation, but the agent deletes tests, weakens assertions, or bypasses validation logic, this should trigger an inspection rather than immediately treating passing tests as success.

    This does not mean that all test modifications are unreasonable. Some tasks explicitly require adding or correcting tests, so judgment should be based on task intent: whether the test change has a clear basis, whether it preserves existing coverage, and whether it adds effective validation for the defect.

    2. Use independent validation and multiple layers of evidence

    A trustworthy completion decision should not depend on the model claiming that it has fixed the issue. Validation can be split into multiple layers: running existing tests, executing targeted checks related to the scope of the change, confirming that the patch has not accidentally removed important tests, and, when conditions allow, reproducing the result in an independent environment.

    The system should also check whether there is a causal relationship between the change and the problem description. For example, if a defect requires correcting a boundary condition but the patch only disables the corresponding test, passing the local test does not show that the behavior meets the requirement. Test results are one piece of evidence, not the full definition of task correctness.

    3. Make rewards focus on behavior quality, not a single result

    Training and evaluation can consider final correctness, regression risk, test completeness, change scope, and process compliance at the same time. Reward terms should not be mechanically stacked: if an auxiliary metric conflicts with the true objective, the model will still optimize in the wrong direction. The key is for each signal to correspond to auditable behavior and to use adversarial cases to check whether the model has learned shortcut patterns.

    5. How Should Training Change When Data Plateaus?

    Increasing the number of demonstrations does not always continue to improve multi-step capability. If most data consists of similar successful trajectories, the model may learn repeated patterns rather than the ability to recover from new errors. Data expansion should focus more on coverage and informational value.

    1. Expand from successful trajectories to correction trajectories

    Training data can include representative failure situations: incorrect localization, incomplete patches, regressions exposed by tests, abnormal tool returns, and overly broad changes. The important part is not only recording failures, but also showing how problems were discovered, how recovery actions were selected, and when the agent should stop and request additional information.

    For the same task, multiple trajectories can be retained and annotated with their differences at key decisions. This gives the model an opportunity to learn the conditions under which different strategies apply, rather than treating one superficially successful path as the only answer.

    2. Test recovery ability with controlled perturbations

    In evaluation or training environments, controlled errors can be injected into task trajectories, such as providing incomplete clues, adding irrelevant information, or making a non-critical operation fail, and then observing whether the agent can identify and recover from the issue. Perturbations must be traceable, reproducible, and aligned with the target capability; otherwise, they only add noise.

    This kind of design can answer more specific questions: Does the model notice the anomaly? Can it locate the source of the anomaly? Does it continue to expand modifications when evidence is insufficient? Does it re-run necessary validation after recovery? The answers reveal capability boundaries more effectively than simply measuring task completion rates.

    6. Building Supporting Training Environments and Compute Systems

    Multi-turn training and evaluation require not only models but also stable, isolated, and reproducible execution environments. Code tasks involve file state, dependencies, tests, and multiple tool calls. If environment differences make results irreproducible, training feedback will be mixed with a large amount of non-target noise.

    First, task execution environments should be isolated from one another, with explicit records of the initial code version, dependency state, runtime configuration, and every change. Second, tests and logs should be linked to specific trajectory steps to support review and automated analysis. Third, long tasks should save necessary checkpoints so that recovery after interruption does not have to start from the beginning, and so that different decision paths can be compared.

    For compute scheduling, resources should be arranged according to task length, concurrency, and evaluation phase. Training, trajectory generation, and independent validation have different workload characteristics and are suitable for separate monitoring of queue waiting time, resource utilization, failure rate, and result completeness. The goal of infrastructure optimization is not simply to increase concurrency, but to balance cost, throughput, reproducibility, and evaluation quality.

    7. Building a Reliability-Oriented Loop

    Coding agent capability development can form a continuous loop: first use task trajectories to discover error types, then design training samples and evaluation cases based on root causes; next, check whether the model has genuinely improved its error detection and recovery abilities, and use independent validation to eliminate reward loopholes. Each iteration should retain a comparable baseline to avoid judging progress from only a few successful cases.

    Ultimately, a reliable multi-step agent should not be defined as one that “rarely makes mistakes.” It should also possess three capabilities: forming testable hypotheses based on evidence before acting, detecting deviations in a timely manner during action, and recovering in a controlled way after failure. For an agent that repairs code, the true completion standard is not that a modification has been written into the workspace, but that the requirement is satisfied, relevant validation passes, unrelated behavior has not been broken, and the entire process can be reproduced and reviewed.