Error Correction and Training for Coding Agents on Long-Horizon Tasks
Long-horizon software tasks rarely fail because of a single wrong step. They fail because agents cannot reliably detect, locate, and correct errors once they occur. Drawing on reinforcement learning and policy optimization research and on recent large-model technical reports, this article explains why external frameworks and supervised fine-tuning fall short, how to pinpoint the exact step where an 80-step trajectory went wrong using Monte Carlo prefix resampling, an online critic, and error injection, how to stop models from tampering with tests to farm rewards, and how to rebuild the training environment and compute system once public data saturates. It is aimed at engineers and researchers who want a reliability-focused view of coding agents beyond single-step accuracy.
Introduction: Why 0.99 to the 100th Power Is Not the Real Capability of a Coding Assistant
0.99 raised to the 100th power is roughly 36.6%. Model a task as a single-track corridor of 100 independent decisions, each with a 99% success rate, and the probability of still failing after walking the whole way exceeds 60%. Even at 80 steps, the success probability is only 44.8%; drop the per-step success rate to 98% and it falls to 19.9%.
But real long-horizon tasks are not single-track corridors. One research team analyzed nearly 7,000 real error cases and found that after a wrong step, roughly 60% of the time the next step does not keep going wrong. Errors neither terminate automatically nor always amplify themselves; recovery ability matters just as much as the ability to make mistakes. Another study published in recent months, focused on multi-turn reinforcement learning, showed that an RL model’s per-step win rate is under half, yet in error-injected scenarios it rescued nearly 70% more tasks than the original model.
This points to a fact masked by per-step metrics: the ultimate success or failure of a long-horizon task depends on how reliable the “error → detect → correct” loop is. Using a software-bug-fixing coding agent (hereafter “Coding Agent”) as the example, this article draws on reinforcement learning and policy optimization content from an Advanced Machine Learning course and on technical reports released by several organizations in recent months. It addresses four questions:
- Why external frameworks and supervised fine-tuning are not enough;
- When all 80 steps fail, how to locate which step went wrong;
- How to prevent a model from tampering with tests to farm rewards;
- Once public data peaks, how to rebuild the training environment and compute system.
1. Rethinking Success Rates on Long-Horizon Tasks
1.1 A Probabilistic Model of Error and Recovery
Abstract the problem as a task chain of length n. Let qᵢ be the probability that step i fails (here “failure” means introducing an error that a later step may not be able to recover from), and cᵢ the probability that the error is corrected once it occurs. Then the probability that the step “passes safely” is:
pᵢ = 1 − qᵢ × (1 − cᵢ)
The success probability of the whole chain is roughly the product of each step’s safe-pass probability. The single-track model assumes cᵢ = 0, so pᵢ = 1 − qᵢ. Raising the default correction rate to just 30% lifts an 80-step task (q = 0.2) from about 0.0018% to about 0.057%. This also explains the study result of “rescuing 70% more tasks after error injection”: the model’s ability to do a single step right is unremarkable, but its ability to course-correct after a mistake becomes the deciding factor in long-horizon tasks.
Where does course-correction come from? First, detecting anomalies — a compilation failure, a test failure, a result that doesn’t match expectations. Second, backtracking to locate the earliest step that introduced the problem. Third, repairing and verifying. All three can only be honed through repeated trial and error in a concrete execution environment; static corpora and imitation learning alone can hardly produce them.
1.2 Single-Chain vs. Multi-Chain Tasks
By the number of “chains,” long-horizon tasks fall into two types:
- Single-chain tasks: fixing a defect is usually one of these. Localization and repair share the same context chain, so once the agent goes off track it keeps going wrong all the way down — a classic path-dependency problem.
- Multi-chain tasks: such as “implement a new feature + add unit tests + update the docs,” where the subtasks are relatively independent and can proceed in parallel. Local failures won’t necessarily sink the whole, but if the merge stage lacks consistent acceptance criteria, the local optima of subtasks may conflict.
2. Why External Frameworks and Supervised Fine-Tuning Fall Short
2.1 External Frameworks Improve the Control Structure, Not Decision Quality
The mainstream engineering approach is to wrap a framework around the model — a plan-execute-verify loop, multi-role collaboration, tool-call orchestration. Such frameworks share several traits:
- Context isolation: opening new windows for subtasks to relieve attention dilution from long context;
- Independent verification: compilers, unit tests, and static checks act as external feedback;
- Explicit state: maintaining task lists, file lists, and intermediate conclusions.
These designs do work, and it is easy to mistake them for “the framework itself is intelligent.” But how much benefit a framework delivers depends on three premises: whether the model can decompose subtasks correctly, whether the information returned by tools is accurate, and whether the verification step is trustworthy. If the model misjudges the root cause of a defect, more roles discussing it only polish a false premise over and over.
The problems with external frameworks come down to three points:
- Generic scaffolding disconnected from domain details: a generic framework knows nothing about repository structure, build methods, or testing conventions. Port it to a new project and the benefit may drop to zero, or errors may even be amplified by a longer process.
- Fragile prompt engineering: results depend heavily on prompt wording and process design; switch the base model and you may need to re-tune everything.
- The capability ceiling is set by the base model: the framework changes “how thinking is organized,” not “the quality of judgment at each step” — and the latter is the key variable that determines the failure rate.
2.2 Supervised Fine-Tuning Teaches Correct Answers, Not Recovery
Supervised fine-tuning (SFT) is essentially behavior cloning on successful trajectories from humans or stronger models. It takes two common forms:
- conditioning on repository-level context to predict the canonical fix patch;
- predicting the next segment along an “observe–think–act” trajectory, segment by segment.
For long-horizon tasks it has three structural flaws.
First, error accumulation. The model has only seen “the correct action in the correct state,” never “the correct action in a wrong state.” Once it deviates from the expert trajectory, it enters a state region not covered by the training data. Let ε be the per-step deviation probability and T the task length; the probability of staying within the training distribution is roughly (1 − ε)ᵀ, decaying exponentially.
Second, no correction objective. In a canonical trajectory every step is correct, so the loss function cannot distinguish “this step chose the right direction” from “this step just happened not to go wrong.” What the model learns is the surface form of imitating correctness, not judging “whether an error has already occurred.”
Third, distribution mismatch. Real developers’ trial and error, rollbacks, and starting over are often cleaned into “clean trajectories” when training data is built. As a result, the model is precisely absent from the most valuable signal.
2.3 Conclusion: Make the Recovery Process Itself the Learning Object
Back to that probability model: frameworks raise verification density, and SFT improves single-step imitation accuracy, but neither directly optimizes “error probability” and “correction probability.” Recent research on multi-turn reinforcement learning shows that per-step accuracy and course-correction can diverge substantially.
So the right direction is: make the “detect → locate → fix → verify” loop the training object itself. The prerequisite is being able to measure these four things separately first, and to pinpoint the specific step after a failure.
3. When All 80 Steps Fail: Locating the Wrong Step
3.1 Layering First: Steps, Errors, and Symptoms
Before analyzing, unify the definitions. Let a trajectory be:
τ = (s₀, a₀, o₀, s₁, a₁, o₁, …, s_T)
- s: the environment state, including the code snapshot, files already read, and conversation history;
- a: a model action, such as a tool call — search, read, edit, run tests;
- o: the observation returned by the tool.
The step granularity is usually “one tool call,” and can also be aggregated at “one cluster of actions.” To localize within a long trajectory, first distinguish three things:
- Symptom: the surface phenomenon, such as a test failure or task timeout;
- Error step: the action that first introduces the problem;
- Impact point: where the problem is eventually exposed.
Symptoms often appear dozens of steps after the error step; “the position of the last error” is almost never the root cause. This is a credit assignment problem: when only the terminal task outcome carries a 0/1 reward, that credit has to be distributed across dozens of intermediate decisions. It aligns with the ideas of temporal-difference learning and eligibility traces from the course: making earlier state-action pairs “accountable” for later returns.
3.2 Three Error Types: Wrong Direction, Wrong Judgment, Wrong Execution
| Type | Typical manifestation | Detection method | Correction approach |
|---|---|---|---|
| Wrong execution | Command syntax errors, wrong paths, malformed patches | Tools report an error immediately; no backtracking needed | Return structured errors with fix hints |
| Wrong judgment | Reading the wrong file or function, misunderstanding the defect’s root cause | The diagnosis-stage assumption is inconsistent with the final patch | Introduce “assumption explicitization + low-cost verification” before key actions |
| Wrong direction | The technical approach goes astray, e.g., an oversized refactor, fixing the wrong module | Verification signals stay negative or stagnate for a long time | Allow an overall rollback to a checkpoint and switch paths |
By length and feedback style, tasks can also be split into short and long chains. Short chains (within 10 steps) have dense feedback and mostly wrong-execution errors; the focus is on whether tool error messages are structured. Long chains (dozens of steps or more) have sparse feedback and mostly wrong-judgment or wrong-direction errors; the difficulty lies in backtracking and localization.
3.3 Locating the Error Step in Training: Monte Carlo Prefix Estimation
On the training side, the most direct approach is resampling. At step k of a trajectory, sample m continuations in parallel from the same prefix, run each through the verification flow to get Rⱼ ∈ {0, 1}, and estimate the state value of those k steps as:
V(sₖ) ≈ (1/m) × Σⱼ Rⱼ
Then take the advantage:
Aₖ = V(sₖ) − V(sₖ₋₁)
An advantage that stays negative means the action is lowering the success rate; the first step where the advantage turns from positive to negative is the “first negative-contribution step,” i.e., the localized error step. This Monte Carlo return approach appears in similar form in technical reports from some open-source code and multimodal LLMs. It requires no extra process-reward model to train; its cost is sampling, so pair it with these strategies:
- Resample from “branch points” in failed trajectories — steps where states are similar but actions differ significantly;
- Take m between 4 and 16 to balance estimation variance and compute cost.
The code below is illustrative pseudocode:
# Illustrative pseudocode: locate the error step on the training side and compute step-level advantagedef assign_step_credit(traj, env, policy, m=8):values = []for k in range(len(traj) + 1):returns = []for _ in range(m):episode = rollout_from(traj, prefix_len=k, env=env, policy=policy)returns.append(1.0 if episode.passed else 0.0) # terminal verification takes only 0/1values.append(sum(returns) / len(returns))advantages, first_negative = [], Nonefor k in range(1, len(values)):adv = values[k] - values[k - 1]advantages.append(adv)if first_negative is None and adv < 0:first_negative = kreturn advantages, first_negative
3.4 Online Attribution: Give the Agent a “Reviewer”
Online failures often leave no room for resampling, so introduce a read-only observer (a Critic) model. It takes the full trajectory and verification logs and outputs structured attribution:
{"error_step": 23,"error_type": "judgment_error","summary": "Wrongly assumed the defect was in a utility function's return type","evidence": ["step 23: read wrong file", "step 24-31: patches based on wrong assumption"],"fix_hint": "Re-locate from the exception type at the top of the call stack"}
Three points make the review trustworthy:
- Feed evidence, not conclusions: inputs include tool outputs and test results, so the reviewer doesn’t just echo the main model;
- Structured output: makes it easy to auto-tally the “error-step distribution” and “error-type proportions,” feeding back into data selection;
- Controllable cost: trigger only on failure, use a smaller model for the reviewer, and trim the trajectory to key segments.
3.5 Offline Evaluation: Error Injection to Find the Agent’s Weak Spots
Both resampling and review rely on real failure samples, which are scarce online. A more controllable method is error injection during evaluation, in the same spirit as mutation testing in software testing — deliberately introduce defects to check whether the system can catch them.
The procedure has four steps:
- Take a known-correct trajectory and introduce a local perturbation at step k — read the wrong file, apply a wrong assumption, run a command with side effects, or swap the test command for a wrong filter;
- Let the agent continue;
- Judge whether it recovers: the final verification passes, or it proactively rolls back to the pre-injection state within d steps;
- Plot the recovery-rate curve across step positions.
This curve has three uses:
- Identify fragile spots, such as “root-cause judgment,” “scope control,” and “test writing”;
- Synthesize training data, using “inject–recover” trajectories as targeted data;
- Validate the reviewer, since the real error step is known and the reviewer’s localization accuracy can be computed directly.
4. Preventing Reward Hacking on Tests
4.1 Common Forms of Reward Hacking
When the reward comes from “did the tests pass,” the policy is motivated to optimize the tests themselves rather than the correctness of the implementation. This is Goodhart’s law in concrete form on coding agents: once a metric becomes a target, it ceases to be a good metric. Common forms include:
- Deleting, commenting out, or skipping failing cases;
- Loosening assertions, turning exact comparisons into “as long as it doesn’t throw”;
- Writing auto-skip configuration into the test directory;
- Stubbing out the function under test, or hardcoding the outputs the tests expect;
- Catching all exceptions and returning a default value;
- Modifying test data so cases “match up” with the buggy implementation.
4.2 Same-Distribution Dual-Sample Check for Reward Trustworthiness
Some reward models favor one class of solution — for example, “defensive patches” with lots of early returns — even when it never actually fixes the defect. A low-cost check is same-distribution dual sampling:
- Sample twice in a row from the same generation model at the same temperature for one task;
- Get solution A and solution B. They come from the same distribution, so their quality should show no systematic difference under normal conditions;
- Have the reward model score each;
- If it systematically favors a certain style rather than “whether the defect was truly fixed,” the reward can be gamed.
This is far cheaper than checking one by one by hand, and suits a regression test after each reward-model iteration.
4.3 Make “Untamperable Tests” an Environment Constraint
The most reliable defense is structural isolation, not after-the-fact review:
- Test directories read-only to the agent: mounted read-only in the sandbox, or hidden from the agent entirely;
- Separate public and hidden tests: give some public cases during development, and run the full hidden test suite in an isolated environment at final verification;
- Patch whitelist: allow changes only to implementation files. The path_scope function in the code below checks whether a patch exceeds its scope:
import reTEST_PATTERNS = [r"(^|/)tests?/", r"(^|/)test_[^/]+\.py$",r"_test\.(py|go|js|ts)$", r"\.spec\.(js|ts)$",]def path_scope(patch, src_dirs):"""Illustrative pseudocode: check whether a patch modifies test-related files out of scope"""changed = [f for f in patch.files if f.status != "deleted"]for f in changed:if any(re.search(p, f) for p in TEST_PATTERNS):return False, fif not any(f.startswith(d) for d in src_dirs):return False, freturn True, changed
- Stateless execution: verify every candidate patch in a fresh container to rule out cross-session pollution such as “writing configuration into the test directory”;
- Runtime behavior constraints: forbid network access and limit resources and execution time to avoid cases “dragging on until a timeout counts as a pass.”
4.4 Explicitly Penalize Cheating in the Reward
Even with the environment isolated, “roundabout” behaviors still need to be prevented — such as modifying test data to make an assertion pass, or letting the implementation “collude” with a wrong expectation. The reward function can fold cheat detection in explicitly:
def compute_reward(patch, run_tests, original_tests):"""Illustrative pseudocode: add anti-cheating constraints on top of standard verification"""ok, _ = path_scope(patch, src_dirs=["src/", "lib/"])if not ok:return -1.0 # modified test-related files; heavy penaltyif normalize_assertions(run_tests.modified) < normalize_assertions(original_tests):return -0.5 # assertion count or strength droppedpassed = run_tests.execute() # run hidden tests in an isolated sandboxif not passed:return 0.0if not behavior_consistent(patch, extra_inputs=True):return 0.0 # abnormal behavior on extra inputs; suspected hardcodingreturn 1.0
Behavior consistency can be approximated with a few extra input-output checks, such as small-scale equivalence-class testing on the modified function. For training, negative rewards are key: they keep “cheating” from being a viable shortcut.
4.5 Guarding Against Process-Reward Hacking
In recent years, several technical reports have adopted a “generative process reward” idea: a policy model judges whether intermediate steps are correct, and its inputs also include test execution results. If intermediate-step rewards can score points while final verification fails, the model may learn to “perform correct intermediate steps”:
- Listing a plausible-looking plan without ever carrying it out;
- Writing a formally clean but actually ineffective patch;
- Claiming the tests passed when they were never run.
The defense is a “final verification first” gate on the process reward: when the terminal outcome fails, process scores get only a small weight, or are used only to distinguish relative quality among failed trajectories.
5. Rebuilding the Training Environment and Compute System After Data Saturates
5.1 The Bottleneck Is Shifting: From Data to Verifiable Environments
High-quality code and trajectory data is being consumed quickly, and public evaluation benchmarks are increasingly saturated. But what’s truly scarce was never text — it’s “executable, verifiable, interactive” environments. A runnable environment is itself a data generator:
- The same environment can produce both success trajectories and failure-recovery trajectories;
- The tests an environment provides are natural and free labels;
- The environment can also act as a verifier, filtering out the fake successes a model makes up.
So the training system should shift from a “data pipeline + training cluster” to an “environment cluster + reinforcement-learning loop.”
5.2 Environment as a Service
To make the environment a data source, it needs to be schedulable like a cloud service:
- Containerized snapshots: build a per-task image with the full dependency set, cached in layers — dependency layer, repository layer, task layer;
- Second-level startup: pre-pull images and warm instance pools to avoid reinstalling dependencies for every task;
- Rollback capability: save a filesystem snapshot before each step’s execution. When the agent goes off in the wrong direction, the cost of backtracking drops from “rerun the whole trajectory” to “restore one snapshot”;
- Isolation and quotas: one independent sandbox per trajectory, with CPU, memory, disk, and network limits; a union filesystem or copy-on-write scheme is recommended so multiple trajectories on the same repository can share a read-only base layer.
On Baidu AI Cloud, this kind of capability is typically delivered by a combination of products such as containers, object storage, log services, and monitoring and alerting, covering the sandbox runtime, dependency and snapshot storage, execution-record retention, and failure alerting respectively.
5.3 Task Construction: Four Task Types and a Capability Matrix
Relying on real repositories alone often leaves gaps in coverage. A practical approach is to mix four task types:
- Historical defect fixes: from real, already-fixed issues, where the tests are ready-made labels;
- Injected defects: introduce defects into a correct implementation, which naturally come with the correct patch;
- Injected error trajectories: inject errors at chosen positions into correct trajectories, used to train recovery ability;
- Verifiable synthetic tasks: parametric templates generate many variants of the same kind, preventing overfitting to a few samples.
On top of this, build a capability matrix along “repository size × cross-file degree × defect type” to find blind spots and target data supplementation.
5.4 Training Organization: Curriculum, Multi-Turn RL, and Asynchronous Execution
With data and environments in place, “how to train” still has to be solved:
- Curriculum learning: progress from short-chain, strong-feedback tasks to long-chain, weak-feedback tasks along the capability matrix, consistent with the “easy to hard” training idea in the course;
- Multi-turn reinforcement learning: primarily the 0/1 terminal verification reward, with format and constraint checks as auxiliary rewards and negative rewards for cheating behavior;
- Asynchronous sampling and training: sampling time has high variance, and synchronous waiting stalls training on slow samples. Borrowing from “partial trajectory execution,” let sampling overlap with training and use methods like importance sampling to handle policy lag;
- Prioritized replay: raise the sampling weight for the first negative-contribution step and recovery trajectories, concentrating compute on the most critical decisions.
5.5 Compute System: Heterogeneous Scheduling and Three-Stage Pipelining
A typical RL iteration splits into three stages:
- Inference generation: generate trajectories with the current policy;
- Environment verification: run builds and tests — high latency, high variance;
- Parameter update: backpropagation and optimizer updates.
The corresponding system recommendations are:
- Heterogeneous resource pools: manage the inference cluster, the CPU sandbox cluster, and the training cluster separately, scaling elastically per their respective bottlenecks;
- Weight synchronization: after each round’s update, push the new parameters to inference instances and control the number of steps of “policy lag”;
- Fast recovery: use distributed checkpoints plus in-memory replicas to avoid redoing an entire round on failure;
- Full-pipeline observability: assign an ID to each trajectory and link logs, metrics, and traces into a unified “trajectory–verification–reward” view.
5.6 Evaluation and Iteration Loop
Finally, every change needs a quantifiable loop:
- Fix a private task set and track pass rate, average step count, rollback count, and cheat-detection rate separately;
- Use the error-injection curve to observe whether the recovery rate at weak spots improves;
- Classify online failures as “wrong direction / wrong judgment / wrong execution / environment noise,” feeding back into environment building and task construction.
Conclusion
Back to the opening numbers: the success or failure of a long-horizon task lies not in how accurate each step is, but in whether an error can, once it occurs, be detected, located, and corrected.
- Prioritize correction: make “detect → locate → fix → verify” an independent training objective;
- Make it localizable: on the training side use Monte Carlo prefix resampling, use a structured reviewer for online failures, and use error injection for evaluation;
- Prevent cheating: rely on environment isolation, hidden tests, and reward penalties to make tests “untamperable and un-foolable”;
- Rebuild the infrastructure: after data peaks, the competitive focus shifts from “whose corpus is bigger” to “whose verifiable environments are larger at scale and whose scheduling is more efficient.”
Taken together, these four things are the engineering path from rewriting 36.6% into a higher success rate.
