Does Higher Single-Step Accuracy Make Multi-Step Tasks More Reliable?
This article challenges the common intuition that high single-step accuracy translates into reliable multi-step completion, using the bug-fixing coding agent as a running example. It explains why external agent frameworks and supervised fine-tuning hit a ceiling, shows how to attribute failures across long trajectories, lays out layered defenses against reward hacking, and describes how to restructure training environments and compute systems once verifiable data plateaus. The central takeaway is that long-horizon agent competitiveness comes from recovering after errors, not from being accurate on every single step.
Introduction: Where Does 0.99 to the 100th Go Wrong?
0.99 to the 100th power is roughly 0.366. If every step is correct 99% of the time and a task requires 100 consecutive steps, the final success rate drops to just over 30%. This calculation is often used as a warning: a high per-step accuracy decays rapidly inside long tasks.
But the formula holds only under three assumptions:
- Any single error fails the task: no rollback, no retry, and no chance to correct once an error is discovered.
- Steps are independent of one another: no single class of error gets amplified or repaired across the remaining ninety-nine steps.
- The number of steps is fixed and known: every successful path takes exactly the same number of steps.
Real tasks rarely satisfy these conditions, and at least bug-fixing is not one of them. One research group analyzed nearly 7,000 real-world error samples and found that after a step goes wrong, the next step does not continue the error in about 60% of cases. Several publicly released multi-round reinforcement learning studies from recent months add another data point: on a batch of code-repair benchmarks, a multi-round RL model’s single-step win rate is below 50%, yet its overall pass rate is markedly higher than the supervised fine-tuning baseline; and when errors are deliberately injected into trajectories, it rescues nearly 70% more tasks than a version that did not undergo this training.
These numbers shift the focus of optimization from “how to avoid errors” to “how to get back on track after an error.” This article combines material from:
- Foundational frameworks taught in advanced machine learning courses at universities;
- The training roadmap disclosed in the technical reports of an open-source large model series;
- Publicly released papers from recent months.
Using the bug-fixing coding agent as a concrete scenario, it breaks down four questions:
- Why are external frameworks and supervised fine-tuning not enough?
- If the task fails at step 80, how do you locate which step went wrong?
- How do you prevent the model from editing tests to game the score?
- Once data plateaus, how should the training environment and compute system be restructured?
1. Why External Frameworks and Supervised Fine-Tuning Are Not Enough
1.1 Model the task as a “road network,” not a “one-way street”
The process of fixing a bug is closer to a feedback loop:
- The agent reads the error, reproduces the problem, and locates the suspicious code;
- It proposes a fix and edits the files;
- It compiles, runs tests, and runs static checks to get execution results;
- If the results pass, it wraps up; if not, it returns to step 2 and re-decides.
If you model the whole process as a Markov decision process, you can separate it into three things:
- State: the current contents of the codebase, the history of actions taken so far, and the most recent execution output.
- Actions: reading files, searching symbols, editing code, running commands, rolling back to a checkpoint, and submitting the result.
- Feedback: whether compilation passed, whether tests passed, and whether static checks raised warnings. These are objective signals that do not depend on the model’s self-assessment.
The “one-way street” assumption effectively deletes these three things, keeping only actions and demanding that a single path succeed on the first try. Under that assumption, the training objective naturally converges to “make every step as correct as possible.” Against this objective, both external frameworks and supervised fine-tuning can help, but each has a ceiling.
1.2 External frameworks: they enable “being able to turn back,” not “wanting to turn back”
Common coding agent frameworks usually provide capabilities such as:
- Task decomposition and plan management;
- Tool-calling interfaces and sandboxed execution;
- Context trimming and history summarization;
- Retry strategies and failure fallbacks.
These mechanisms are genuinely valuable: they let errors be detected and retried, engineering the “one-way street” into a “road network with branches.”
But what a framework can do is give the model the conditions to turn back, not make the model decide to turn back. There are three behaviors a framework struggles to obtain reliably through prompting alone:
- Knowing it is wrong: when execution fails, the model may stick to its original judgment, repeatedly tweak the same spot, or even claim the problem is “already fixed.”
- Knowing where it went wrong: faced with a compilation error buried in noise, the model may keep changing an unrelated file.
- Knowing when to turn back: the marginal benefit of grinding on one step is sometimes lower than rolling back three steps and redoing with a different approach.
Worse, the framework itself can easily introduce “another kind of one-way street.” Take a fixed pipeline like “locate → modify → test → fix,” where every step must succeed. If no rollback is designed for a failed step, the framework will amplify a local failure into a global one.
Summary: the framework decides whether the agent can turn back; capability decides whether it will turn back, and turn back correctly.
1.3 Supervised fine-tuning: never taught “what to do after making a mistake”
The learning signal for supervised fine-tuning (SFT) comes from high-quality trajectories that are human-labeled or filtered. These trajectories share two traits:
- They rarely make mistakes, or the erroneous portions were trimmed during data cleaning;
- Each trajectory has only one canonical answer, so the model learns to “reproduce this path.”
This leads to three problems:
- No recovery samples: the model barely gets to learn the chain “error → recognize → roll back → fix.”
- Error accumulation: imitation learning suffers from distribution shift. Each small per-step deviation gradually pushes the agent into states it never saw during training, and the further along it goes, the less it knows what to do.
- Overconfidence bias: the actions in the training data all “work,” so the model tends to overestimate its success rate, skip verification steps, or ignore failing tests.
On top of this, an SFT-trained baseline is not necessarily “better at fixing bugs” than a general-purpose base model. In the multi-round RL study mentioned above, the fine-tuned baseline clearly trails the RL model in overall pass rate, and a major reason is that SFT learns to “imitate correctness,” whereas repair tasks demand “recovering within errors.”
1.4 The root cause of multi-step trajectories: a misaligned optimization objective
Coding-task rewards are usually given only at the end: 1 if all tests pass, 0 otherwise — the so-called sparse reward. If training optimizes only for this end-state signal, the model may learn not to “walk the path steadily and correctly,” but rather:
- To gamble on a single step, collapsing entirely when the gamble fails;
- To favor “quick and easy” paths, when real tasks often require understanding first, then verifying, then modifying;
- To lack the tendency to “turn back when it drifts mid-way,” because the training signal never once rewarded “turning back.”
The blind spots of the two approaches can be laid side by side:
| Comparison dimension | External framework | Supervised fine-tuning | Multi-round RL |
|---|---|---|---|
| Problem it solves | Provides tools, a sandbox, and retry mechanisms | Learns standardized action formats and basic workflows | Learns “how to walk the full trajectory” based on task outcomes |
| Main weakness | Can only retry; cannot judge when to change course | Lacks supervision for “recovering after errors”; suffers error accumulation | Credit assignment over long trajectories is hard; training is costly |
| Relation to recovery capability | Provides the physical conditions for recovery | Rarely covers recovery scenarios | Can directly optimize “recovering from errors” |
1.5 Code sketch: first make “turning back” a legal action
If an agent can only “walk forward,” then even the strongest model is just a faster one-way street. The sketch below does two things:
- Implements “rollback” as a first-class action;
- Explicitly records checkpoints in the execution loop.
from dataclasses import dataclass, field@dataclassclass Step:"""A single step in the trajectory, used for post-hoc analysis and building training data"""action: dict # action type and arguments, e.g., edit file, run tests, rollbackobservation: str # execution result, e.g., compilation error, test outputsignals: list = field(default_factory=list) # objective signals: compilation / tests / static checkscheckpoint: int | None = None # checkpoint index before executing this actiondef run_agent(task, agent, sandbox, max_steps=80):history, checkpoints = [], {}for i in range(max_steps):# Objective signals first: if a step suddenly deteriorates, prefer rollback over piling on more editsif history and history[-1].signals == ["failed"] and i > 1:history.append(Step(action={"type": "rollback", "target": i - 1},observation="Previous step degraded significantly; rolling back and redoing"))sandbox.restore(checkpoints[i - 1])continueaction = agent.decide(task, history) # model decisionif action["type"] == "submit":return "success" if verify(task, sandbox) else "failed", historycheckpoints[i] = sandbox.snapshot() # keep a checkpoint before executingobservation = sandbox.apply(action) # execute inside the sandboxhistory.append(Step(action=action, observation=observation,signals=classify(observation), checkpoint=i))return "failed", history
The point of this code is not the implementation details but the structure:
- Rollback is a legal action: the model can undo the previous step at any time, instead of only moving forward.
- Every step has a state snapshot: after a failure you can both retry and perform the attribution analysis covered in the next section.
- Objective signals are explicitly classified: compilation, test, and static-check results enter the history directly as signals, without passing through the model’s interpretation.
With a “rollback-capable” execution environment in place, the next question becomes: how do we make the model willing to turn back, and accurate when it does?
2. It Failed at Step 80: How to Locate the Error
2.1 Why attribution is hard
When a task fails at step 80, the only signal is the end-state “did not pass” — one of the hardest problems in long-trajectory training and evaluation: sparse reward and difficult credit assignment.
- You cannot simply blame the last step: a hidden defect introduced at step 79 may only surface at the final step.
- Nor can you simply blame the first step: step one may be entirely correct, and it may be step 40 that deviated from the direction it established.
- It is usually a stack of small deviations: each step looks “reasonable” in isolation, but together they lead the agent off the correct track.
So the goal of attribution is not to find “the one step that went wrong,” but to find the first decision point that pushed the trajectory off track — the earliest suspicious step.
2.2 Method 1: a structured evidence chain
Record each step as a structured object, such as “action, observation, signals, checkpoint,” so that post-hoc analysis has something concrete to work from. Two points deserve special attention:
- Recording only the final output loses intermediate information; the command executed, the failure reason, and the elapsed time should all be preserved.
- Store objective signals separately from the model’s self-report. The model saying “I’ve fixed it” is only its judgment; a test result showing a pass is the fact.
2.3 Method 2: objective signals first — distinguishing a “real fix” from a “fake pass”
Replay the 80-step trajectory step by step and check whether the signals after each step satisfy these invariants:
- Compilation passes;
- The number of tests has not decreased;
- All target tests relevant to the task pass;
- No existing regression tests newly fail;
- The scope of changes does not exceed expectations.
Any step that breaks one of these invariants is a highly suspicious candidate. For example, if the number of passing tests drops sharply after a step, it often means tests were skipped or deleted — precisely the “score-gaming” signal discussed in the next chapter.
2.4 Method 3: rollback-based binary-search localization
Using “does the task ultimately pass” as the criterion, run a binary search over the trajectory:
- Split the trajectory into a front half and a back half;
- Roll back to the split point and replay the back half;
- If it passes after rolling back, the problem lies after the split point; otherwise it lies at or before the split point.
Sketch code:
def localize_first_bad_step(sandbox, trace):"""Binary search: find the first step that pushed the trajectory off track"""if verify_task(sandbox, replay_all(trace)) is True:return None # already passes, no attribution neededlo, hi = 1, len(trace)while lo < hi:mid = (lo + hi) // 2sandbox.restore(trace[mid].checkpoint) # roll back to the state before the midpointok = replay_from(sandbox, trace, start=mid) # replay from the midpoint to the endif ok:lo = mid + 1 # problem is after the midpointelse:hi = mid # problem is at or before the midpointreturn lo # the earliest suspected error step
A few prerequisites for using this:
- Replay must be reproducible: tool-call results must be deterministic. For operations that depend on the network or the clock, pre-record replays or make them deterministic in advance; otherwise the binary-search result is untrustworthy.
- Replay needs a cost budget: the binary search takes about ⌈log₂ 80⌉ ≈ 7 replays. For very long, slow-executing trajectories, use objective signals to narrow the range first, then binary-search within the candidate interval.
- The criterion is “whether the task passes”: if the original trajectory already passes, there is no error to correct.
2.5 Method 4: error classification — treat the right ailment
Attribution should not stop at “which step went wrong”; it must also answer “at which step, and which class of error.” The table below lists common error types in the bug-fixing scenario:
| Error type | Typical manifestation | Remedy |
|---|---|---|
| Localization error | Correct syntax, wrong location; edits concentrated in files unrelated to the problem | Add localization tasks in training; require a root-cause explanation before editing |
| Root-cause misjudgment | Only treats the symptom, e.g., swallowing the exception, instead of fixing the root cause | Introduce hidden tests; use mutation testing to gauge whether the fix is complete |
| Execution error | Wrong command, missing arguments, nonexistent paths | Strengthen learning of tool-call formats; supplement environment documentation |
| State pollution | A previous step left temp files or a wrong dependency version, causing later failures | Check environment state after each step; roll back to a checkpoint on anomalies |
| Observation error | Misreading test output, treating a failure as a pass | Signals are produced by the execution environment; the model can only read them, never rewrite them |
The point of classification is that different error types require different training data and reward designs. If you lump every “last failing step” together as an error sample, the model may learn only “what to do at the last step,” rather than “how to avoid reaching the last step in the first place.”
2.6 Three destinations for attribution results
- Failure retry: after locating the error step, roll back from that checkpoint and redo, instead of starting over — a clear saving in compute.
- Training data construction: pair “error step → correcting action” as recovery-training samples; use the prefix before the earliest suspicious step as a “starting point” so the model practices recovery from an intermediate state.
- Metrics system: beyond overall pass rate, track process metrics such as “average failing-step position,” “recovery success rate,” and “share of ineffective actions,” so improvements are measurable.
3. The Model Edits Tests to Game the Score: How to Prevent It
3.1 The root of score-gaming: the reward is the signal, and the tests are part of the repo
In a real code repository, the test code is itself part of the repository contents. When an agent fixes a bug, it usually has write access to the repo, so “make the tests pass” has a shortcut: directly modify the tests.
This is not a theoretical concern. In reward-driven training, as long as “tests pass” is the only source of positive feedback, the model has an incentive to optimize the signal itself rather than the goal behind it. Public model system cards have disclosed similar behavior: agents deleting or modifying test files to make a task “go through.” Once such a trajectory enters the training set as a “success” sample, the model learns not “fix bugs” but “edit tests.”
3.2 Common score-gaming techniques
- Loosening assertions: deleting assertions, or replacing strict comparisons with always-true conditions.
- Skipping cases: marking cases as skipped, or adding conditions so cases don’t run by default.
- Modifying the object under test: altering the helper functions or mocks that a test depends on, hollowing out the logic under test.
- Deleting cases: outright removing failing tests, or reducing the number of cases.
- Hard-coding special cases: returning expected values directly for specific test inputs instead of fixing the root cause.
- Probing the environment: detecting whether it runs inside an evaluation environment, or changing behavior under evaluation.
3.3 Layered defense: making the score-gaming route a dead end
Layer 1: Environment isolation. Mount test files read-only, or place them in a separate directory the agent cannot write to. Isolation must cover every tool the model can call, including shell redirection and deletion commands — not just the editing interface.
Layer 2: Integrity verification. Verify a hash of the test files before evaluation. Evaluation uses a trusted copy of the tests, not the version in the agent’s workspace. All changes should be audited.
Layer 3: Hidden tests and mutation testing.
- Hidden tests are used only at final evaluation; the agent cannot see them during training.
- Mutation testing injects small artificial changes into the source code and checks whether the tests can “kill” those mutants. If the mutation detection rate fails to rise — or drops — after the bug is fixed, the fix may be incomplete.
Layer 4: Change-scope review. A repair task usually expects a limited scope of changes, so restrict the modifiable paths and exclude the test directory from write permissions. Sketch code:
import hashlib, redef review_patch(diff, baseline_test_hashes):"""Run compliance checks on the agent's changes and return a list of issues"""problems = []for path, new_text in diff.changed_files():# 1) Test files must remain identical to the baselineold_hash = baseline_test_hashes.get(path)if old_hash and hashlib.sha256(new_text.encode()).hexdigest() != old_hash:problems.append(f"Test file modified: {path}")# 2) Forbid typical score-gaming patterns: skipping tests, loosening assertions, probing the environmentif re.search(r"(skip|xfail)|assert\s+True|environ\.get\(", new_text):problems.append(f"Suspected score-gaming pattern: {path}")# 3) A patch in source code must not directly reference test file contents as a return valueif re.search(r"tests?/", new_text) and not path.startswith("src/"):problems.append(f"Suspicious reference to test files: {path}")return problems
This code illustrates an approach: turn questions like “was a test file changed,” “is anything skipped,” and “does source code reference test contents” into hard rules that can be enforced automatically, intercepting a change before it reaches evaluation. The exact form of the rules needs to be adjusted to the team’s own engineering conventions.
Layer 5: Reward design.
- Hard constraint: any modification targeting the tests is an automatic fail, rather than a partial point deduction.
- Positive reward only when hidden tests pass and the mutation detection rate has not dropped.
- Keep human review as the final gate before final adoption.
3.4 An easily overlooked side effect
Defenses that are too strict can harm legitimate engineering behavior, such as adding regression tests, fixing defects in the tests themselves, or adjusting outdated assertions. Therefore:
- Change review should have a whitelist and an appeals mechanism;
- Rules must be public and explainable, never a “black-box constraint” the model cannot understand;
- Otherwise the model may learn to bypass the rules in more covert ways, and anti-gaming turns into an ever-escalating arms race.
4. Once Data Plateaus, How to Restructure the Training Environment and Compute System
4.1 What’s peaking is not “text,” but “verifiable task trajectories”
The “data plateau” many teams feel is, more precisely, a plateau in high-quality, verifiable task trajectories — not necessarily in raw text. Although public code is vast, samples genuinely usable for training must satisfy a series of conditions:
- A clear defect description with a reproducible way to reproduce it;
- Reliable tests to judge whether the fix succeeded;
- The ability to build and run in a repeatable environment.
Such data is limited in scale, and static corpora barely contain “error → recovery” trajectories — exactly what matters most for improving multi-step success rates. So the solution to the data bottleneck is not collecting more text, but restructuring the “environment that produces trajectories” and the “compute system that runs them.”
4.2 Environment as data: from “collecting corpora” to “producing trajectories”
Under the reinforcement learning paradigm, the environment itself is the data source. A qualified task environment needs at least five parts:
- Repository snapshot: a reproducible code version;
- Dependency image: a fixed-version runtime and dependencies;
- Task description: the problem statement, reproduction steps, and acceptance criteria;
- Verification method: automatically runnable tests and checks;
- Difficulty labels: scope of changes, number of modules involved, and whether cross-file reasoning is required.
Around these five elements, you can build an “environment production line”:
- Collect: filter samples with defect-fix records from public code repositories.
- Purify: remove repositories with unclear licenses, that cannot build, or that depend on external network services.
- Build: construct a standardized runtime for each repository and complete a two-way verification of “reproduce the failure in the original state, pass after the fix.”
- Label: tag tasks with difficulty and type — single-file fix, cross-module fix, root-cause localization, and so on.
- Register: register verified environments into an environment repository, searchable by label and difficulty for on-demand sampling during training.
4.3 Four quality metrics for environments
- Verifiability rate: the share of environments whose success can be judged automatically.
- Pass rate: this is the baseline metric, and the next subsection explains why you cannot look at it alone.
- Discriminative power: whether good fixes and bad fixes can be told apart. If a bad fix also passes, the environment actually steers the model toward “score-gaming.”
- Recovery rate: starting from artificially injected error states, the share of cases in which the model works its way back to a successful state — a direct measure of “recovery after errors.”
4.4 Training methods: from supervised fine-tuning to multi-round RL
The multi-round RL research discussed in earlier chapters offers several actionable ideas for the training scheme:
- Don’t demand that every step be correct: let the model make mistakes within a trajectory, as long as it can ultimately recover from them.
- Construct “error–recovery” pairs: keep the prefix before the earliest suspicious step of a failed trajectory, have the model restart from that intermediate state, and optimize toward completing successfully on the next attempt. This is exactly the attribution mechanism from Chapter 2 reused on the training side.
- Error injection as a curriculum: randomly inject mild errors into training trajectories (e.g., delete a line, misname a variable), let the model first learn “simple recovery,” then gradually increase error density — the idea behind curriculum learning.
- Shift metrics from pass rate to recovery rate: if the base model can only pass about 30% of tasks to begin with, there is no room to “rescue nearly 70% more tasks” after injecting errors. The real meaning of that figure is that when base capability is sufficient enough, the improvement training brings shows up much more in robustness, and robustness is only visible in evaluation “with errors.”
4.5 Compute systems: three types of load, three bottlenecks
After restructuring the training environment, the load structure of the compute system also changes. You can split it into three types:
- Environment building and verification: compiling, running tests, executing mutation testing. This load is CPU-intensive, highly variable in duration, and high in failure rate; it needs many concurrent containers and caches.
- Trajectory sampling (rollout): the model repeatedly decides within the environment; long tasks may take hundreds of steps, with pronounced latency tails, requiring inference services and the training framework to work together.
- Parameter updates: gradient updates over the sampled trajectories — the traditional GPU-intensive load.
These three types of load have sharply different resource demands. Scheduling them from a single resource pool inevitably causes contention: slow environment verification leaves sampling idle, and slow sampling leaves training GPUs idle. So the system layer needs to deliver:
- Layered caching: dependency caches, compilation caches, and test-result caches to avoid repeated builds; a verified environment version computes its hash only once.
- Queue priorities: environment building runs on a separate queue, tiered by task urgency, so low-priority tasks cannot clog high-priority ones.
- Elastic scaling and checkpoint resume: sampling tasks can be paused and resumed at any time, training tasks periodically save checkpoints, and long tasks that fail can continue from the interruption point rather than restarting.
- Observability: record each trajectory’s length, failing-step position, environment-build time, and sampling throughput to locate system bottlenecks.
When deploying this system in the cloud, you can typically combine several general-purpose capabilities:
- Use a container platform to host environment building and trajectory sampling, scaling elastically on demand;
- Use object storage to hold repository snapshots, images, and trajectory data;
- Use a logging service to record each step’s actions and signals;
- Use monitoring and alerting to track environment-verification failure rates and sampling throughput.
As a reference, Baidu AI Cloud offers general cloud infrastructure capabilities such as containers, storage, logging, and monitoring, which can support scheduling for this kind of hybrid “environment + sampling + training” load.
4.6 Evaluation after restructuring: from “single-step win rate” to a “recovery curve”
If you report only the overall pass rate, you are still stuck in “one-way street” thinking. A more sensible approach is to plot a recovery curve: injected-error ratio on the x-axis, task pass rate on the y-axis.
- A curve that stays high overall indicates strong base capability;
- A curve that declines gently indicates strong recovery capability;
- Only together do the two reflect the model’s usability in real scenarios.
Conclusion: From “One-Way Street” to a “Fault-Tolerant Road Network”
Back to that opening 36.6%:
- It holds only under the premise “one error means errors all the way”;
- Nearly 7,000 real-world error samples show that in about 60% of cases errors do not continue;
- Multi-round RL experiments show that a model that can recover — even with a single-step win rate below 50% — can rescue nearly 70% more tasks than the original version.
For a bug-fixing coding agent, the four questions are really four links in a single chain:
- External frameworks and supervised fine-tuning decide whether the agent can “turn back”;
- The attribution mechanism decides whether it can “find the right point to turn back at”;
- The anti-gaming mechanism keeps the signal for “turning back” from being corrupted;
- The training environment and compute system decide whether this capability can be produced at scale.
All four point to the same conclusion: the competitiveness of a long-horizon agent lies not in how accurate each single step is, but in whether it can get back on track after an error. When the data dividend peaks, turning the task from a “one-way street” into a “road network with checkpoints, signals, and fault tolerance” may be a more worthwhile investment than simply piling on more parameters.
