Long-Horizon Agent Reinforcement Learning: Counterfactual Branch Valuation in Practice
As LLMs move from single-turn responses to end-to-end task completion, long-horizon coding agents require better evaluation, verifiable environments, and more precise credit assignment. This article examines agent capability benchmarks, the emergence of security skills, the challenge of sparse rewards, and a search-driven Self-Adaptive Optimization approach that uses counterfactual branch rollouts to estimate the value of critical decisions. It also covers implementation details, engineering constraints, and practical recommendations for building scalable training environments.
Long-Horizon Agent Reinforcement Learning: Counterfactual Branch Valuation in Practice
The competition among large language models is shifting from the quality of single-turn answers to the ability to complete complex tasks end to end. In the open-source model ecosystem, newer releases increasingly position long-horizon coding agents as a core capability. Some comparisons suggest that their performance in software development and automated vulnerability remediation is approaching that of models several times larger.
At the same time, model developers commonly claim that their capabilities in network security and offensive security are beginning to emerge. An easily overlooked fact lies behind these results: the decisive factor may not be further scaling parameter counts, but how post-training generates useful learning signals for trajectories spanning dozens of interaction turns.
This article discusses three questions:
- How should long-horizon agent capability be evaluated?
- How should rewards be assigned?
- How should the training environment be built?
1. What Should Capability Evaluation Measure?
The basic form of a long-horizon coding task is as follows: the model receives a bug report or feature request, then repeatedly performs a cycle of reading code, searching for relevant locations, modifying files, running tests, analyzing errors, and making further corrections inside an isolated sandbox until the task passes acceptance checks.
A moderately complex repair task may require 30 to 80 interaction turns. Evaluation therefore needs to move beyond code generation and measure agent capabilities such as terminal operation, multi-file coordination, and error recovery.
Current evaluation methods can generally be grouped into three categories.
- Benchmark tasks: A problem statement is provided in a containerized, real-world code repository, and hidden tests determine whether the patch is correct. Examples include code repair, defect detection, and vulnerability patch generation benchmarks.
- Terminal and tool-chain operation: These tests measure whether the model can install dependencies, run scripts, inspect logs, and use debugging tools to locate problems in a command-line environment.
- Internal experience-based evaluation: Public benchmarks can saturate and may be affected by training-data contamination. Model developers therefore build private test sets and supplement them with human comparative evaluations to identify changes in real-world usability.
Parameter count and performance are not strictly proportional. In one public three-way comparison, a 744B-total-parameter model using a heavily modified version of an earlier sparse architecture was compared with a 1.6T-total-parameter model using a newer-generation architecture.
| Task | 744B Model | 1.6T Model |
|---|---|---|
| Long-horizon code repair | 62.7 | 66.9 |
| Automated vulnerability remediation | 83.3 | 84.5 |
| Long-horizon composite tasks | 25.7 | 28.5 |
The first model used less than half as many parameters, yet remained competitive across all three tasks, with relatively limited gaps. Notably, the larger model led by approximately four points on code repair, while its advantage narrowed to roughly three points on the longer and more complex composite tasks.
This suggests two things:
- Architecture determines single-step capability, but as task chains become longer, the impact of post-training quality on final performance becomes increasingly significant.
- The substantial improvement achieved without replacing the base architecture depends primarily on exceptionally strong post-training—in other words, pushing reinforcement learning and data engineering far beyond conventional levels.
This does not mean that Scaling Laws no longer apply. A more reasonable interpretation is that the focus of scaling is shifting from parameter count toward environment scale and post-training investment. The composite tasks in the table are precisely where these methods can have the greatest impact.
2. Why Do Security Capabilities “Emerge”?
Within the same agent capability framework, models can also demonstrate stronger vulnerability comprehension, exploit-strategy construction, and automated penetration-testing abilities. This emergence is not accidental.
- Shared underlying capabilities: Code understanding, debugging, dependency analysis, and script writing are fundamental both to fixing defects and to analyzing attacks.
- Sandbox amplification: A large number of parallel sandbox environments allows models to repeatedly try different repair or exploitation strategies and learn from real execution feedback.
Security capabilities are inherently dual-use. Governance should therefore follow the principle of making capabilities usable while keeping the process controllable.
- Sandbox isolation: Code execution and network access should be strictly limited to containers. External network access should be disabled by default, and download sources and dependency repositories should be managed through allowlists.
- Data boundaries: Vulnerability verification on systems without authorization may be illegal. Training and evaluation should use self-hosted test ranges, open vulnerability datasets, or environments for which explicit authorization has been obtained.
- Least privilege: Keys, tokens, and cloud-resource permissions used by an agent should be narrowly scoped and revoked after use.
- End-to-end auditing: Model decisions and command execution should be recorded in full to support investigation and post-incident review.
For cloud platforms that provide both model-training infrastructure and agent-development capabilities, these practices can be implemented using general-purpose services such as object storage, elastic containers, logging, and monitoring and alerting. For example, these categories of services can be used to store trajectory data, host sandbox clusters, aggregate audit logs, and observe the resource usage and cost of training jobs. Specific choices should be evaluated against the project’s scale and compliance requirements.
3. Credit Assignment Under Sparse Rewards
The central training difficulty in long-horizon tasks is extremely sparse reward feedback. Consider a code-repair task:
- At step 78, the model accurately identifies the root cause through code search and architectural analysis. This is a highly valuable decision.
- At step 79, it introduces a variable-name error while modifying the code.
- At step 80, the unit tests fail and the environment returns
Reward = 0.
Traditional policy-gradient methods, such as PPO and GRPO, see only that the final trajectory received a score of zero. They may therefore apply negative gradients uniformly across all tokens from steps 1 through 80.
This is similar to judging a Go game by saying, “The player lost, so every move was bad.” The earlier brilliant move receives no recognition simply because the final position was unsuccessful.
This result-only approach causes three problems:
- Useful signals are diluted: Valuable decisions made during hundreds of exploratory steps are buried among many irrelevant tokens.
- Exploration is penalized: Trajectories that try new approaches are more likely to fail, causing the model to prefer conservative behavior. It may repeatedly inspect files, seek confirmation, and delay taking action.
- Costs increase: Token consumption remains high while task completion becomes more difficult, creating a negative cycle in which more training produces more conservative behavior, and more conservative behavior leads to more failures.
Why not assign a score to every step directly?
First, humans have difficulty judging the quality of a long-horizon coding trajectory. Evaluating software-engineering decisions across files and modules requires substantial engineering knowledge and is extremely expensive.
Second, using another model as a process reward model creates a circular dependency. Accurate evaluation requires a sufficiently capable evaluator, while a sufficiently capable evaluator itself depends on accurate evaluation signals. In the early stages, this is a major bottleneck for long-horizon agent training.
4. Self-Adaptive Optimization: Using Search to Generate Labels
In code-execution and command-execution environments with verifiable outcomes, there is a natural way to break this deadlock: the environment itself can perform simulations.
A technique used by newer models, referred to here as Self-Adaptive Optimization (SAO), is essentially a search-driven form of process supervision. Its underlying idea is closely related to self-play in reinforcement-learning systems for Go.
The key technique can be called counterfactual local branch valuation. The system keeps the preceding context unchanged, starts from a critical decision point, and runs multiple possible futures: “What would have happened if a different action had been selected?” The value of the decision can then be estimated by comparing the resulting outcomes.
Consider the Go analogy. If a game is lost, how can we determine whether move 78 was actually brilliant? Keep the board state after move 77 unchanged, clone the position at the move-78 decision point, and have players of comparable strength continue from multiple candidate moves. If the continuation win rate from the move selected at move 78 is significantly higher than the rates from alternatives, the move should be recognized and reinforced.
The same principle applies to coding agents:
- Before step
k, save the context, including the conversation history and tool-call records, together with the environment state, including the file system, processes, and network state. - Starting from that state, let the same policy continue from the actual action and from several alternative actions until task termination.
- Compare their success rates or progress scores to estimate the local advantage of the selected action.
- Assign positive credit to tokens associated with positive-advantage decisions and moderately suppress decisions with negative advantage.
Implementation Considerations
- Branch granularity: It is not necessary to branch at every step. Start with critical decisions, such as choosing a repair strategy, selecting a search path, or deciding which command to run. Policy entropy and changes in action type can help identify such points automatically.
- Alternative actions: In addition to the action actually selected by the model, sample several candidate actions. Another option is to let the model continue directly from the branch state and use whether the continuation succeeds as a proxy signal.
- Valuation scale: The number of branches and simulation depth directly determine compute cost, so they should vary throughout training. Early training can use more branches and deeper simulations. Once the policy becomes stable, the system can reduce branching and retain only critical decision points.
- Prioritize reliable signals: Begin with verifiable outcomes such as passing tests, successful compilation, and reproduced defects. Static checks, failure counts, and other proxy signals can be introduced gradually.
- Avoid excessive credit assignment: Branch valuation is only a local approximation. It should not assign the entire reward to one step. Instead, it can be combined with the final reward as a weighted advantage correction.
Example Implementation
from dataclasses import dataclass, replace@dataclassclass Step:text: str # Model outputtokens: list # Corresponding token sequence@dataclassclass Rollout:prefix: list # Historical steps up to the branch pointbranch_step: Step # Actual step at the branch pointchosen: float # Success rate of the actual-action branchbaseline: float # Average success rate of alternative branchesdef evaluate_local_advantage(prefix, step, policy, env, n=8, max_turns=40):"""Counterfactual local branch valuation:compare the actual action with alternative actions."""base_state = env.snapshot(prefix) # Save context and environment statechosen_success = 0for _ in range(n):# Align other sources of randomness: same model and temperature,# with only the action changedfinal = policy.rollout(base_state, first_step=step, temperature=0.7, max_turns=max_turns)chosen_success += float(final.success)chosen = chosen_success / nbaseline_success = 0for i in range(n):# Alternative action: sample a candidate or let the policy# continue directly from the branch statealt_step = policy.sample_alternative(prefix, seed=i) if i % 2 == 0 \else policy.continue_from(base_state, temperature=0.7)final = policy.rollout(base_state, first_step=alt_step, max_turns=max_turns)baseline_success += float(final.success)baseline = baseline_success / nreturn Rollout(prefix, step, chosen, baseline)def advantage(rollout):# Local advantage = success rate of actual action# minus average success rate of alternativesreturn rollout.chosen - rollout.baselinedef apply_credit(training_batch, alpha=1.0, beta=0.2, eps=1e-6):for r in training_batch:a_local = advantage(r)# alpha * local credit + beta * original reward;# tune hyperparameters for the taska = alpha * a_local + beta * r.rewardfor tok in r.branch_step.tokens:# Update only tokens in the branch step.# Length normalization helps avoid over-weighting long steps.tok.grad_weight = a / (len(r.branch_step.tokens) + eps)return training_batch
Key Engineering Challenges
- Context isolation: Every branch must have an independent file system and process space. Changes in one branch must not contaminate another.
- State snapshots: Container snapshots, OverlayFS copy-on-write, and Git branches can all serve as lightweight approaches. The preferred solution should start quickly and minimize cost.
- Test nondeterminism: Network fluctuations, port conflicts, and test-order dependencies can cause identical branches to produce different results. The environment should be reproducible through locked dependency versions and fixed random seeds. Repeated sampling can then be used to reduce noise.
- Cost and efficiency: If every branch reruns the full task from the beginning across dozens of turns, the cost increases dramatically. Possible optimizations include sharing common prefixes across grouped samples, executing branches in parallel, using shallow continuation as a rough filter, running full simulations only for high-potential branches, and reusing warmed environments in a sandbox pool.
- Reward shaping: To prevent low-quality trajectories from gaming the reward, length penalties or token-efficiency terms can be added. This helps prevent the model from improving its success rate through meaningless verbosity.
5. From Outcome Rewards to Search-Driven Process Supervision
| Dimension | Outcome Rewards (PPO/GRPO) | Process Reward Model | Search-Driven Counterfactual Valuation (SAO) |
|---|---|---|---|
| Signal source | Final task success or failure | Model- or human-assigned step scores | Environment simulation and branch comparison |
| Credit granularity | Entire trajectory | Individual steps or segments | Critical decision steps |
| Main bottleneck | Sparse and high-bias signals | Evaluator cold start and labeling cost | Sandbox and sampling overhead |
| Required conditions | Verifiable task rewards | A sufficiently accurate evaluator | Cloneable, parallel, reproducible environment |
Traditional methods treat reinforcement learning as optimization over outcomes. SAO instead treats search as a source of supervision.
Search generates signals that improve the policy, while a stronger policy improves the quality of future search. This creates a positive feedback loop. The approach closely resembles the “search guides learning, and learning improves search” paradigm used in Go systems. It can also be viewed as an engineering adaptation of MCTS-style value estimation for long-horizon agent tasks.
6. Recommendations for Deployment
Teams building long-horizon coding agents should consider the following sequence.
- Build a verifiable environment first: Develop a sandbox cluster that can create isolated environments in seconds and run them at scale. Each task should have a stable acceptance script. This is the foundation of post-training.
- Optimize inference costs: Use pretraining compute as the foundation and focus post-training compute on data and training methodology. Whenever possible, reuse or adapt the existing architecture rather than concentrating investment only on architectural changes.
- Design multidimensional rewards: Use task success as the primary metric, supplemented by test pass rates, defect reproduction, and token efficiency. This reduces the risk of any single metric being exploited.
- Deploy security capabilities cautiously: For public-facing systems, establish refusal policies for requests with clear malicious intent. Audit logging, monitoring, and alerting should be treated as standard operational mechanisms.
Conclusion
The rapid progress of open-source models on coding and security tasks shows that substantial gains can be achieved even without a generational architecture change, provided that post-training and verifiable-environment engineering are executed exceptionally well.
The real bottleneck in long-horizon agent training is not whether a model can write code. It is whether the system can accurately determine which step was good.
Counterfactual branch valuation uses sandbox clusters and parallel sampling to transform sparse terminal feedback into local, comparable value signals. This provides a practical path toward credit assignment under sparse rewards.
In verifiable environments, reinforcement learning may be approaching a moment similar to search-driven self-play: search improves the policy, and the improved policy makes search more effective. How far this pattern can be taken in long-horizon agent training remains an important question.
