Long-Horizon Agent Reinforcement Learning: An Open Recipe for Performance and Cost Optimization
A practical framework for training long-horizon terminal and coding agents with full-parameter reinforcement learning. This article explains environment design, verifiable rewards, training stability, parallel sampling, end-to-end cost accounting, evaluation methodology, and the engineering practices needed to move the performance–cost Pareto frontier. It also summarizes a public result in which a 300B-class model reached a 46.2% single-run pass rate on a terminal benchmark using 64 high-memory accelerators in less than five days, while reducing per-task cost by more than 30 times compared with a high-reasoning baseline.
1. Why Long-Horizon Agents Are Difficult
Terminal operation and code repair are representative long-horizon agent tasks. An agent must read the environment state, execute commands, inspect feedback, modify files, run validation, and continue adjusting its actions based on failure information over multiple interaction rounds.
Unlike single-turn question answering, success depends not only on model knowledge, but also on the stability of the policy across multiple decisions, its ability to recover from errors, and how efficiently it uses tool feedback.
Long-horizon agent training therefore involves three major challenges:
- Long trajectories and sparse rewards: A task may require dozens of tool calls but provide only a final success or failure signal. Intermediate actions are difficult to supervise directly.
- High evaluation cost: Each trial may require starting an isolated environment, executing commands, collecting logs, and validating the result. End-to-end evaluation is substantially more expensive than text-only reasoning.
- A strong coupling between performance and cost: Increasing reasoning effort or adding retries can improve the pass rate, but it also raises the cost of each task and makes the overall system difficult to sustain.
A recently published practice offers a notable answer. Starting from a 300B-class general-purpose foundation model, the team used full-parameter reinforcement learning to train terminal and coding agents. With 64 high-memory accelerators and a training budget of less than five days, the agent achieved a 46.2% single-run pass rate on a terminal-task benchmark, reaching a level comparable to a strong high-reasoning baseline. At the same time, its per-task cost was reduced by more than 30 times.
The result also transferred to two software engineering benchmarks, where performance was evaluated as the average across five trials. More importantly, the team released a complete training recipe, turning the performance–cost Pareto frontier from an isolated result into an engineering approach that can be discussed and reproduced.
2. Interpreting the Result: Moving the Pareto Frontier
These results should not be judged by pass rate alone. Two dimensions must be considered together:
- Task success rate
- Average cost per task
The cost should include model inference, tool calls, sandbox execution, log storage, retries, and the long tail of failed runs. Comparing only model pricing can hide the amplification caused by retry counts and interaction length in long-horizon tasks.
Based on the public information, the value of the approach can be summarized in three points:
- Comparable performance: The single-run pass rate reached 46.2% on the terminal-task benchmark, at a level comparable to a strong high-reasoning baseline.
- Substantially lower cost: Per-task cost fell by more than 30 times, suggesting that the learned policy relied less on extremely long reasoning traces, repeated trial and error, and high-intensity sampling.
- Effective transfer: Consistent gains were observed on software engineering repair benchmarks, indicating that the learned capability was not limited to fitting a single environment and had some cross-task generalization.
Together, these results represent a movement of the performance–cost Pareto frontier. The goal is not simply to obtain a higher score by spending more, but to achieve comparable performance with substantially lower end-to-end cost.
3. Method Breakdown: From a Foundation Model to a Long-Horizon Policy
3.1 Task Environments and Trajectory Construction
The first step in long-horizon agent training is not hyperparameter tuning. It is building a task environment that supports repeatable execution, reliable verification, and parallel sampling.
Terminal and programming tasks generally require:
- Isolated file systems
- Controlled command execution
- Timeout handling
- Resource limits
- Complete log replay
Training data should not consist only of final answers. It should include states, actions, observations, validation results, and failure causes.
A reproducible trajectory typically contains the following fields:
Trajectory = {task description,initial environment snapshot,multi-turn action and observation sequence,intermediate validation signals,final validation result,resource usage and timeout information}
Trajectory quality determines the upper bound of reinforcement learning performance. If the environment is unstable, repeated executions of the same policy may produce random failures, contaminating the reward signal with noise. If logs are incomplete, failure cases cannot be used to diagnose policy problems.
Environment standardization and trajectory replay are therefore foundational infrastructure for the training recipe.
3.2 Verifiable Rewards and Process Constraints
Long-horizon tasks are well suited to a reward design that combines verifiable outcomes with process constraints.
Outcome rewards can come from tests, compilation, static analysis, or task acceptance checks. Process constraints can be used to discourage dangerous operations, ineffective loops, and resource abuse.
A general pseudocode example is:
def compute_reward(trajectory, verifier):final_ok = verifier.check_final_state(trajectory)if not final_ok:return -1.0step_penalty = 0.01 * trajectory.num_stepsretry_penalty = 0.02 * trajectory.retry_countsafety_penalty = 1.0 if trajectory.has_unsafe_action else 0.0efficiency_bonus = 0.05 if trajectory.num_steps < trajectory.budget else 0.0return 1.0 + efficiency_bonus - step_penalty - retry_penalty - safety_penalty
The key is not reward complexity, but whether the reward is verifiable, reproducible, and difficult to exploit.
For programming tasks, passing the final test suite is the strongest signal. For terminal tasks, the final environment state and task acceptance script are the primary sources of truth. Process penalties should be applied carefully. If they are too strong, the policy may become overly conservative and avoid taking necessary actions.
3.3 Stabilizing Full-Parameter Reinforcement Learning
Full-parameter reinforcement learning can produce more consistent policies than training only a small adaptation layer, but it also places greater demands on training stability.
Common engineering practices include:
- Reference-policy constraints: KL-style constraints limit policy drift and reduce the risk of output-distribution collapse in later training stages.
- Advantage estimation and normalization: Normalizing rewards across long trajectories reduces variance caused by differences in task difficulty.
- Curriculum learning: Training progresses from short, high-success-rate tasks to longer and more failure-prone tasks.
- Failure-sample replay: Typical trajectories involving timeouts, loops, and failed error recovery receive focused analysis.
- Staged evaluation: The policy is periodically evaluated on a fixed, small evaluation set using single-run pass rate and average step count, rather than relying only on training rewards.
A simplified training loop is shown below:
for step in range(train_steps):tasks = sampler.next_batch()trajectories = policy.rollout(tasks, env_pool)rewards = [compute_reward(t, verifier) for t in trajectories]advantages = normalize(rewards, group_by=task_family)loss = policy_loss(policy, trajectories, advantages)loss += kl_coef * kl_divergence(policy, reference_policy)loss.backward()optimizer.step()if step % eval_interval == 0:report = evaluate(policy, fixed_eval_set, trials=5)log(report.pass_at_1, report.avg_cost_per_task)
3.4 Parallel Training and Resource Efficiency
A budget of 64 high-memory accelerators and less than five days requires the training system to address both throughput and fault tolerance.
Long-horizon sampling is expensive. Environment startup, command execution, and validation can all become bottlenecks. Typical engineering measures include:
- Decoupling training from sampling and buffering trajectories through asynchronous queues
- Pooling environments and restoring snapshots to reduce repeated initialization
- Dynamically allocating sampling budgets according to task difficulty
- Quickly stopping timed-out or abnormal trajectories
- Aggregating logs, metrics, and costs by task to identify long-tail behavior
In cloud deployments, isolated environments can be hosted on a general-purpose container platform, while object storage preserves trajectories and logs. Logging and monitoring services provide observability for training.
The specific products are less important than turning the environment lifecycle and trajectory data flow into infrastructure that is orchestrated, auditable, and repeatable.
4. Why the Cost Can Fall by an Order of Magnitude
Per-task cost is not simply the price of model inference. It is an end-to-end accounting model:
Per-task cost =model input and output cost+ tool invocation and environment execution cost+ retry and long-tail failure cost+ sandbox resource and log storage cost
The cost reduction enabled by reinforcement learning primarily comes from improvements in policy quality:
- A higher first-attempt success rate reduces unnecessary retries.
- More reliable error recovery reduces excessively long trajectories.
- Better stopping behavior reduces overthinking and ineffective tool calls.
For long-horizon tasks, long-tail failures often account for a large share of total cost. As a result, even a modest increase in single-run success rate can significantly reduce average cost.
The use of a self-deployed 300B-class model and a comparable public billing basis also makes the optimization space more controllable. However, organizations reproducing the result still need a consistent cost methodology. They should record input and output tokens, tool calls, environment runtime, and failed retries under a fixed number of trials. Without a unified accounting standard, a claim such as “30 times lower cost” cannot be compared reliably across systems.
5. Evaluation and Transfer: Three Levels to Measure
Evaluation should cover at least three levels:
- Single-run pass rate on terminal tasks: Measure first-attempt success and avoid hiding policy weaknesses behind repeated retries.
- Transfer to software engineering repair: Evaluate performance on two software engineering benchmarks using the average across five trials to test cross-task generalization.
- Cost–performance curves: Report pass rate, average step count, average retry count, and per-task cost together to create a Pareto comparison.
Fair evaluation also requires fixed environment versions, validation scripts, timeout budgets, and trial counts. Otherwise, environment drift or changes in validation criteria can introduce significant bias.
6. Reproduction Checklist for an Open Training Recipe
Teams seeking to reproduce a similar long-horizon agent reinforcement learning system can proceed with the following checklist:
- Define task boundaries: Specify separate success criteria for terminal operation, code modification, and test repair.
- Build verifiable environments: Isolation, snapshots, timeouts, resource limits, and log replay are all essential.
- Design the reward function: Make outcome rewards primary and process penalties secondary, while ensuring that the reward can be recomputed.
- Establish training stability mechanisms: Use reference-policy constraints, advantage normalization, curriculum learning, and failure replay.
- Account for end-to-end cost: Include retries, sandbox execution, and storage in the per-task cost.
- Fix the evaluation protocol: Use five-trial averages, fixed environments, and fixed validation scripts.
- Apply safety and permission controls: Restrict dangerous commands, sensitive paths, and high-resource operations.
- Publish reproducible configurations: Make data versions, reward rules, evaluation scripts, and resource budgets auditable.
7. Conclusion
This public practice with a 300B-class terminal and coding agent shows that long-horizon agent competitiveness should not be measured only by the highest possible pass rate. The more important question is whether the system can achieve the target performance consistently at an acceptable cost.
Through full-parameter reinforcement learning, verifiable rewards, stable training, and end-to-end cost accounting, teams can improve both performance and cost within a limited accelerator budget and move toward a better performance–cost Pareto frontier.
For enterprises, the most reusable contribution is the engineering methodology: environment, trajectory, reward, evaluation, and cost accounting must form a closed loop. Only then can agents move from demonstrations to sustainable production systems.
