Long-Horizon Agentic Reinforcement Learning in Practice: A Lower-Cost Performance-Cost Frontier

    This article distills a publicly released recipe for long-horizon agentic reinforcement learning into a reusable method. Starting from a medium-sized base model with lightweight reasoning, a research team trained 300B-class terminal and coding agents on a cluster of general-purpose accelerators using only dozens of accelerators in under five days. The resulting agent reached a 46.2% pass@1 rate on mainstream terminal agent benchmarks, in the same range as the strongest frontier models at the time, while pushing per-task inference cost down to roughly 1/30 of a max-reasoning-intensity frontier model. Rather than restating scores, the article explains where long-horizon agentic capability comes from, why reinforcement learning can improve both quality and cost at once, the six building blocks of the recipe (environment, reward, sampling, optimization, systems, evaluation), and how teams can reproduce it to build their own performance-cost frontier.

    Tasks in terminal command lines and real code repositories are among the most demanding scenarios for measuring agent capability today. The terminal has no natural-language interface: a model can only probe, modify, and verify with a limited set of commands. Code-repository tasks, meanwhile, often span multiple files and test reasoning and localization over long contexts. The two share one trait: a single task is rarely solved in one shot. It typically takes dozens of tool calls, sometimes hundreds of steps, with trial, error, and rollback along the way. The model is not just “answering one question correctly” but “finishing an engineering problem for real.”

    A research team recently released a complete recipe for long-horizon agentic reinforcement learning. Starting from a medium-sized base model with lightweight reasoning, they trained 300B-class terminal and coding agents on a cluster of general-purpose accelerators. Training used only dozens of accelerators and took less than five days. The resulting agent reached a 46.2% pass@1 (first-attempt pass rate) on mainstream terminal agent benchmarks, in the same range as the strongest frontier models at the time, and improved on software-engineering benchmarks in parallel. The more important part is cost: it pushed per-task inference cost down to roughly 1/30 of a max-reasoning-intensity frontier model, more than a 30x reduction per task. The team published the full recipe, spanning data, rewards, algorithms, and systems.

    The most valuable lesson here is not how high a score it reached, but that it answers a question every enterprise faces when adopting agents: once the capability gap narrows, cost becomes the decisive variable. Rather than restating “how a given model scored,” this article abstracts the recipe into a reusable method:

    • Where long-horizon agentic capability comes from;
    • Why reinforcement learning can improve both quality and cost at the same time;
    • Which modules make up the recipe;
    • How enterprises can reproduce it and build their own performance-cost frontier.

    1. The Overlooked Dimension: An Agent’s “Unit Cost”

    1.1 Flagship Models Push Performance Higher and Cost Deeper

    Mainstream terminal and coding benchmarks generally require an agent to complete a full workflow in an isolated environment, from diagnosis through modification to verification, with a hidden automated test suite as the sole arbiter. Because the outcome can be judged objectively, these tasks are naturally verifiable and make a good yardstick for “whether a model can actually get work done.”

    Over the past year or so, pass rates on these benchmarks have kept climbing. Leading models generally offer adjustable “reasoning intensity” tiers: the higher the tier, the longer a model’s internal thinking before it answers, and usually the higher the pass rate on hard problems. The catch is that cost does not grow linearly. When reasoning intensity is raised, input and output token consumption often multiplies, and failed tasks get billed repeatedly across multiple retries.

    This produces a structural pattern on the “pass rate × cost” plane: the right end of the curve (strongest model plus highest reasoning intensity) leads on pass rate, but the cost per task can be several orders of magnitude higher. For most enterprise scenarios, this range is hard to sustain. The reason is practical: a task often needs several retries before it succeeds, and every failure is billed in full.

    1.2 Two Improvement Routes: Compress “Cost per Inference” and “Inference Volume per Task”

    Moving this curve comes down to two routes:

    • Route A, lower the per-token price: use smaller models, quantization, speculative decoding, or mixture-of-experts sparse activation to make each inference cheaper;
    • Route B, reduce the inference volume needed to finish a task: let the agent complete work with fewer turns, fewer tokens, and a higher first-attempt success rate, making each task cheaper.

    Route A is bounded by the physics of chips and algorithms, with diminishing marginal returns, and small models hit a clear capability ceiling on long-horizon tasks. Route B is different: it acts directly on the agent’s “decision efficiency.” Task cost can be written as:

    1. task cost ≈ Σ(task type × task share × [retries × tokens per turn × turns × price per token])

    Here, retries are determined by pass rate, tokens per turn by reasoning intensity and context length, and turns by the quality of the agent’s policy. When the base model is fixed and the per-token price is fixed, the only variables left to improve substantially are retries, turns, and token utilization. And that is exactly what long-horizon reinforcement learning can optimize directly.

    2. Why Full-Parameter Reinforcement Learning Improves Both Capability and Cost

    2.1 From “Imitating the Right Answer” to “Optimizing the Final Outcome”

    Conventional supervised fine-tuning is imitation learning on human-written trajectories or trajectories distilled from a strong model. At its core, it maximizes the probability that “the next action matches the reference action.” It is valuable for cold starts but has two ceilings:

    • Reference trajectories are not necessarily optimal, so the model’s upper bound is capped by the demonstration data;
    • The model mostly “repeats what it has seen” and tends to drift off track on unseen combinations of long-horizon states.

    Reinforcement learning changes the objective function: instead of constraining “what to do at each step,” it maximizes “whether the task ultimately succeeds.” Driven by verifiable rewards, the model discovers shorter, more stable paths through extensive trial and error. What it learns is not just knowledge but “decision habits”: when to read a file, when to run tests, when to abandon a path that will not work. This is precisely where “fewer turns and lower token consumption” come from.

    2.2 Why “Full-Parameter” Matters

    In practice, “parameter-efficient fine-tuning such as LoRA plus reinforcement learning” is a common combination. Parameter-efficient fine-tuning has a small memory footprint and is simple to implement, and it works well on single-turn or short-horizon tasks. But adjusting a long-horizon agent policy involves many implicit “work habits” spread across different layers, such as the timing of tool calls, context-management strategy, and how to recover after a failure. These behaviors often require coordinated updates across the whole network’s weights to change stably.

    The public recipe emphasizes full-parameter updates for this reason: in the later stage of reinforcement learning, gains come mainly from a systematic restructuring of policy behavior rather than local fitting. At the same time, full-parameter updates demand stricter stability control, which the recipe handles with measures such as reference-model constraints and advantage normalization (see Chapter 3).

    2.3 The Core Difficulty of Long-Horizon Tasks: Rewards Are Too Late and Too Sparse

    A typical terminal task may involve dozens of interactions, but the reward arrives only at the very end: 1 if all tests pass, otherwise 0. This creates three concrete problems:

    1. Credit assignment is hard. Neither the step that led to success nor the step that caused failure can be attributed.
    2. Gradient noise is high. The entire trajectory shares a single scalar reward, so the model struggles to tell which actions were actually useful.
    3. Efficiency is low. Many trajectories fail completely due to randomness and provide very little learning signal.

    The recipe mitigates this with three techniques: curriculum sampling (concentrating training on tasks with moderate pass rates), in-group relative advantage (comparing within the same sampling group, which naturally removes bias from task difficulty), and partial-trajectory continuation (carrying unfinished samples into the next round so long trajectories are not truncated and wasted).

    3. A Reusable Training Recipe: Six Modules

    Setting aside specific numbers, this recipe breaks into six relatively independent modules. Their coupling works like this: environment and tasks decide “what can be learned,” reward decides “in which direction to learn,” sampling strategy decides “how fast to learn,” the optimization algorithm decides “how stable the learning is,” system implementation decides “how soon it finishes,” and the evaluation system decides “whether what was learned is trustworthy.”

    3.1 Environment and Tasks: First a Verifiable Ground Truth, Then the Agent

    This is the most underestimated link, yet the one that most determines the ceiling.

    • Environment isolation and reproducibility: each task runs in a one-time container with dependencies and toolchains preinstalled, the file system snapshotted, and network access restricted on demand. This ensures both fair scoring and repeatable failures.
    • Task sources: issue-and-fix pairs from real open-source repositories, manually constructed terminal-operation tasks, and a task pool mixing auto-generated and human-verified tasks. Task difficulty needs to span a gradient from “small single-file change” to “cross-module refactor.”
    • Acceptance criteria: each task ships with a hidden test script and a “fails before fix, passes after fix” judgment, ensuring the reward is objective and cannot be faked.
    • Scale and diversity: the size and diversity of the task pool directly determine generalization and resistance to overfitting. Training on only one repository family easily teaches the model that repository’s “dialect.”

    The task-construction stage should also build “difficulty buckets”: run the current policy once, record each task’s pass rate, and later prioritize tasks whose pass rate falls in the middle range. This step adds almost no cost but can raise the density of effective gradients severalfold.

    3.2 Reward Design: Make “Getting It Right” More Rewarding Than “Doing More”

    The goal of reward design is not “to give points” but “to shape behavior.” In practice, several types of signal are usually combined:

    • Result reward: whether the hidden test passes is the primary signal and carries the highest weight;
    • Process and format constraints: whether output format is followed, whether necessary verification traces are left, and whether timeouts or illegal operations occur (such as unauthorized access or damaging the environment);
    • Penalty terms: invalid turns, repeated commands, overly long context, and so on, suppressed with small negative rewards to discourage “step-padding” behavior.

    Watch especially for “reward hacking”: a model may learn to modify test files, fake passing logs, or pre-plant answers in the environment. Defenses include separating the hidden test set from the tests visible during training, making key-path files read-only or checksum-verified, and manually spot-checking high-risk trajectories. The essence of reward design is to translate “the behavior we actually want” into computable signals, not to treat “what is easy to compute” as the goal.

    3.3 Sampling and Curriculum: Spend Compute on Tasks You Can Actually Learn From

    This is the highest-ROI part of the recipe.

    • Pass-rate-driven difficulty scheduling: if a batch of tasks is “all correct” or “all wrong” for the current policy, the gradient information they contribute is close to zero. Concentrating sampling on tasks with mid-range pass rates is equivalent to concentrating compute in the “zone of proximal development.”
    • In-group sampling and relative advantage: sample a group of trajectories in parallel for the same task, use the group’s average as the baseline, and compute relative advantage. The benefit is that it naturally removes bias from task-difficulty differences and needs no separate value network.
    • Partial-trajectory continuation: sampling for long-horizon tasks is expensive. Saving the state of unfinished trajectories and continuing next round significantly raises sample utilization and avoids the waste of “getting cut off halfway.”

    Pseudocode sketch:

    1. for step in range(num_steps):
    2. tasks = curriculum.sample(target_pass_rate=(0.2, 0.8))
    3. group = [env.rollout(policy, task) for task in tasks for _ in range(G)]
    4. # In-group relative advantage: compare within the same task/group
    5. # to remove task-difficulty bias
    6. adv = group.advantage_baseline(group.rewards, per_task=True)
    7. adv = adv / (adv.std() + 1e-6)
    8. loss = policy.compute_loss(group, adv,
    9. kl_coef=kl_coef,
    10. ref_model=ref_model)
    11. optimizer.step(loss)
    12. # Refresh the curriculum distribution with the latest pass rates
    13. curriculum.update(task_pass_rates)

    3.4 Optimization Algorithm and Stability: Guardrails for Full-Parameter Updates

    Full-parameter reinforcement learning fears two things above all: policy collapse and entropy collapse. Common guardrails include:

    • Reference-model constraint: use a KL penalty or clipping to limit how far the policy drifts from the initial model, preventing it from “learning to misbehave just to score points”;
    • Advantage normalization and clipping: control the magnitude of each update so a few extreme samples do not dominate the gradient;
    • Length normalization combined with penalty terms: prevent the model from trading “writing longer” for higher reward;
    • Entropy regularization or temperature scheduling: preserve exploration and avoid converging too early to a single script;
    • Small and staged learning rates: reduce step size in the later stage of training to cut oscillation.

    None of these has a single correct answer, but they share the same goal: to make policy updates controllable, reproducible, and rollback-able.

    3.5 System Implementation: Turning “Sampling” into Throughput

    The engineering difficulty of long-horizon agentic reinforcement learning often lies not in the algorithm but in the system.

    • Decouple sampling from training: sampling (inference) and training (backpropagation) have different hardware needs. An asynchronous or semi-asynchronous architecture keeps the inference cluster continuously producing trajectories while the training cluster consumes them, substantially raising cluster utilization;
    • Inference-side optimization: batch scheduling, prefix reuse, and key-value cache management can sharply reduce the marginal cost of long contexts;
    • State management and snapshots: serializing, restoring, and concurrently isolating environment state is the prerequisite for large-scale parallelism;
    • Fault tolerance and checkpoint resume: long tasks can be interrupted at any moment by environment anomalies, so progress and rewards need to be recorded at task granularity to avoid discarding a whole batch;
    • Observability: recording each trajectory’s turns, token consumption, reward, and failure reason serves both as a basis for tuning and as raw data for cost attribution.

    3.6 Evaluation System: Don’t Let “Scores” Fool You

    • Fix the evaluation framework: the same tool-call protocol, the same sandbox, the same timeout settings. Switching frameworks often causes swings of several percentage points or more.
    • Average over multiple runs: a single run has high variance, so trials are usually averaged and confidence intervals reported.
    • Report cost alongside: pass rate, average turns, average token consumption, and per-task cost must be read together; otherwise you cannot tell whether “progress” comes from capability or simply from spending more.
    • Prevent contamination: keep the evaluation set strictly separate from the training task pool, and run periodic blind tests on undisclosed tasks.

    4. Why This Recipe Is “Cheaper”: Four Levers of Cost Attribution

    Back to the core question. Decomposing task cost reveals four levers that can be compressed:

    1. Retries: raising the pass rate from 30% to 46% significantly lowers the average number of attempts under a “retry on failure” business logic. This is the most direct way to save money;
    2. Turns: a better policy means less ineffective exploration, cutting “wasted inference” rather than “useful thinking”;
    3. Token utilization: fewer repeated reads and more precise localization directly reduce the secondary cost of context bloat;
    4. Reasoning-intensity matching: a lightweight base model does not need “maxed-out reasoning intensity”; put the compute into “getting it right” rather than “thinking longer.”

    Worth emphasizing: none of these four depends on switching to cheaper hardware or on lowering model capability. What they change is “the amount of computation required per task,” so cost reduction and efficiency gains are two sides of the same coin, not a trade-off. This is the recipe’s most valuable takeaway for enterprises: it does not require you to own a stronger model, only to use the model more intelligently.

    5. How to Build Your Own Performance-Cost Frontier

    5.1 First Define a “Task–Metric–Budget” Triple

    Before choosing a model or an algorithm, translate the business problem clearly:

    • Task distribution: what share is simple queries, and what share is complex long-horizon tasks? The cost difference between the two can reach tens of times;
    • Metric definition: is it “first-attempt pass rate” or “pass within three attempts”? The former demands more of the model; the latter allows trading “retry plus verification” for cost;
    • Budget constraints: the per-task cost ceiling, daily task volume, and peak concurrency together determine which model tiers are viable.

    5.2 Choose a Point on the “Pass Rate–Cost” Curve, Not “the Strongest”

    Plot candidate approaches on the same chart: the horizontal axis is average per-task cost (log scale), the vertical axis is the target metric. Only points on the frontier are the “best value” candidates. In engineering practice, there are three common types of selectable points:

    • Lightweight model plus low reasoning intensity plus limited retries: suited to high-frequency, low-complexity tasks, with the lowest cost;
    • Medium model plus a reinforcement-learning-optimized policy: suited to medium- and long-horizon tasks; this is the “sweet spot” the recipe targets;
    • Frontier model plus high reasoning intensity: suited to low-frequency, high-value, low-tolerance tasks, trading cost for certainty.

    5.3 Routing and Tiering: Spend Cost Where It Cuts

    Real systems rarely use just one model. A common approach is “tiered routing”: first judge task complexity (length, file count, whether cross-module reasoning is needed), send simple tasks to a low-cost channel, escalate complex tasks to a high-capability channel, and set a budget cap on escalations. Combined with result caching and reuse of similar tasks, overall cost can drop further.

    5.4 Implementation Roadmap (Usable Directly as a Project Plan)

    Stage Goal Key Actions Exit Criteria
    0. Baseline Understand the current state Run the target business task set with the existing model; record pass rate and cost A reproducible capability baseline and cost baseline
    1. Environment Meet training prerequisites Containerized sandbox, task pool, hidden tests, snapshots and rollback Tasks run repeatedly with stable scoring
    2. Reward Controllable behavior Result reward plus format and penalty terms; spot-check to prevent gaming Manual evaluation matches the automatic reward
    3. Small-scale RL Validate effectiveness Small model, small task set, a small GPU budget for a trial run A stable upward trend in pass rate
    4. Scale-up Full-parameter training Full-parameter updates, curriculum sampling, async sampling, checkpoint resume Target metrics improve and cost drops
    5. Evaluation & Launch Close the loop Fixed framework with repeated evaluation; connect routing and monitoring Online cost and pass rate match expectations

    6. Common Pitfalls and Takeaways

    1. Watching only pass rate, not cost. Many “improvements” come from raising reasoning intensity and retries, which may not pay off in business terms.
    2. Treating the evaluation framework as a constant. Switching the tool-call protocol, sandbox image, or timeout settings often causes swings of several percentage points, easily misread as “model progress” or “training failure.”
    3. Ignoring task-pool diversity. Training only on tasks from a single source overfits the model to a specific repository structure and naming convention, and online performance regresses sharply.
    4. Ignoring reward hacking. Any “computable objective” will be gamed, so hidden tests and manual spot-checks must be built in tandem.
    5. Ignoring system efficiency. When sampling throughput is insufficient, even the best algorithm cannot run; an asynchronous architecture and state management often determine the project timeline.
    6. Expecting one training run to solve everything. A more realistic approach is to break training into a continuous process of “data and rewards first, algorithm iteration next, and evaluation to close the loop.”

    Conclusion

    The signal from this public recipe is clear: the competition in long-horizon agentic capability is shifting from “whose model is bigger” to “whose recipe is better and whose engineering is more solid.” When a training run can be compressed to dozens of accelerators in under five days, and per-task cost can drop to a fraction of a flagship approach, the capability gap itself is no longer the only variable. What determines success in deployment is whether you can turn tasks, rewards, sampling, and systems into a sustainably iterating closed loop.

    For engineering teams, the actionable advice is:

    • First lock down the evaluation and cost definitions;
    • Then invest in environment engineering and reward design;
    • Only last, expand the algorithm and compute scale.

    Get the order wrong, and no amount of compute will do more than expensively repeat the same problem.