Full-Parameter Agentic Reinforcement Learning on 64 GPUs in 5 Days: The Performance-Cost Frontier

    Agentic reinforcement learning for coding and terminal agents is entering an era defined by "output per unit of compute." This article breaks down a publicly shared recipe that starts from a 300B-class open-source base model and applies full-parameter RL to train terminal and coding agents. The reported results include a 46.2% Pass@1 on the latest version of a mainstream terminal-interaction benchmark—on par with top closed-source flagship agents—while the average cost per task is more than 30x lower, and the entire run uses just 64 high-performance GPUs for under 5 days. Rather than repeating experimental details, the article distills the approach into a transferable methodology: how to define the objective around a performance-cost frontier, how to build auto-verifiable environments, how to design layered rewards, how to schedule full-parameter training, and how to evaluate results. It closes with practical guidance for engineering and enterprise readers on what to adopt and what to prepare in advance.

    Introduction: Agent Competition Is Shifting from “Smarter” to “More Cost-Effective”

    Over the past two years, the capability curve of coding and terminal agents has climbed steeply. They can read a codebase, execute commands, modify files, run tests, and iterate on their failures. What these agents have in common is that they are “long-horizon”: completing a single task often takes dozens of tool calls, context accumulates continuously, and any single misstep can cascade and amplify.

    At the same time, the enterprise adoption perspective is changing. R&D teams no longer care only about “can it get the task done,” but about “how much time and money does it cost to get one task done.” To solve the same bug, some approaches require repeated “deep thinking,” with token consumption growing several-fold; others reach the same result with a much shorter trajectory. The right coordinate for evaluating an agent has therefore shifted from a single “accuracy” axis to a two-dimensional “performance–cost” plane.

    Recently, a research team publicly shared a long-horizon agent RL recipe worth studying. Starting from a 300B-class open-source base model, they used full-parameter reinforcement learning to train terminal and coding agents. The main results can be summarized in three points:

    • Performance matches the flagship: On the latest version of a mainstream terminal-interaction benchmark, its Pass@1 reached 46.2%, on par with the top closed-source flagship agents of the same period.
    • Cost is dramatically lower: The average cost per task is more than 30x lower than that of the top closed-source flagship at its highest reasoning intensity, placing it further toward the front on the “Pass@1–cost” plane.
    • Training overhead is controllable: The entire pipeline used only 64 high-performance GPUs and took less than 5 days, with the full recipe published.

    Note: The model was self-deployed by the team. Costs are converted based on the public pricing of the open-source base it builds on, and do not include the premium of the closed-source flagships.

    This article does not restate the experimental details. Instead, it decomposes these results into a transferable methodology: how the objective should be defined, how the environment should be built, how the reward should be designed, how full-parameter training should be scheduled, and how results should be evaluated. The goal is to help developers and enterprise readers judge which practices are worth borrowing and which conditions need to be prepared in advance.

    Redefining the Objective: From Score Chasing to the Performance-Cost Frontier

    Why “Accuracy” Is Not Enough

    Common agent evaluation metrics include Pass@1, Pass@k, and success rate. Pass@1 is the probability that the model produces a correct result in a single sample; it most closely matches real usage, where a user submits a task once and usually does not retry repeatedly. Pass@k allows multiple samples and takes the best, which is better suited to measuring a model’s capability ceiling.

    Looking at accuracy alone leads to one misconception: trading “unlimited compute” for a higher score. Pushing the model to use longer chains of thought and more retries will raise the score, but the cost per call usually climbs in tandem, with diminishing marginal returns. In production, such a “high-scoring model” may not be usable at all.

    The Frontier Idea: No Single Best, Only Non-Dominated

    The “performance–cost frontier” borrows the idea of the Pareto frontier. In an “accuracy–cost” coordinate system, each approach corresponds to a point. If another approach exists that achieves no lower accuracy while costing strictly less, the original approach is “dominated”; the boundary formed by the points that are not dominated by any approach is the frontier.

    Approaches on the frontier have no absolute ranking; the choice depends on business constraints:

    • Cost-sensitive: For large-scale automated batch fixes, prefer the lower-cost end of the frontier.
    • Quality-first: For core logic changes in critical modules, a higher cost may be acceptable in exchange for a higher success rate.

    This perspective has direct implications for selection:

    1. Do not compare across coordinate systems: scores obtained under different harnesses, sampling counts, and temperature settings are not comparable.
    2. Do not ignore fixed costs: model deployment, environment maintenance, and human review all get amortized into the per-task cost.
    3. Pay attention to the cost of failed tasks: failures also consume tokens, so raising the success rate directly lowers the average cost.

    Reasoning Intensity: An Easily Overlooked Cost Curve

    Many closed-source models offer multiple levels of reasoning intensity; the higher the level, the longer the model “thinks” before answering. There is already substantial academic research on this, and common inference-time scaling methods in industry include:

    • Chain-of-thought prompting;
    • Multi-path sampling followed by voting or selection;
    • Search guided by process rewards;
    • A more generous thinking budget in the prompt.

    Their common pattern is: trade extra compute at the inference stage for accuracy, with diminishing returns. The following table is a purely illustrative example (all numbers are hypothetical) to show that “the highest intensity is not necessarily the best”:

    Approach Reasoning intensity Avg. output tokens per task Cost per task (relative) Pass@1 Cost per success (relative)
    A Low 3,000 1.0 28% 3.6
    B Medium 6,000 2.0 36% 5.6
    C High 15,000 5.0 40% 12.5
    D Max 30,000 10.0 42% 23.8

    Note: cost per success = cost per task ÷ Pass@1.

    From A to D, Pass@1 improves by only 14 percentage points, while the cost per task rises to 10x. The reason is that cost grows roughly linearly with reasoning intensity, while accuracy gradually saturates. More importantly, once “cost per success” is introduced, the ranking matches Pass@1 exactly: A is lowest, while C and D deteriorate sharply because so many tokens are spent on attempts that never produce a successful trajectory.

    This is precisely where RL post-training creates an opportunity. Writing “get it right in one shot within a reasonable budget” into the reward teaches the model to plan steps, correct errors promptly, and avoid ineffective loops. Performance gains then no longer depend solely on “thinking longer,” but on “doing it more correctly.”

    Choosing the Starting Point: Why Begin from a 300B-Class Open-Source Base

    Post-Training, Not Pre-Training from Scratch

    Pre-training a base with strong code and tool-use ability typically requires a large-scale cluster and a cycle of several months, a very high barrier. By comparison, post-training (supervised fine-tuning, reinforcement learning) is more like “targeted shaping on top of existing capabilities”:

    • The base already masters syntax, common library usage, and basic command-line semantics;
    • What RL needs to solve is planning, error correction, and tool-calling strategy in long-horizon tasks.

    Choosing a 300B-class open-source base is a compromise in terms of cost structure:

    • Sufficient capability: Models at this scale usually have strong reasoning and instruction-following ability, which is a prerequisite for post-training to yield gains. Small models are limited by a “capability ceiling,” leaving little room for RL to unlock.
    • Controllable cost: Compared with larger flagship models, inference cost is significantly lower, which directly determines the lower bound of per-task cost.
    • Controllable and deployable: Open weights allow self-deployment and free sampling, so the massive rollouts needed for training are not constrained by pay-as-you-go billing, and deep throughput optimization is possible.

    A Checklist for Base Selection

    Not every open-source base is suitable for agent post-training. It is worth evaluating along these dimensions:

    • Instruction following and tool calling: Can it stably emit structured calls (e.g., a unified command or function-call format)? Is the format-violation rate low enough?
    • Long-context capability: Can the target window hold dozens of interaction turns? Is mid-context information easily forgotten?
    • Inference throughput: Generation speed per unit of compute, which determines whether rollouts can run fast enough.
    • License: Are the restrictions on commercial use and redistribution compatible with business requirements?

    Building the Environment: The First-Principle Bottleneck of Long-Horizon Agent Training

    In long-horizon agent RL, the model and the algorithm are often not the scarcest resources; a high-quality, automatically verifiable environment is. There are three reasons:

    1. Tasks must be objectively gradable: Terminal tasks are a natural fit—success or failure can be judged by unit tests, compilation results, file diffs, and command exit status.
    2. Feedback latency: For long tasks, going from initiation to a result can take tens of seconds to minutes, and environment throughput directly determines training speed.
    3. Reproducibility: Every rollout must start from the same initial state, or reward noise will drown out the learning signal.

    Isolation, Snapshots, and Concurrency

    Common engineering practice is as follows:

    1. Build an independent container image for each task, containing the project code, dependencies, and runtime, to avoid “the environment won’t install” polluting the reward signal.
    2. Restore a snapshot before the task starts, and reclaim the container after it finishes, so that each rollout is independent of the others.
    3. Maintain an elastic container pool that scales up and down with concurrency demand, avoiding task queuing that slows sampling.
    1. Task queue
    2. │
    3. ▼
    4. ┌──────────────────── Container pool (concurrency N) ────────────────────┐
    5. │ Container 1: snapshot → execute trajectory → run verification → compute reward → reclaim │
    6. │ Container 2: snapshot → execute trajectory → run verification → compute reward → reclaim │
    7. │ ... │
    8. └───────────────────────────┬───────────────────────────────────────────┘
    9. ▼
    10. Experience samples (trajectory + reward) → RL training

    The container pool is responsible for “producing trajectories in bulk,” while the training side is responsible for “consuming trajectories and updating parameters.” The two are decoupled through a queue, which is the prerequisite for the asynchronous sampling and throughput optimization discussed later.

    Task Sources and Difficulty Distribution

    High-quality tasks mostly come from real-world scenarios, for example:

    • Extracting “problem–patch” pairs from the historical issues and fix commits of real open-source projects;
    • Distilling high-frequency operations from terminal usage logs;
    • Internal defect tickets, work orders, and code-review records within an enterprise.

    When building the task set, note three things:

    • Difficulty stratification: Mix easy, medium, and hard tasks in certain proportions. Purely easy tasks give a weak gradient signal, while purely hard tasks fail almost entirely with low reward variance; neither is good for learning.
    • Hidden verification: The tests used for verification should be invisible during inference, to prevent shortcut behavior such as “writing code to pass the test.”
    • Preventing data leakage: When constructing tasks from public projects, check whether the verification cases already exist in the model’s pre-training corpus; otherwise, evaluation results will be inflated.

    Reward Design: From “Is the Result Correct?” to “Is the Process Learnable?”

    Outcome Reward: Sparse but Reliable

    The most direct reward is “did verification pass”: give a positive reward if all tests pass, and 0 or a negative reward on failure. This reward is objective and hard to game, but in long-horizon tasks it is extremely sparse—a trajectory has dozens of steps, but feedback arrives only at the end, making credit assignment very difficult.

    Layered Reward: A Denser Signal

    A common industry practice is to split the reward into several layers:

    Reward layer Typical signal Purpose Risk
    Outcome (main) Hidden test pass rate, whether the command succeeded Aligns with the true objective Verification must not be tamperable by the execution process
    Format & constraint Valid call format, adherence to constraints (e.g., no network access) Reduces ineffective trajectories Weight should not be too high; avoid “format over result”
    Process/subgoal Completing key intermediate steps, passing some tests Alleviates sparsity Easily exploited by “step farming”; needs outcome-reward constraints

    Constraint and Efficiency Penalties

    • Length penalty: Applying a mild penalty on steps or tokens can suppress ineffective loops and redundant reasoning. This is precisely how the “performance–cost frontier” is reflected at the reward level.
    • Format and failure termination: Trajectories with invalid formats or that trigger hard-failure conditions can be terminated early, saving sampling compute.
    • Priority: Constraint-type and efficiency-type rewards should only be auxiliary. The dominant term is always the outcome reward; otherwise, the model may learn to “fail compliantly.”
    1. def compute_reward(traj, env):
    2. """Compute the reward for a complete trajectory (illustrative, not a concrete implementation)."""
    3. # 1. Format violation or a hard-abort condition: end early to avoid wasting sampling budget
    4. if traj.format_violation or traj.hard_abort:
    5. return -0.1
    6. # 2. Main reward: pass rate on hidden tests (0~1)
    7. reward = env.run_hidden_tests(traj.workspace).pass_rate
    8. # 3. Constraint reward: whether the task's given usage restrictions were respected
    9. reward += 0.05 * int(traj.constraints_satisfied)
    10. # 4. Mild efficiency penalty: suppress verbose, ineffective trajectories (weight far below the main reward)
    11. reward -= 1e-3 * traj.total_tokens
    12. return reward

    Curriculum Learning and Adaptive Sampling

    A fixed-difficulty task set can easily stall training: either “there is nothing new to learn,” or “it is too hard to make progress.” A common approach is to dynamically adjust the sampling distribution:

    • Track each task’s recent success rate;
    • Prioritize sampling tasks whose success rate sits in a middle band (e.g., 20%–80%), where the gradient signal is strongest;
    • For tasks that fail consistently over a long period, lower their sampling frequency or first fix the environment, for example by completing dependencies or adjusting verification scripts;
    • For tasks that already pass reliably, reduce sampling and give the compute to tasks “on the boundary.”

    This effectively achieves “automatic difficulty adjustment” at the training level, spending the limited sampling budget where the information content is highest.

    Full-Parameter RL: Why “Full” Matters

    Full-Parameter vs. Parameter-Efficient Differences

    Parameter-efficient fine-tuning (such as low-rank adaptation methods) updates only a small number of additional parameters. It has low memory overhead and is simple to implement, and is widely used in instruction tuning. But in long-horizon agent RL, full-parameter updates have two irreplaceable advantages:

    1. Behavior shaping requires deep participation: Long-horizon policies rely on the model’s deep planning and reasoning ability. Adjusting only surface-level additional parameters makes it hard to change the model’s overall behavior pattern across dozens of interaction steps.
    2. Avoiding distribution mismatch: In RL, the sampling policy and the training policy need to stay consistent. When most parameters are frozen, policy updates are constrained, and it is easy for “sampled trajectories to drift out of sync with the updated policy,” destabilizing training.

    The cost is memory and engineering complexity. Taking a 300B-class model with a mixed-precision Adam-type optimizer as an example (a general estimate, to convey the order of magnitude):

    • Parameters, gradients, and optimizer first- and second-order moment states together require roughly 300B × 16 Bytes ≈ 4.8 TB of GPU memory;
    • This does not yet count activations, the KV cache, and communication buffers;
    • If a separate frozen reference model is introduced for a KL constraint, another ~0.6 TB is needed.

    Clearly, a single GPU cannot hold this, so sharding and parallelism strategies are required.

    General Combinations of Parallelism

    Parallelism / optimization Role Caveats in long-horizon RL
    Data parallelism + optimizer state sharding Shards parameters, gradients, and optimizer state across GPUs to cut per-GPU memory High communication volume; overlap it fully with compute
    Tensor parallelism Splits single-layer compute across GPUs; suits models with very large per-layer parameters Needs high inter-GPU bandwidth; keep it within a single node
    Pipeline parallelism Splits layers across GPUs to reduce the number of layers per GPU Schedule micro-batches to reduce pipeline “bubbles”
    Sequence/context parallelism Splits very long sequences to cut long-context activation overhead Nearly mandatory for long-trajectory training
    Activation recomputation Trades compute for memory; recomputes forward activations during backpropagation Adds about 20%–30% compute; decide based on available memory
    Memory offloading Temporarily moves some state to host memory or storage Bandwidth is the bottleneck; use only as a fallback

    Real systems usually combine several of these approaches, and there is no single “correct” configuration; trade-offs must be made based on model size, sequence length, and the cluster’s interconnect conditions.

    Stability Essentials

    Full-parameter RL is prone to “collapse during training,” so in practice it is usually necessary to watch for:

    • KL constraint: Limits how far the new policy deviates from the reference model, preventing capability collapse and output degradation;
    • Gradient clipping and a conservative learning rate: RL gradient variance is far higher than in supervised learning;
    • Advantage normalization: Reduces the impact of reward-scale differences across tasks;
    • Entropy monitoring: A sudden drop in entropy often means premature convergence to a single mode, while an entropy explosion may mean the training is diverging.

    Training Scheduling: Finishing on 64 GPUs in Under 5 Days

    Where Does the Time Go: The Contest Between Sampling and Training

    The time in long-horizon agent training is spent mainly in two places:

    • Generation (rollout): The model interacts with the environment over multiple turns, constrained by environment execution speed, model generation speed, and task length;
    • Training (backward update): Forward pass, backward pass, and parameter synchronization.

    If the same set of GPUs first runs all rollouts to completion and then trains all at once, you get “training GPUs sitting idle during generation, and generation GPUs sitting idle during training.” The mainstream approach is separating sampling from training in an asynchronous pipeline:

    • One group of GPUs is dedicated to inference generation (using a high-throughput inference engine with continuous batching) and continuously feeds trajectories into the experience queue;
    • Another group of GPUs handles training and consumes the experience queue;
    • The two are decoupled through the queue, so their throughput does not block each other.
    1. Inference GPU pool ──generate trajectories──▶ Experience queue (capacity cap) ──▶ Training GPU pool
    2. ▲ │
    3. └──────────── periodic sync of latest weights ◀────────────────┘

    The “strategy lag” introduced by asynchrony (the weights used for sampling are slightly older than the training weights) must be controlled: too long a lag breaks policy consistency. Common approaches include setting a maximum lag in steps, down-weighting older samples, and appropriately strengthening the KL constraint.

    Balancing Memory and Compute

    When inference and training share the memory budget, common strategies include:

    • Separate deployment: Inference and training clusters are independent, sized to match throughput needs;
    • Co-located time-sharing: During the training phase, temporarily offload optimizer states to free memory for the inference engine’s KV cache, then swap back when switching.

    The former gives more stable resource utilization; the latter gives higher hardware utilization and lower cost, but with significantly higher engineering complexity.

    What “64 GPUs, Under 5 Days” Demonstrates

    The value of this recipe is that the barrier to full-parameter RL training of long-horizon agents has dropped from “only top-tier institutions can afford it” to “an experienced engineering team can reproduce it.”

    For comparison, multiple industry studies have shown that pre-training a base of the same scale can require thousands or even tens of thousands of accelerator cards and weeks or more; while full-parameter post-training on an existing base often needs only a few tens of a fraction—or less—of those resources.

    As a result, what truly determines “how fast you can produce a usable agent” is no longer “whether you have enough GPUs,” but:

    • How ready the data and environment are;
    • The level of automation in the training pipeline;
    • The efficiency of locating failures and tuning parameters.

    Interpreting the Results: Where Do Capability Transfer and Cost Advantage Come From?

    Cross-Benchmark Transfer: Not “Single-Point Score Chasing”

    The team reports that the advantage is not only shown on the terminal-interaction benchmark but also transfers to two types of software-engineering benchmarks:

    • One is an end-to-end defect-repair evaluation targeting real code repositories;
    • One is a high-quality software-engineering problem evaluation that has been human-verified.

    The two have different task shapes—the former leans toward “locating and fixing in a real repository,” the latter toward “producing the correct patch from a problem description”—but both share common demands on long-horizon planning, code understanding, and verification. Leading on both benchmarks with the same model indicates that what the training learned is not surface techniques tailored to a particular interface, but more general agentic behavior capabilities.

    That said, it should be viewed rationally:

    • Evaluations are mostly done in the team’s self-built harness, which may differ from official leaderboards in tool set, prompt templates, and maximum steps;
    • The base model’s official scores and the post-trained model are not a controlled comparison under the same conditions, so cross-comparison should be cautious.

    Four Sources of the Cost Advantage

    1. Base selection: Starting from a cost-effective open-source base rather than the most expensive closed-source flagship.
    2. Reasoning-intensity optimization: Through RL, learning to “take fewer detours,” reducing ineffective long chains of thought and repeated trial and error.
    3. Trajectory efficiency: Step penalties and format constraints make the model more inclined to complete tasks via shorter paths.
    4. Self-deployment: The deployment freedom brought by open weights lets inference cost be optimized for actual throughput, rather than passively accepting a vendor’s pricing.

    These four points correspond exactly to a shift of the “performance–cost frontier”: lower cost at the same performance, or higher performance at the same cost.

    Implementation Lessons: How Enterprises Can Reproduce and Borrow from This

    Build the Environment First, Then Talk About the Algorithm

    For most enterprises, the scarcest resource is not GPUs but a “task set that can be graded automatically.” The recommended path:

    1. Inventory scenarios: Start with high-frequency scenarios whose results are verifiable, such as automated test repair, log-troubleshooting script generation, and configuration inspection.
    2. Accumulate tasks: Organize historical work orders, defect tickets, and postmortem records into standardized tasks of “input + verification method.”
    3. Containerize: Build quickly restorable container images for each task type to ensure rollouts are reproducible.
    4. Validate at small scale: First prove feasibility with parameter-efficient fine-tuning or prompt engineering, then invest in full-parameter RL.

    A Training-Side Checklist

    • Base evaluation: Tool-call format compliance rate, long-context stability, and inference throughput—clear all three before starting;
    • Reward system: Outcome reward as the main term, with format/process/efficiency rewards as auxiliary; tune weights in small steps;
    • Parallelism strategy: Prioritize tensor parallelism within a single node; use data and pipeline parallelism across nodes;
    • Asynchronous sampling: Set the experience-queue capacity and a maximum policy lag to avoid stale samples;
    • Stability monitoring: KL divergence, entropy, gradient norm, average reward, and average steps—intervene promptly if any metric fluctuates abnormally.

    Cloud Compute: Balancing Elasticity and Cost

    For most enterprises, building a thousand-GPU-scale cluster in-house is not economical. A more realistic path is a “hybrid”: steady-state inference services run on owned or annual-commitment resources, while peak compute during the training phase is supplemented by elastic cloud resources. For example, a cloud provider such as Baidu AI Cloud may offer:

    • GPU cloud servers: For training and inference, with billing options such as pay-as-you-go or annual/monthly commitment;
    • Container services: Hosting the environment pool and inference services, with elastic scaling to handle peaks;
    • Object storage: Storing training data, container images, and checkpoints;
    • Monitoring and alerting services: Tracking key metrics such as GPU utilization, task success rate, and token consumption.

    Cost Measurement: Build Your Own “Cost per Success” Dashboard

    It is worth bringing the following metrics into daily monitoring:

    Metric Definition Use
    Avg. cost per task Input + output tokens converted to cost per task Cross-compare approaches
    Pass@1 Single-sample success rate Quality baseline
    Cost per success Cost per task ÷ Pass@1 Overall cost-efficiency
    Avg. steps / tokens Interaction rounds and context consumed to complete a task Locate “inefficient trajectories”
    Environment failure rate Share of invalid rollouts caused by environment issues Safeguard training-signal quality

    The “environment failure rate” is especially easy to overlook: when a large share of rollouts fail due to missing image dependencies, network timeouts, or verification-script errors, the model is actually receiving noisy rewards, and training effectiveness drops sharply.

    Common Pitfalls

    1. Score-only thinking: Watching leaderboards without watching cost, then losing control of the bill after launch.
    2. Parameter-count-only thinking: Blindly chasing a larger base while ignoring “output per unit of compute.”
    3. Imbalanced reward design: Overweighting process rewards, so the model learns to “perform steps.”
    4. Ignoring environment quality: Treating an engineering problem as an algorithm problem.
    5. Ignoring evaluation consistency: Comparing scores across different harnesses, which distorts conclusions.

    Conclusion

    This recipe—64 GPUs, 5 days, and more than a 30x reduction in per-task cost—conveys a core signal: competition among long-horizon agents has entered the era of “output per unit of compute.”

    For enterprises, three takeaways are worth keeping:

    1. Change the coordinate of the objective: Shift from “highest accuracy” to the “performance–cost frontier,” using “cost per success” as the core metric.
    2. Treat the environment as a first-class citizen: An auto-verifiable, quickly restorable task environment matters more to training success than the choice of model.
    3. Full-parameter RL is now affordable: With elastic cloud compute and a mature open-source training stack, a mid-sized team can complete a high-quality post-training run within an acceptable cycle.

    An agent’s true moat lies not in leading a single leaderboard, but in whether it can continuously, cost-effectively, and reproducibly turn model capability into business results.