Low-Cost Reinforcement Learning for 300B-Class Terminal and Coding Agents

    This article reviews a published practice for training terminal and coding agents with full-parameter reinforcement learning on a 300B-class base model. It covers the four-layer system design, verifiable-reward formulation, isolated task environments, curriculum sampling, cost accounting, evaluation and transfer, and key risks. It also clarifies the caveats around the reported 46.2% Pass@1 and the roughly 30x per-task cost difference under a specific comparison setting.

    1. Background: The Capability–Cost Tension in Terminal and Coding Agents

    Terminal and coding agents must complete multi-step, highly interactive, verifiable tasks: read environment state, execute commands, modify files, run tests, revise plans based on feedback, and reach an expected final state. Compared with single-turn question answering, these tasks place higher demands on reasoning stability, tool-call accuracy, long-context management, and error recovery.

    But capability gains often come with higher cost: larger parameter counts, higher inference intensity, and longer trajectory sampling all increase per-task cost. How to train usable terminal and coding agents under limited compute is a core engineering question. A publicly disclosed practice from a research team offers a reference path: starting from a 300B-class base model, the team used full-parameter reinforcement learning and completed training in under five days on 64 high-compute accelerators. The disclosed data shows Pass@1 of 46.2% on a public terminal task benchmark, close to comparable high-capability systems at the same level, while under a specific controlled comparison, per-task cost can be about 30x lower. The improvement also transferred to two software engineering benchmarks.

    It is important to stress that the 30x cost gap is a result under a specific task distribution, comparison configuration, and inference intensity. It should not be directly generalized as a universal conclusion for all scenarios. Its engineering ideas are still worth breaking down: use verifiable rewards to drive full-parameter reinforcement learning, use isolated environments to ensure trustworthy evaluation, use curriculum and sampling controls to manage training cost, and use a unified recipe to improve reproducibility.

    2. Solution Overview: From Base Model to Verifiable Agent

    The overall solution can be abstracted into four layers:

    1. Base model layer: Choose a 300B-class general base model with code understanding, terminal operation, and long-context capabilities.
    2. Task environment layer: Build a pool of terminal and coding tasks. Each task includes an initial state, goal description, verifier, and resource constraints.
    3. Reinforcement learning layer: Use full-parameter updates rather than training only lightweight adapters. Use executable verification results as the primary reward signal, supplemented by cost and trajectory constraints.
    4. Evaluation and reproducibility layer: Fix environment snapshots, hold out task sets, standardize timeouts and resource quotas, and publish the training recipe and key hyperparameters.

    The value of full-parameter reinforcement learning is that it adapts the model’s overall policy to terminal tasks, not just local output style. The cost is greater pressure on memory, communication, and stability, so it needs engineering support such as parallel sampling, gradient checkpointing, parameter sharding, mixed precision, and asynchronous evaluation.

    3. Training Recipe: Rewards, Environments, and Curriculum

    3.1 Reward Design: Verifiable Outcomes First

    The advantage of terminal tasks is that outcomes can be verified. The reward function should prioritize objective signals such as test passes, target-state achievement, and successful command execution, then add a small number of process constraints. This avoids rewarding the model for producing plausible-looking text.

    Example pseudocode:

    1. Initialize policy parameters θ
    2. Prepare task pool T, verifier V, isolated environment E
    3. for step in 1..N:
    4. Sample a batch of tasks t ~ T
    5. Execute trajectories in parallel in E: τ = rollout(θ, t)
    6. Compute reward r = V(τ, t) - λ·cost(τ) - μ·risk(τ)
    7. Estimate advantage A = group_normalize(r)
    8. Compute policy gradient loss L = -E[A·log πθ(τ)]
    9. Update all parameters θ
    10. if step % K == 0:
    11. Evaluate pass@1, per-task cost, and stability on held-out tasks

    Here, V should cover unit tests, target file state, command exit codes, final artifact validation, and similar checks. cost can constrain trajectory length and the number of tool calls. risk is used to penalize dangerous operations or unauthorized access. The key to reward design is that it should be executable, reproducible, and hard to game.

    3.2 Isolated Environments: Keep Training and Evaluation Consistent

    When an agent executes commands in a terminal, it must run in an isolated, snapshot-capable, and rollback-capable environment. Each task should have a deterministic initial state, dependency versions, resource quota, and timeout policy. Training and evaluation should use the same environment images and verifiers to avoid distribution shift where a task passes during training but fails during evaluation.

    From an engineering perspective, teams can use a general container platform, object storage for task snapshots, logging services for trajectory records, and monitoring and alerting for anomalies. The environment should restrict network access, filesystem permissions, and resource consumption to prevent reward hacking and security risks.

    3.3 Curriculum Learning: Easy First, Then Hard, with Controlled Sampling Cost

    Terminal tasks vary widely in difficulty. Direct mixed sampling can overrepresent easy tasks, making it hard for the model to learn long-horizon repair. Sampling only hard tasks lowers success rates and wastes compute. A practical approach is dynamic stratification by verification pass rate: early stages emphasize short trajectories and single-file tasks; middle stages add multi-file modifications and test repair; later stages introduce long-tail, multi-step, and error-recovery tasks. Difficulty sampling should be tied to training progress, maintaining challenge without producing only failed samples.

    4. Cost Control: Separate Training Cost from Inference Cost

    The disclosed “30x lower per-task cost” mainly reflects unit task overhead on the inference and deployment side. It is not equivalent to total training cost. Cost analysis should be split into three parts:

    • Training cost: Training on 64 accelerators in under five days suggests that full-parameter reinforcement learning, rather than large-scale multi-stage training, compressed the training cycle. However, full-parameter updates require high memory and communication, so parallel strategies and checkpoint optimization are needed.
    • Sampling cost: Reinforcement learning depends on large-scale trajectory sampling. Curriculum sampling, early stopping, parallel isolated environments, and cached verification results can reduce invalid rollouts.
    • Inference cost: Per-task cost depends on model size, inference intensity, number of tool calls, and retry strategy. Lower inference intensity with a stable policy may be more economical than high inference intensity with repeated retries.

    Therefore, the 30x gap should be understood as “significantly improved unit-task economics under a specific comparison configuration.” When implementing this, teams should establish their own cost accounting: measure average tokens per task, tool calls, sandbox duration, retry rate, and pass rate, then calculate the comprehensive cost of “passing one task” rather than looking only at per-call price.

    5. Evaluation and Transfer: Look Beyond a Single Benchmark

    The practice reached 46.2% Pass@1 on a public terminal task benchmark and showed transfer on two software engineering benchmarks. For engineering teams, evaluation should cover at least:

    1. Terminal tasks: command execution, file modification, environment repair, and multi-step troubleshooting.
    2. Software engineering tasks: defect localization, patch generation, test passing, and regression risk.
    3. Stability: variance in success rate when the same task is run multiple times.
    4. Safety: dangerous-command blocking, permission boundary violations, and data-leak protection.
    5. Cost: average trajectory length, sandbox duration, and retry count for passing a task.

    Transfer capability suggests that reinforcement learning learns not just a benchmark-specific pattern, but a general “observe–execute–verify–revise” policy. However, benchmark contamination and overfitting must be guarded against: training tasks and evaluation tasks should be strictly isolated, verifiers should not be exposed to the model, and evaluation sets should be refreshed regularly.

    6. Risks and Boundaries

    Full-parameter reinforcement learning for terminal agents carries several risks: reward hacking may exploit verifier loopholes; long-tail tasks may fail due to insufficient sampling; improperly configured isolated environments can introduce security risks; and full-parameter training has high compute and stability requirements, making direct replication costly for small teams. A sensible approach is to first validate the reward function, environment snapshots, and evaluation process on smaller models and subtask sets, then gradually expand parameter scale and task complexity.

    7. Practical Recommendations

    • Build a verifiable task set and a unified verifier before starting reinforcement learning.
    • Keep rewards primarily objective, with process constraints as a supplement, and audit for reward hacking regularly.
    • Share environment snapshots between training and evaluation, and fix timeouts, resources, and dependencies.
    • Use curriculum sampling and dynamic difficulty control to manage sampling cost, and record the cost of each trajectory.
    • Full-parameter training requires parameter sharding, gradient checkpointing, mixed precision, and asynchronous evaluation.
    • When publishing a recipe, also publish the environment, hyperparameters, evaluation criteria, and cost accounting method.

    Conclusion

    The core lesson of this practice is not a single number, but an engineering loop: use a 300B-class base model as the foundation, align it to verifiable tasks with full-parameter reinforcement learning, ensure trustworthy evaluation with isolated environments, improve economics with curriculum and cost constraints, and ultimately obtain transferable capability on terminal and software engineering tasks. For teams that want to build intelligent coding and terminal operations capabilities, getting rewards, environments, and cost accounting right is more important than blindly pursuing larger models or higher inference intensity.