Low-Card Short Training: Cost-Efficient Long-Horizon Agent Reinforcement Learning

    This article explains a practical recipe for long-horizon agent RL: full-parameter training of a 300B-class model on 64 accelerators in under five days, reaching 46.2% Pass@1 on terminal interaction benchmarks at more than 30× lower per-task cost, with transfer to software-engineering tasks. It covers trajectory sampling, verifiers, stability, and cost metrics.

    1. Background: The Core Tension in Long-Horizon Agent Training

    Terminal and coding agents are moving from single-turn Q&A to multi-turn execution. They must read files, call tools, run commands, observe feedback, and correct mistakes in real or near-real environments until a verifiable task is done. Compared with ordinary text generation, these tasks have long trajectories, complex state, and sparse rewards. A task may span dozens of turns; one bad action can derail the rest of the trajectory. Rewards often appear only at the end via tests, exit codes, or a verifier, with little intermediate supervision.

    Teams therefore trade off performance, cost, and reproducibility. Better performance usually means larger models, more samples, longer trajectories, and stronger verifiers. Cost grows with sampling rounds, environment concurrency, and retries. Reproducibility needs transparent data construction, reward functions, hyperparameters, and evaluation. The approach discussed here is a useful data point: full-parameter RL of a 300B-class model on 64 high-performance accelerators in under five days, 46.2% Pass@1 on a terminal interaction benchmark, more than 30× lower per-task cost than a comparison setup, with transfer to software-engineering tasks.

    The point is not a single score. It shows that long-horizon agent RL does not have to depend on huge clusters and very long runs. With a sound training recipe, reward design, and cost accounting, teams can find a better Pareto frontier between quality and spend.

    2. Recipe: From Base Model to Full-Parameter RL

    2.1 Full-parameter training, not adapter-only fine-tuning

    The method starts from a general base model and updates all parameters rather than a lightweight adapter. Full updates keep policy capacity intact so the model can jointly improve tool-call format, error recovery, long-horizon planning, and code understanding. The downside is higher demand on memory, communication, and stability. Engineering typically combines parameter sharding, optimizer-state partitioning, gradient checkpointing, hybrid parallelism, and recomputation to train a 300B-class model on a limited card count.

    Full-parameter training is not blindly scaling. Training signal must cover what the agent actually needs: observe the environment, choose actions, handle exceptions, and verify results. Reward only on the final answer and the model may learn to guess; reward only on intermediate format and it may ignore task completion. Full-parameter RL must therefore be paired with long-horizon sampling and executable verifiers.

    2.2 Long-horizon sampling and environment isolation

    Terminal and coding work naturally fit executable environments. On each sample, the agent operates a filesystem, runs commands, reads output, and continues from feedback inside an isolated sandbox. To avoid pollution and abuse: per-task containers or snapshots; CPU/memory/time limits; network controls; filesystem rollback; full audit logs. Coding tasks also need repos, dependencies, and tests so rewards are repeatable.

    Manage samples by difficulty. Easy tasks stabilize early training; hard tasks raise the ceiling. Do not drop failed trajectories without analysis—command errors, missing deps, failed tests, timeouts, or reward hacking. Curriculum and dynamic sampling concentrate compute on informative traces.

    2.3 Reward design and anti-hacking verifiers

    Long-horizon rewards usually come from a verifier. Terminal tasks can use exit codes, target file state, output match, and partial completion. Coding tasks can use unit-test pass rate, regression tests, static checks, and hidden tests. To ease sparsity, use staged scores, pass ratios, and recovery bonuses—without over-shaping toward “looks correct” intermediate states.

    Harden verifiers against reward hacking: hidden tests, randomized inputs, restricted file permissions, no access to answers, human spot-checks of unusually high scores. Multi-trial averaging matters; the reported numbers use a 5-run average to reduce noise. Disclose evaluation-framework differences: sandbox, timeouts, and test sets all affect comparability.

    2.4 Stability and cost control

    Full-parameter RL can suffer policy drift, reward collapse, and credit assignment over long traces. Common tools: KL constraints, gradient clipping, LR warmup, advantage normalization, periodic offline eval, and checkpoint rollback. Cap max steps, use truncation, and assign credit in segments so one huge trajectory does not wreck batch statistics.

    Cost is driven more by environment sampling and verification than by text forward passes. Reduce it with dynamic sampling, early stop, reusable environment caches, parallel sandboxes, and sampling budgets by difficulty. Do not stop at per-token price; track cost per successful task or total cost to a target score. Plot performance and cost together to judge the Pareto frontier.

    Illustrative training loop:

    1. for batch in long_horizon_tasks:
    2. trajectories = []
    3. for task in batch:
    4. traj = agent.run(env=task.env, max_steps=N)
    5. reward = verifier.score(traj, hidden_tests=task.tests)
    6. trajectories.append((traj, reward))
    7. advantages = normalize(rewards=trajectories.rewards)
    8. policy.update_full_params(trajectories, advantages)
    9. if step % eval_interval == 0:
    10. score, cost = evaluate(policy, dev_tasks)
    11. log_pareto_point(score, cost)

    3. Reading the Results: Performance and Cost Together

    On a terminal interaction benchmark, Pass@1 is 46.2%, in line with strong contemporaneous systems. More important, per-task cost is more than 30× lower than a high-inference-intensity comparison. Cost should be computed on comparable compute, sampling, and environment execution—not list price of API calls. Because the team hosts the model, state whether cost is equivalent FLOPs, public on-demand pricing, or all-in including sandbox runtime.

    Capability also transfers to software-engineering benchmarks: full-parameter RL improves not only command sequences but cross-file understanding, use of test feedback, and bug fixing. Reproducers should note that frameworks, sandboxes, and test sets change absolute scores. A public recipe should include training curves, failure cases, and cost–performance plots, not only the best number.

    4. Value of a Public Recipe and Reproduction Checklist

    A public recipe turns a result into a verifiable method. A complete agent-RL recipe should include: base model size and init, task construction and difficulty tiers, environment images and dependencies, reward function and hidden-test policy, hyperparameters, parallelism, eval scripts, logged metrics, and failure analysis. Record compute, sampling steps, pass rates, and per-task cost for each run.

    Main risks: environment drift, verifier bugs, and benchmark contamination. Evaluate on a closed test set, refresh tasks, avoid train/test overlap, spot-check high-scoring traces, and adversarial-test the reward. Reliable verifiers make RL signal reliable.

    5. Engineering Recommendations

    To build a similar stack, start small and verifiable: isolated sandboxes on a container platform, object storage for traces and artifacts, logs of execution, alerts on timeouts, failure rate, and resource use. Add task queues and permission isolation so sampling is parallel, rollback-capable, and auditable. Validate rewards on small models and short traces before full-parameter, long-horizon runs.

    Evaluation should keep three rules: multi-trial averages, hidden tests, and cost logging. Report Pass@1, cost per successful task, average interaction steps, and failure-type mix—not a one-shot high score. For coding, also watch regression pass rate and maintainability so models do not overfit tests.

    6. Conclusion

    Training 300B-class terminal and coding agents with modest compute and short schedules is feasible. The keys are full-parameter RL to unlock the base model, executable environments and reliable verifiers for signal, long-horizon management and stability techniques for risk, and cost per successful task as the efficiency metric. 46.2% Pass@1 on a terminal benchmark, per-task cost a small fraction of a comparison system, and transfer to software-engineering tasks show that quality and cost need not be an either/or. A practical path is to first lock down reproducible rewards and cost accounting, then scale model size and environment complexity into a sustainable long-horizon agent training loop.