A Practical Recipe for Long-Horizon Agent RL Under Limited Compute
This article explains how to train long-horizon terminal and coding agents with full-parameter reinforcement learning on a modest GPU cluster. It covers verifiable environments, reward design, training systems, cost control, evaluation, and a staged roadmap for reproducing stable, lower-cost agent behavior.
A Practical Recipe for Long-Horizon Agent RL Under Limited Compute
1. From “can it work” to “can it work at low cost”
Traditional code-generation and Q&A models are judged mainly by single-turn output quality. Long-horizon agents face a different problem: they must loop through observation, planning, execution, verification, and repair in a CLI, filesystem, repository, and tool environment until a task is actually accepted.
The bottleneck is not only whether the model “knows” an answer, but whether it can keep state, manage risk, call tools, and converge within a budget. Evaluation therefore has to include:
- Success: full acceptance, not surface completion
- Stability: variance across trials
- Cost: tokens, tool calls, and wall-clock time per task
- Tail behavior: loops, environment damage, edge cases
- Safety: unauthorized access, leaks, irreversible actions
A recently published training recipe is useful as an abstraction, not as a product pitch: start from a general foundation model at the hundreds-of-billions-of-parameters scale, then run full-parameter RL for long-horizon terminal and coding agents. Public numbers report 64 general accelerators, under five days of training, 46.2% Pass@1 averaged over five trials on a public terminal-style benchmark, in the same ballpark as a high-reasoning-intensity strong baseline, with per-task cost (under a unified accounting) below one-thirtieth of that baseline, plus gains on code-repair transfer. The method, in short: produce high-quality trajectories in verifiable environments, learn from outcome rewards, and use cost constraints to shape shorter, more stable policies with less wasted trial-and-error.
2. Method overview: base model, verifiable env, full-parameter RL
A reproducible pipeline usually has five parts:
- Base model: instruction following, structured output, and tool use, plus enough context and code understanding. Biggest is not always better.
- Tasks and environments: real workflows turned into resettable sandboxes with initial state, action constraints, acceptance scripts, and budgets.
- Trajectory sampling: the current policy tries each task multiple times; log tool calls, observations, errors, tests, and artifacts.
- Reward composition: primary credit from acceptance, plus format, process, efficiency, and safety terms.
- Policy update and eval: full-parameter RL, with periodic checks on held-out success, cost, and stability to limit overfitting and reward hacking.
base model↓ sampletask pool → sandbox execution → trajectories and acceptance↑ ↓policy update ← reward composition ← cost / safety / process constraints
The point is not “run RL more times.” It is to engineer tasks, environments, rewards, and eval. Sparse, high-variance rewards without a stable env, strict acceptance, and cost terms easily collapse into test-guessing or step-padding.
3. Data and environments: verifiability is the training signal
Agent data should be full task units: task, initial snapshot, constraints, acceptance script, budget—not just question–answer pairs. Cover:
- Terminal work: search, config, logs, processes, dependencies
- Code edits: bugs, APIs, tests, refactors, compatibility
- Combinations: locate, patch, test, explain
- Recovery: failed commands, timeouts, permissions, failing tests
- Long context: state across files, logs, and test rounds
Each task needs:
- Repeatability: resettable snapshots, controlled randomness
- Verifiability: acceptance independent of the prompt; hidden tests help
- Constraints: network, FS, resources, max steps; no irreversible ops
- Metering: tokens, tool calls, latency, failure reasons
Use a curriculum: short, clear-feedback tasks first, then multi-file, long-chain, weak-feedback work. Keep difficulty in a learnable band. Cluster failures (weak observation, invalid commands, misread tests, oversized patches, loops) instead of dropping them.
Prevent leakage: no hidden tests in prompts; split visible vs. final tests; scrub snapshots and logs; avoid train/eval code overlap for software-engineering tasks.
4. Rewards: outcome first, process second, cost to cut wasted actions
A practical structure:
- Main: full credit if all acceptance passes; small partial credit from visible tests or staged goals
- Validity: legal tool calls, parseable format, constraint compliance
- Efficiency: mild penalties on tokens, tool calls, steps, timeouts
- Safety: heavy penalty for privilege abuse, destructive ops, leaks
- Stability: prefer reproduce → edit → verify over blind edits
def compose_reward(traj, result, cfg):if result.safety_violation:return -1.0pass_term = 1.0 if result.all_acceptance_passed else 0.0partial = result.visible_test_pass_ratiovalid = result.valid_action_ratiocost_penalty = min(1.0, traj.cost / cfg.budget) * cfg.cost_weightstep_penalty = min(1.0, traj.steps / cfg.max_steps) * cfg.step_weightreturn (pass_term+ cfg.partial_weight * partial+ cfg.valid_weight * valid- cost_penalty- step_penalty)
Do not let cost penalties dominate early, or the policy stops early. Partial credit must come from trusted visible tests. Normalize at task level so budget differences do not unbalance gradients.
Reward hacking is a core engineering issue: keep acceptance scripts out of the model’s write path; rotate or generate tests; whitelist FS/network/commands; audit successful traces.
5. Training system: async sampling, near-on-policy updates, long-sequence stability
Full-parameter RL can change planning, tool use, and self-repair more than small adapters, but sampling is expensive, lengths vary, rewards are sparse, and updates can collapse. Completing training on a modest accelerator count means engineering efficiency matters more than raw scale.
Practical designs:
- Decouple sampling and training: workers generate trajectories; trainers consume them; versioning avoids stale off-policy data
- Group sampling per task: relative advantages within a group reduce variance; down-weight tasks with success rates near 0 or 1
- Variable length: packing, segmented attention, gradient accumulation; summarize huge observations but keep error details
- Stable updates: reference-model constraints, entropy, clipping, periodic eval, rollback on sudden drops
- Curriculum and filtering in parallel: short tasks first; filter invalid/unsafe traces without keeping only successes
Infra is ordinary: containers, object storage for snapshots/traces/checkpoints, logs, and alerts on success rate, steps, tokens, and crash rate. “64 accelerators, under five days” is not a hardware requirement; it shows that clear acceptance, efficient sampling, and stable updates can produce useful long-horizon RL on a mid-size cluster. Most teams should start with a small high-quality task pool and eval stack.
6. Inference cost: cheaper means less wasted search, not “think less”
Per-task cost is tokens in/out, tool calls, sandbox time, and retries. Endless reasoning, rereading irrelevant files, and repeating commands still explode cost.
Trained policies can:
- Build task state earlier (files and errors first)
- Verify in small steps after key edits
- Change strategy after repeated failure instead of looping
At deploy time: dynamic step budgets, staged timeouts, cache deterministic intermediates, parallelize low-risk reads, require confirmation for writes. Track both cost per task and cost per successful task. Public cost multiples depend on pricing, baseline reasoning intensity, and a fixed eval harness; remeasure on your own workload.
7. Evaluation: three questions
Did it really pass? Hidden tests and independent scripts. For patches: reproduce the bug, keep the diff small, check regressions. For terminal tasks: final state and logs.
Is cost repeatable? Log tokens, tool calls, max steps, time, retries, failure reasons. Report mean and quantiles (median, p95, max).
Does it transfer? Keep out-of-domain eval. Ablate cost terms, curriculum, and task diversity. Fix seeds, run multiple trials, report means and intervals. Official vs. in-house harnesses differ in env, tests, timeouts, and retries—compare tables carefully.
8. A reproducible roadmap
Stage 1: Task and eval base. Verifiable, rollback-safe tasks; sandbox, acceptance, logging, cost meters.
Stage 2: Cold start and trajectory hygiene. Optional SFT on a few high-quality successes if tool use is weak; classify failures.
Stage 3: Curriculum RL. Short, strong-feedback tasks first; raise success, then cost and safety weights; check a fixed eval set every round.
Stage 4: Deploy and close the loop. Budgets, isolation, human review, rollback; desensitize production failures and feed them back.
Checklist: isolate acceptance from prompts; keep secrets out of the sandbox; gate high-risk actions; make traces auditable; version model, env, and eval together; alert on success, cost, timeouts, and safety; red-team and reward-hack test before launch.
9. Risks and limits
This recipe fits tasks with clear acceptance, resettable envs, and rollback. Fuzzy requirements, irreversible actions, or heavy human coordination need approvals and tighter permissions.
Risks include eval contamination, reward hacking, sandbox drift, tail cost blowups, over-compressed steps that skip verification, and sensitive data in logs. Mitigations: hidden tests, dynamic acceptance, pinned env versions, cost-quantile monitoring, safety penalties, end-to-end audit.
10. Conclusion
High-performing long-horizon agents under limited compute come from a closed engineering loop: verifiable environments for trajectories, outcome rewards for capability, process and cost terms for efficiency, and strict eval against fake progress. When data, rewards, and systems are designed well, a mid-size cluster can train agents that are stable and cost-controllable on terminal and software-engineering work. Start from your own task mix, make success and cost measurable, then scale training while balancing success, stability, safety, and per-task cost.
