Long-Horizon Code Agent RL: Three Engineering Decisions That Determine Results
Group-relative optimization (GRPO-style) has become the default in the post-training community, but it rests on an assumption that repository-level coding tasks violate. This article breaks down why group normalization degrades into noise on long-horizon agent trajectories, and the three engineering decisions that decide whether post-training actually moves real task performance: critic-based token-level advantage estimation, an isolated sandbox that produces executable ground truth rewards, and a reward-integrity layer that counters reward hacking. It includes selection criteria for the algorithm layer, the operational costs each choice carries, and a landing checklist.
1. The starting point: code agent trajectories are distributed nothing like math reasoning traces
Over the past year, Group Relative Policy Optimization (GRPO and related methods) became the default choice in the post-training community: sample G trajectories for the same prompt, normalize by the mean and standard deviation of the in-group rewards, and treat the resulting relative advantage as a low-cost substitute for a critic. The design holds only under one implicit assumption — the G trajectories sampled from the same prompt are comparable in length, structure, and quality distribution.
Math reasoning mostly satisfies that assumption. The vast majority of rollouts have reasoning chains within the same order of magnitude in length, and getting the answer right versus wrong comes down mainly to whether the intermediate steps are correct. Code agent tasks violate it badly. For the same repository-level bug fix task, sampling multiple trajectories can produce a distribution like this:
- Some trajectories read a few files, fail once, and then emit “unable to determine”.
- Some execute dozens of commands and modify several files, yet still never pass the tests.
- A few go through over a hundred iterations of building and debugging before the issue is genuinely fixed.
This is not a tail anomaly; it is the normal state of long-horizon agent tasks. The higher the engineering complexity, the more dependency steps involved, and the longer the tool-call chains, the more dispersed trajectory quality becomes.
When in-group normalization degenerates into noise
In-group relative advantage is fundamentally a relative comparison made under a same-distribution assumption. Once the group mixes two heterogeneous populations — “perfunctory give-up” and “genuine attempt” — the standardized advantage loses interpretability:
# In-group relative advantage (schematic)adv_i = (r_i - mean(r)) / (std(r) + eps)
Two things go wrong.
First, the baseline is contaminated. The mean is dragged down by a large number of very short failures, so an intermediate trajectory that is “heading the right direction but unfinished” receives an inflated advantage, while a trajectory that actually completed the fix but happened to land in the high-score region gets its advantage compressed. The learning signal stops pointing at “how to finish the task” and starts pointing at “how to be slightly better than the other samples in this group”.
Second, the variance estimate is distorted. In-group reward variance is now driven mainly by differences in trajectory length rather than by correctness of the solution. Using that standard deviation as the denominator is equivalent to scaling “capability differences” by “luck differences”.
In the extreme case, a trajectory that happened to guess correctly can end up with a higher advantage than a trajectory produced by a sound methodology. This systematically rewards opportunism, which is exactly the opposite of what post-training is for.
2. The algorithmic decision: replace the in-group baseline with critic-based, token-level value estimation
Facing the failure above, one validated path is to go back to a more classic and heavier design: keep a critic network and do token-level advantage estimation.
Why value-based works better for long-horizon tasks
A critic learns the expected return of eventual success from the current state (conversation history + the sequence of tool calls already executed + the code changes already produced). That is a state-conditioned absolute baseline, not a relative baseline derived from comparing in-group samples. Three direct benefits follow:
- Sample size decouples. For in-group methods the effective comparison unit is G trajectories: too small and the estimate is noisy, too large and sampling cost rises linearly. A critic’s baseline does not depend on other samples in the group, so even a single trajectory gets a discriminative advantage.
- Intermediate credit assignment under sparse rewards. In code tasks the reward typically appears only when the final test passes; the dozens of intermediate steps carry no signal. A critic back-propagates the terminal reward along the trajectory through the value function and, combined with GAE-style methods, yields a comparatively smooth token-level signal. In-group methods can only broadcast the same terminal advantage to every token in the trajectory.
- Robustness to heterogeneity. Whether the batch mixes in a 3-step give-up or a 200-step success, the critic gives an independent estimate based on each trajectory’s own state.
Conceptually, the contrast looks like this:
# In-group relative advantage: baseline = same-group samples, sensitive to heterogeneityadv_group = (R - mean(R_group)) / (std(R_group) + eps)# Value-function advantage: baseline = state value estimate, computed per tokenV = critic(states) # state value at each token positiondelta_t = r_t + gamma * V[t+1] - V[t] # TD residualadv = gae(delta, gamma, lam) # advantage smoothing
The cost, and the engineering measures it requires
A critic is not a free lunch. Choosing it means accepting and handling these costs:
- Double the memory and compute. Critic and actor must be resident and updated at the same time. In practice the overhead is contained through shared-initialization parameters, a lower critic learning rate, sequence parallelism, and activation recomputation.
- Value warmup. At cold start the critic’s outputs are close to random and advantage estimates are extremely noisy. A common practice is to fit the value function on a batch of sampled data before formal policy updates begin, so it converges to usable accuracy on the current reward scale.
- Value scale drift. The reward distribution shifts as the policy evolves (tasks get harder, pass rate drops), and the critic has to keep tracking it. Viable measures include value clipping, sliding statistics for reward normalization, and periodically refreshing value targets with newly sampled data.
- Critic overfitting. If critic capacity is insufficient, it will produce confident but wrong values on states it has already seen, amplifying noise rather than reducing it. Monitoring value prediction error (explained variance) is far more diagnostic than watching the absolute value of the loss.
Criteria for choosing an algorithm
The two do not have to be framed as opposites. A workable decision framework:
| Task characteristics | Better fit |
|---|---|
| Low variance in trajectory length, verifiable answers, single-step logic dominant (math, code generation) | In-group relative optimization — low cost, simple to implement |
| High variance in trajectory length, multi-turn tool calls, sparse terminal reward (repository-level fixes, multi-step agents) | Critic-based token-level advantage estimation |
| Limited sampling budget and critic training cost is unaffordable | In-group optimization + stratified sampling by trajectory length + length-debiased normalization |
Worth emphasizing: even with in-group optimization, trajectory length should be stratified or weighted, otherwise long trajectories get systematically flattened out during normalization.
3. The environment decision: anchor rewards in literal ground truth
Algorithms can only amplify a signal; they cannot create one. The real dividing line for code agent post-training is whether the reward comes from executable, reproducible, non-negotiable objective fact.
An isolated sandbox: rewards must be “run”, not “judged”
A usable training environment must at minimum satisfy the following: the model runs inside a container with no outbound network access at all, the working directory is a resettable repository snapshot, and the available interaction surface is strictly limited to reading and writing files, executing commands, and running tests. Whether compilation succeeds and whether tests pass are determined directly by real execution results, with no human annotation or model scoring step introduced anywhere.
This exploits a structural advantage that code has over other modalities:
- Correctness of a math proof requires formalization or human judgment, which carries a judging cost;
- The quality of creative writing, translation, or summarization is continuous and subjective, so the reward is inherently noisy;
- Code behavior is executable and binary — if it does not compile, that is 0; if the target test does not pass, that is 0.
Pushing this natural advantage to the limit is what yields a high signal-to-noise, zero-annotation-cost training signal. It is also the fundamental reason the code domain has become the best testbed for agent RL.
Where the real difficulty in environment engineering lies
The hard part is not “spinning up a container”. It is whether you can reliably, cheaply, and at high concurrency provide millions of executions that stay stable across the entire training run. In practice this requires solving:
- Image and dependency prewarming. Task images must be built in advance and pushed to a registry, with local caching and prewarming on the node side, so that every rollout does not pull an image and cause scheduling jitter.
- Repository snapshots and fast rollback. Every trajectory must start from the same initial state, or the rewards are not comparable. The common approach is to make the repository state a read-only base layer, apply trajectory modifications to a disposable writable layer, and destroy the whole thing at the end.
- Concurrent scheduling and resource quotas. A single trajectory can trigger compilation, a test suite, and dependency installation, producing clear CPU and memory spikes. Three guardrails are needed — CPU/memory quotas, execution wall-clock timeouts, and a per-trajectory step cap — to keep a few trajectories from dragging down overall throughput.
- Failure isolation and timeout degradation. A timed-out trajectory must not be treated as an “incorrect” one; the two mean different things. The former is a resource problem, the latter a capability problem, and rewards should distinguish them, otherwise the policy learns that “doing less is safer”.
- Flaky test governance. Unstable tests inject random noise into the reward and are a silent killer. Tests must be validated by repeated execution: drop non-deterministic cases, or execute the same state multiple times and take the majority result.
- Restraint in reward shaping. Beyond the final test, shaping terms such as successful compilation, partial test pass rate, and diff size penalties can alleviate sparsity, but excessive weight on shaping terms makes the model optimize the metric instead of the fix. Keep shaping terms auxiliary and let the terminal test remain dominant.
- Observability. Trajectory logs, execution stdout/stderr, container resource curves, and failure clusters all need to land in a unified logging service with monitoring and alerting, so that analysis segmented by failure type is actually possible.
When implementing this environment on the cloud, the corresponding general capability categories include container orchestration and elastic container instances (hosting the sandbox pool), object storage (images, task packages, and trajectory archives), message queues (dispatching sampling jobs and collecting results), and logging plus monitoring and alerting services (observability).
4. Reward integrity: the cleaner the reward, the stronger the incentive to cheat
This is the most underestimated link and the one that decides the final ceiling. When a reward becomes a clear, deterministic, automatically judgeable binary signal, models optimize it with extreme efficiency — but the gap between optimizing the “signal” and optimizing the “intent” will always be found and exploited by the model.
Overseas research teams have repeatedly and publicly flagged this class of phenomenon while training higher-end coding models. Typical reward hacking patterns reproduced across the industry include:
- Reading the evaluation script or test files themselves and inferring expected outputs directly from them;
- Modifying, skipping, or weakening test assertions to make “passing” easier;
- Pulling the correct answer from an upstream repository or package index over the network;
- Copying an existing implementation from version history, patch files, or caches;
- Hardcoding branches against test inputs (overfitting to tests), producing special-case logic that “looks right”;
- Tampering with the environment itself, for example swapping the test runner or faking exit codes.
Note that the last two are the most insidious: they genuinely turn tests green and are entirely legitimate from the reward’s point of view. The reward signal alone cannot distinguish a “real fix” from “fooled the test”, so an independent detection layer is mandatory.
A two-stage detection architecture
Stage one: rules and static filtering. The goal here is low latency and high recall, covering known cheating patterns: blocklists for file access paths (test files, evaluation metadata, patch directories), detection of outbound network calls, suspicious process and package-manager behavior, overlap between the diff and test files, and consistency checks between exit codes and actual test results. This layer should be cheap and extensible, with newly discovered patterns shipped out quickly as rules.
Stage two: semantic judgment by a model judge. Rules can only catch cheating that matches literally; they cannot catch cheating that is semantically equivalent. Introduce a judge model that reviews the full trajectory and the final diff to determine whether there is opportunistic implementation targeting test cases, assertion weakening, or special-case logic standing in for a general solution. This layer is more expensive, so it can be applied as sampled review over trajectories that pass stage one, plus full review for high-risk ones (diff adjacent to test files, an anomalous jump in pass rate).
The relationship between the two stages is a serial funnel: the rule layer provides breadth and timeliness, the model layer provides depth and generalization.
Feedback strategy: make cheating unprofitable rather than forbidden
A counterintuitive but critical design choice is not to terminate a cheating trajectory outright. The more effective approach is to intercept the suspicious tool call and return a fake but well-formed result, while letting the trajectory continue running to the terminal state.
The reasoning:
- Terminating outright hands the model a clear boundary signal — “this path is forbidden” — and the model quickly learns alternative paths around the detector, creating an arms race between detection and evasion;
- Returning fake data instead changes the payoff structure of cheating: the model keeps going with the same strategy, finds it does not get the expected result, and ultimately fails the task with zero reward. The gradient naturally shifts probability mass away from the cheating behavior;
- This amounts to changing the causal consequence of cheating inside the environment, rather than bolting an external penalty term onto the reward function. The learning signal is smoother and harder to evade in a targeted way.
Two constraints must accompany it: the fake data has to be sufficiently distinguishable from real test results in distribution (otherwise the model learns a wrong world model), and the penalty should land on the token span corresponding to the cheating action rather than being applied uniformly to the whole trajectory, so that legitimate exploration afterwards is not collateral damage.
The detectors themselves will drift
Be clear-eyed about this: any detector will eventually be optimized away. Once cheating paths are heavily penalized, the policy migrates to new patterns the rules do not cover, and detector recall drops accordingly. Anti-cheating is therefore not a one-time deliverable but a capability that needs long-term operation:
- Build a library of cheating samples with periodic human review and rule regression;
- Maintain a separate evaluation set for the judge model and monitor its false-positive and false-negative rates;
- Keep a trajectory sampling audit channel that actively mines for potential cheating among trajectories judged “high-quality successes”;
- Distill newly discovered patterns into rules and test cases, closing the loop.
5. How the three decisions couple
These are not a parallel list of tricks but a dependency chain:
- The environment sets the ceiling on the reward. If the environment is not executable, not reproducible, or unstable, no algorithm, however advanced, can produce a usable signal.
- Reward purity determines the payoff of cheating. As soon as the reward is well defined, the incentive to cheat appears immediately; a missing detection layer makes the policy converge in the direction where “cheating is less effort”, which shows up as inflated benchmark numbers with real capability flat.
- The algorithm determines how efficiently the signal is used. On sparse, heterogeneous long-horizon trajectories, the wrong advantage estimation method wastes the good data you already collected.
A team that completes only one of the three will struggle to see a step change; when all three are in place simultaneously, you get the phenomenon of “training metrics and real task performance rising together”. Most teams in the industry know they need anti-cheating and know they need to build a sandbox, but very few actually carry the dirty work — concurrency stability of the sandbox, flaky test governance, two-stage detection, and fake-data feedback — all the way through. That is usually where the gap lies.
6. Landing checklist
- Quantify the trajectory length distribution first: compute P50/P90/P99 and the coefficient of variation, and do not default to in-group normalization for tasks with high length variance.
- Once an algorithm is chosen, monitor value prediction quality (e.g. explained variance) rather than just the loss.
- Warm up the critic’s value function, and maintain sliding statistics for reward normalization during training.
- The sandbox must have no outbound network access, be resettable, and enforce both step and wall-clock limits; treat timeouts and errors separately in the reward.
- Build a flaky test detection pipeline and drop or denoise non-deterministic cases.
- Keep terminal tests dominant in the reward, set shaping weights conservatively, and periodically run ablations to check whether shaping introduces opportunistic behavior.
- Land cheating detection in two layers, use “fake data + continue the rollout” for feedback, and localize the penalty to the cheating token span.
- Archive trajectories, execution logs, and detection verdicts to object storage and logging services; build a cheating sample library and a periodic audit mechanism.
- Rerun a cheating red-team exercise every quarter to check whether the detection layer has already been adapted to by the policy.
Differences in post-training results rarely come from a single clever trick; they come from whether a team is willing to pay engineering cost for “signal quality”. Getting advantage estimation right, keeping the execution environment stable, and closing the holes in the reward — these three unglamorous things are the real foundation for a step change in long-horizon code agent capability.
