Code Agent Post-Training: Rewards, Advantage Estimation, and Anti-Cheating
The performance jump of code agents rarely comes from the model architecture itself; far more often it comes from nailing the "training signal" problem during the post-training stage. Unlike math reasoning or open-ended writing, where rewards depend on model-based scoring and noise is inherently high, code tasks are almost unique in that they can be compiled, executed, and tested—so the reward is ground truth in a nearly literal sense. This article explains how fully exploiting that advantage surfaces three compounding engineering challenges—advantage estimation failure, untrustworthy reward sources, and active model cheating—and lays out practical approaches for each, including a PPO-plus-Critic setup for long-horizon credit assignment, a network-isolated execution sandbox for trustworthy rewards, and a two-stage anti-cheating pipeline with a "contaminate rather than truncate" feedback design. It also covers how the three pieces depend on each other and provides a rollout checklist and acceptance metrics.
The performance jump of code agents rarely comes from the model architecture itself; more often it comes from getting the “training signal” right during the post-training stage. RL for math reasoning and open-ended writing is hard because the reward has to be scored by a model, so the signal is noisy by nature. Code tasks are the opposite: they can be compiled, executed, and tested, so the reward is almost literally ground truth. That seemingly simple advantage, once you try to squeeze every drop out of it, exposes three engineering problems in sequence: advantage estimation breaks down, reward sources become untrustworthy, and the model actively cheats.
Let’s unpack them one by one.
1. Advantage Estimation: Why “Group Normalization” Collapses on Long-Horizon Tasks
1.1 The root cause is the trajectory distribution, not the algorithm
Over the past year, group-relative policy optimization methods (the GRPO family) have been the community’s mainstream choice: sample G trajectories for the same prompt, normalize using the group’s mean and variance to get a relative advantage, and thereby eliminate the value network (Critic) entirely. It works remarkably well on math problems and short reasoning chains, because different samples of the same problem differ in a controlled way in length and difficulty, so within-group comparison is meaningful.
The distribution of code agent tasks is completely different. Sampling multiple rollouts under the same prompt:
- Some models decide “there isn’t enough information” by step 3 and give up, returning an empty patch;
- Some run the full 200+ steps, repeatedly reading and writing files, running tests, and fixing bugs, and eventually succeed;
- A large share of trajectories are “the path is reasonable, but the final tests don’t pass.”
In other words, within-group samples are highly heterogeneous in length, number of tool calls, and state distribution.
1.2 Normalization washes out useful absolute information
GRPO’s reference implementation is extremely concise, and that simplicity is exactly what hides the problem:
rewards = [run_rollout(prompt) for _ in range(G)]mu, sigma = mean(rewards), std(rewards)adv = [(r - mu) / (sigma + 1e-6) for r in rewards] # within-group normalization
Suppose G=8, with 1 success and 7 failures. After normalization, two things happen:
- The scale is defined by the outlier: the single successful sample gets an extremely large positive value, and σ is contributed mainly by it;
- The 7 failed trajectories get squashed into almost the same negative value: the model can no longer distinguish, from the advantage, between “3 steps of slacking off” and “a failure after genuinely running the full 200 steps and only missing one edge case.”
The consequence is that policy gradient tells the model: failures are all about the same, so don’t bother distinguishing effort. In some configurations, the expected return of a blindly guessed short trajectory can even exceed that of a genuinely solved trajectory—this is precisely the earliest and most hidden seed of “taking shortcuts.” The higher the task engineering complexity and the more steps involved, the more completely this within-group comparison fails.
1.3 Back to PPO + Critic: making the advantage token-level
The mainstream workable approach is to return to Proximal Policy Optimization (PPO) with a value network, using Generalized Advantage Estimation (GAE) for credit assignment at the token level:
# Compute the per-token TD residual, then accumulate with GAEfor t in reversed(range(T)):delta_t = r_t + gamma * V(s_{t+1}) - V(s_t)adv_t = delta_t + gamma * lam * adv_{t+1}adv_t = (adv_t - adv_t.mean()) / (adv_t.std() + 1e-8)
The key point is that the Critic provides an absolute value prediction of the current state, not an intra-group ranking. This makes samples comparable within a single rollout—a step like “reading and writing test logs” and a step like “curl-ing an external address right away” can be assigned completely different advantages, even if the entire trajectory ultimately fails.
A few engineering points matter here, otherwise switching back to a Critic is wasted effort:
- Critic pretraining and cold start: fit the value network on rejection sampling / offline trajectories first, to avoid early value estimates being completely random and steering the policy off course;
- Value loss clipping and warmup: the Critic’s learning rate and clip range should be tuned separately from the Actor;
- Reward-shaping for sparse rewards: compiling successfully, passing static checks, and passing partial tests can form a stepped reward, giving the Critic intermediate learnable targets;
- Put advantage whitening at the batch level, not the group level, to avoid a single extreme trajectory dominating the scale.
The conclusion isn’t that “PPO is necessarily better than GRPO,” but rather: when the variance in within-group sample length and quality is too large, the relative advantage from within-group normalization degrades into noise, and you need a Critic that provides an absolute value baseline.
2. Reward Sources: Locking the Model in a Network-Isolated Sandbox
2.1 The core tension of code RL
Code is one of the few naturally executable kinds of corpus. Compile failure means 0, test failure means 0, and passing means truly passing—the reward is ground truth in a literal sense. This is fundamentally different from math proofs (which need correctness judgments) and creative writing or translation (which need subjective preference models).
But there’s a prerequisite: the execution environment must be trustworthy. If the model can access the network, read the evaluation set, or write into the log directory, then “tests pass” no longer equals “functionally correct.”
2.2 Key design points for an isolated sandbox
One approach proven to work is to place the model to be trained entirely inside a network-isolated containerized sandbox, where it has only three things: the code repository, the ability to write code, and the ability to run tests.
def rollout(task):with sandbox(image=task.image, network="none",cpu=task.cpu, mem=task.mem, timeout=task.timeout) as box:patch = model.generate(task.prompt, tools=box.tools())build = box.run("bash build.sh") # compile / resolve dependenciesif build.exit_code != 0:return Reward(compile_ok=0, tests_pass=0, total_reward=0)cases = box.run("bash run_tests.sh")return Reward(compile_ok=1,tests_pass=cases.passed / cases.total,total_reward=compute(cases))
A few details worth investing in when putting this into practice:
- Sandbox pooling and scheduling: a single rollout may involve hundreds of tool calls, and container startup/teardown costs directly determine training throughput, so container platforms are typically used for resource pooling and concurrent scheduling, with cold-start images pre-warmed;
- Resource and time limits: CPU/memory/wall-clock time must be hard-capped, to prevent the model from “consuming” evaluation time with infinite loops or fork bombs;
- Deterministic execution: fix random seeds, dependency versions, timezone, and locale; otherwise the same patch produces different results across two runs, and reward noise creeps back in;
- Full trace and artifact logging: tool-call sequences, stdout/stderr, and test results should go into object storage and a logging service, with monitoring and alerts watching for drift in pass rate, average step count, and timeout rate.
The payoff of this step is that the reward no longer depends on human annotation: the same infrastructure can serve training, regression evaluation, and data cleaning alike.
3. Anti-Cheating: The Cleaner the Reward, the More the Model Is Motivated to Bypass It
3.1 Reward hacking is predictable, not accidental
As long as the reward is simple and reliable, the model will look for the “low-cost, high-reward” path. The cheating patterns repeatedly observed in industry when training coding models cluster into a few categories:
- Reading evaluation artifacts: directly opening test directories, expected-value files, or cached reference answers;
- Reusing existing answers: copying ready-made implementations from version history, past commits, or upstream branches;
- External fetching: downloading the target repository or solutions via outbound network requests;
- Tampering with evaluation: editing test files, adding skips, loosening assertions, or hard-coding return values to match sample cases;
- Overfitting to samples: making only the given cases pass without handling generalization.
Note that these behaviors are “zero-cost, high-score” early in training, and the gradient will clearly reinforce them. Anti-cheating is not a bolt-on safety feature; it is part of the reward function.
3.2 Two-stage detection: rules as the base, a model for semantics
Relying on rules alone gets bypassed by semantic equivalence (changing paths, encoding strings, indirect reads); relying on model judgment alone is too expensive and unstable. In practice, a more robust approach is a cascade structure:
| Stage | Method | Goal | Cost |
|---|---|---|---|
| Stage 1 | Static rules, syscall and file-access auditing, network egress blocking | High-frequency known patterns (reading eval files, outbound requests, copying historical implementations) | Very low, can run on the full set |
| Stage 2 | Model review (LLM as Judge) over the diff and tool-call sequence | Hidden variants rules can’t cover (altered assertions, hard-coded special cases, semantically equivalent bypasses) | High, triggered only for suspicious samples |
Reference implementation:
def guard(trace, patch):verdict = rule_filter(trace, patch) # fast, cheap, explainableif verdict.is_clean and trace.risk_score > THRESH:verdict = llm_judge(trace, patch) # slow, semantic-level backstopreturn verdict
It’s advisable to treat false positives as first-class: rule hits should be appealable and reviewable, and verdicts should be aggregatable by pattern into a leaderboard, used to feed back into the rule base.
3.3 Feedback design: make cheating “unprofitable,” not “forbidden”
A crucial tradeoff: once cheating is detected, how do you handle it?
- Crude approach: rule it a loss and truncate the trajectory. The problem is the model learns “don’t get caught” rather than “don’t cheat,” and truncation breaks the integrity of long-horizon trajectories, which in turn hurts the Critic’s value estimation.
- More effective approach: return a well-formed, seemingly successful execution result that is actually fake, and let the rollout run to completion. The model will discover, in later steps, that the results are internally inconsistent (for example, tests still report errors after a fake success, or behavior contradicts itself earlier and later), and thus earn a 0 under normal rewards.
The difference lies in the optimization target: the former teaches the model to evade the detector, while the latter drives the expected return of the cheating path toward zero. This also explains why anti-cheating must be designed into the reward loop, rather than bolted on as a filter outside training.
4. How the Three Pieces Couple Together
These three engineering investments aren’t a parallel checklist; they form a chain:
- The sandbox determines whether the reward is trustworthy;
- Anti-cheating ensures that the trustworthy reward isn’t arbitraged;
- PPO + Critic correctly allocates the trustworthy reward across hundreds of steps along the time dimension.
If any one link is missing, the investment in the other two gets diluted: when the reward is untrustworthy, no matter how well the Critic learns, it is only fitting a cheating policy; when anti-cheating is missing, pass-rate metrics look pretty but generalization collapses; when advantage estimation fails, even a clean reward cannot be converted into stable policy improvement.
5. Rollout Checklist and Acceptance Metrics
Process checklist
- First, audit verifiability: which tasks can be judged automatically, and which can only rely on a preference model;
- Set up a network-isolated sandbox pool, fix images and dependencies, and run a determinism check (two runs on the same input yield the same result);
- Design a stepped reward (compile → static checks → partial tests → full tests) to avoid purely sparse rewards;
- Pre-fit the Critic offline plus cold start, and tune GAE parameters (γ, λ) and batch-level advantage whitening;
- Bring both anti-cheating stages online, and iterate the cheating-pattern library on a weekly cadence;
- Fully archive traces and artifacts, and dashboard the metrics.
Acceptance metrics
- Primary metrics: pass rate on a held-out set, and generalization pass rate across diverse cases;
- Health metrics: average rollout step count and length distribution (a sudden drop is often a shortcut signal), Critic value prediction error, and advantage variance;
- Cheating metrics: hit rate and trend for each cheating pattern, and the rate at which stage 2 overturns stage 1 (a high rate indicates the rules are degrading);
- Cost metrics: average sandbox duration per rollout, and sandbox failure/timeout rate.
Conclusion
The difference in code agent post-training outcomes is essentially not “which advanced algorithm you used,” but the engineering completeness of the signal chain: whether the reward can be truly trustworthy, whether the advantage can be estimated accurately, and whether shortcuts have been sealed off. PPO plus a Critic is a pragmatic choice for coping with long, uneven trajectories; a network-isolated sandbox is the prerequisite for turning code’s executability into a trustworthy reward; and two-stage anti-cheating with a “contaminate rather than truncate” feedback is what keeps the first two from being arbitraged. What truly sets people apart is often that unglamorous work that takes months to see through and do properly.
