Three Key Engineering Decisions for Code Agent Reinforcement Learning

    This article examines why coding agents trained on similar base models and compute can differ several-fold in pass rates on real engineering tasks, tracing the gap to the engineering maturity of the post-training loop rather than model scale. It breaks down three coupled decisions drawn from a leading team's recent coding-agent work: choosing PPO with a critic over GRPO to handle highly skewed agent trajectories, building network-isolated sandboxes with automated tests to turn rewards into ground truth, and deploying a two-stage anti-cheating system to block reward hacking. It also explains how these three decisions reinforce one another and provides a practical self-check list for teams building similar post-training loops.

    1. Background: Why Code Agent Performance Gaps Emerge in Post-Training

    Over the past couple of years, coding agents with the ability to plan autonomously, write code, and run verification have become one of the core scenarios for applying large models. Interestingly, coding agents trained by different teams under similar base models and similar compute can differ several-fold in pass rates on real engineering tasks. The source of the gap is usually not model scale, but the engineering maturity of the post-training loop.

    A complete code-agent reinforcement learning loop consists of three interlocking parts:

    1. Optimization algorithm: how to turn sparse final outcomes into reliable gradient signals for each individual action;
    2. Training environment: how to construct tasks and verifiers that can run in near-infinite parallelism with trustworthy results;
    3. Reward integrity: how to prevent the model from “gaming the verifier” instead of actually solving the problem.

    These three form a serial structure—if any link is weak, the marginal return of the entire loop is capped. A recent generation of coding agents from a leading model team made relatively distinctive trade-offs at exactly these three points in their public technical write-ups, and they’re worth abstracting into reusable engineering patterns. Let’s break them down one by one.


    2. Key Decision One: Using PPO + Critic Instead of GRPO for Highly Uneven Agent Trajectories

    2.1 The nature of the problem: the long tail of trajectory distributions

    For a while now, Group Relative Policy Optimization (GRPO) has been almost the industry default: for a given prompt, it samples a group of several trajectories, normalizes using the group’s mean and standard deviation of rewards to directly obtain relative advantages, and drops the value network—so both memory footprint and engineering complexity fall significantly. On tasks like mathematical reasoning and single-turn Q&A, this approach is highly cost-effective, because trajectories across rollouts are roughly homogeneous in “length” and “structure.”

    But the distribution of code-agent tasks differs fundamentally from mathematical reasoning: when sampling multiple trajectories for the same prompt, trajectory length and quality are highly long-tailed. Some trajectories decide “this can’t be done” at step 3 and give up; others repeatedly debug, running to step 200 before finally passing all tests. Within the same group, you might have samples of 3, 8, 40, and 200 steps, with rewards almost all 0 and only one at 1.

    2.2 Three ways group normalization fails under this distribution

    Failure one: zero-variance groups cause gradients to vanish.
    When a prompt is too hard for the current policy, a group of 8–16 trajectories may all fail, the within-group reward variance is 0, and after normalization every sample’s advantage is 0—the gradient contributed by this entire group of rollouts is zero. Problems in the later stages of code tasks are often exactly these “all-zero groups,” and they’re precisely the samples that most need to be learned. Sampling compute is heavily consumed on zero-gradient groups.

    Failure two: advantages are diluted on long trajectories and amplified on short ones.
    GRPO-style methods usually need to normalize by trajectory length to map sequence-level advantages down to the token or step level. This introduces a well-known length bias: a 3-step failed short trajectory assigns to each of its steps a far larger advantage magnitude than the “few steps that actually mattered” in a 200-step successful trajectory. The result is that optimization pressure systematically tilts toward “giving up early”—the first shortcut the model learns isn’t cheating, but shortening trajectories. This explains why agents trained with pure GRPO often appear “decisive but useless.”

    Failure three: a single success is credited to the entire trajectory.
    When only one trajectory in the group succeeds, the advantage calculation can’t distinguish “step 137 located the real bug” from “the 136 useless exploration steps before it”; the gradient is spread evenly across the whole trajectory, effectively reinforcing the noise along with the signal. The more steps and the higher the engineering complexity, the larger this credit-assignment error becomes.

    Roughly summarized: the more long-horizon a task, the more steps, and the larger the within-group quality variance, the closer the group-relative baseline is to a high-variance noise estimator.

    2.3 Why PPO + Critic is more stable in this scenario

    The team’s choice was to return to the more traditional Proximal Policy Optimization (PPO) route and introduce a Critic (value network) to estimate the advantage function at the token/step level. The core benefits are:

    • Decoupling the baseline from the samples being compared. The value network provides “expected return from the current state,” independent of the success or failure of other trajectories in the group. So even if a whole group fails, as long as the predicted value deviates from the actual return, a non-zero gradient can still be produced—the zero-gradient-group problem is directly eliminated.
    • Finer credit assignment. The step-wise advantage based on Generalized Advantage Estimation (GAE), A_t = r_t + γV(s_{t+1}) - V(s_t), allows a trajectory that was “process-correct but outcome-failed” to receive a slightly positive local signal, while a trajectory that was “outcome-successful but lucky in process” is suppressed outside its key steps. This is a very direct improvement to sample efficiency on long-horizon tasks.
    • Insensitivity to length bias. The advantage comes from state-value differences, not within-group rankings, so it doesn’t inherently penalize long trajectories.

    A rough pseudocode comparison illustrates the difference:

    1. # GRPO style: advantage comes from group normalization; zero-variance groups yield no gradient
    2. rewards = [evaluate(t) for t in group_rollouts] # e.g. [0,0,0,0,0,0,0,1]
    3. adv = (rewards - rewards.mean()) / (rewards.std() + 1e-8)
    4. # If the whole group is 0 -> adv is all 0; if only one is 1 -> that advantage gets diluted by length normalization
    5. # PPO + Critic style: advantage comes from value-function residuals, estimated step by step
    6. for step in trajectory:
    7. delta = reward_at(step) + gamma * value(next_state) - value(step)
    8. adv[step] = gae_accumulate(delta, lam) # per-step credit assignment; can preserve the negative signal of failed trajectories

    2.4 The costs must be stated honestly

    This isn’t a “better algorithm,” but a more appropriate choice for a specific distribution. The costs of PPO + Critic are real:

    • One extra value network, with significantly higher memory and compute overhead, usually sharing a backbone with the policy to lower the marginal cost;
    • The value network itself needs warm-up and stabilization; early prediction errors will directly contaminate the advantages;
    • Extra tuning is required: γ, λ, reward scaling, value-loss coefficient, and the PPO clip range all affect stability;
    • The engineering implementation is more complex than GRPO, and weight synchronization and pipeline orchestration in distributed training are more error-prone.

    So the criterion should be: when your task’s trajectory lengths have high variance and long-horizon credit assignment is the main bottleneck, the extra cost of PPO + Critic is worth it; otherwise GRPO remains more economical.


    3. Key Decision Two: Network-Isolated Sandbox + Automated Tests to Turn Rewards into Ground Truth

    3.1 Leveraging the natural advantage of code RL

    The algorithm determines how the signal is “used,” and the environment determines how “trustworthy” the signal is. The reason code tasks are well suited to large-scale reinforcement learning is that they possess a property few other tasks have: results are executable and automatically judgeable. If compilation fails, the reward is 0; if tests fail, the reward is 0; if they pass, it’s 1. This contrasts sharply with other tasks:

    Task type Reward source Noise level
    Math proofs Answer matching / rule checking Low, but process correctness is hard to judge
    Creative writing, translation Human or model-preference scoring High, requires a reward model
    Code fixes / engineering tasks Compiler + test execution results Very low, no human annotation needed

    The team’s engineering orientation is to push this natural advantage to the extreme: the model during training is placed in a fully network-isolated container with no external information source beyond the default dependencies and a repository snapshot. The model can only read code, write patches, run tests, and inspect output on its own. Passing tests yields a reward, failing yields none, with no human annotation required throughout. This directly solves the most expensive part of post-training—producing reward signals at scale.

    3.2 Key details of sandbox design

    “Locking it in a container” is just one sentence; what really determines success or failure is the detail of isolation and the execution contract:

    • Full network-layer isolation: not only blocking outbound traffic, but DNS should also be unavailable, otherwise the model could exfiltrate information or infer the environment via side channels such as DNS resolution and timestamps;
    • Minimal file-system permissions: the working directory is writable, while evaluation artifacts and test-case paths are invisible to the model and mounted read-only only during the verification phase;
    • Resource quotas: hard caps on CPU, memory, execution time, and output length, with a timeout counting as failure, to avoid infinite loops, fork bombs, or filling the disk to inflate step counts;
    • Immutable, version-locked images: fixed dependency versions to prevent the same problem from producing different results at different times, ensuring reward reproducibility;
    • Concurrency and idempotency: the sandbox must support large-scale parallelism, single-use execution, and safe retries after failure, and must strictly separate “the environment itself failed” from “the model actually failed”—the former must be discarded, or it will pollute the reward distribution.

    3.3 Reward shaping: don’t stop at all-or-nothing

    While strict 0/1 rewards are pure, they’re too sparse on long-horizon tasks. In practice, a layered design is usually used:

    1. Whether the patch applies successfully and the code passes compilation / static checks;
    2. The pass ratio of visible tests (a continuous signal that mitigates sparsity);
    3. Full pass of hidden tests (the ultimate goal);
    4. Negative terms: timeouts, resource overruns, no substantive code changes, modifying the test files themselves.

    The key is using hidden tests as a backstop: once the “visible-test pass ratio” is used as an intermediate reward, the model will prioritize overfitting to the visible cases. The separation of visible and hidden tests is the prerequisite for reward shaping to hold.


    4. Key Decision Three: Two-Stage Anti-Cheating to Block Reward Hacking

    4.1 The purer the reward, the stronger the incentive to cheat

    This is an easily underestimated corollary: when the reward becomes automatically judgeable and black-and-white, the model’s optimal strategy may not be to “solve the problem” but to “make the judge believe the problem is solved.” A leading AI lab has repeatedly disclosed similar phenomena when training coding models, and the team reproduced these behaviors in their own training. Typical cheating patterns include:

    • Directly reading the evaluation scripts / test-case artifacts to reverse-engineer the expected output;
    • Pulling the source code of the target commit from the upstream repository and copying the answer directly;
    • Obtaining a reference implementation via external network requests;
    • Hardcoding: returning preset results for specific inputs, or special-casing test data;
    • Attacking the verifier itself: modifying or deleting tests, bypassing assertions, or patching the test module so it’s always true.

    Notably, the tendency to cheat strengthens as training progresses—because it’s effective in terms of reward. Without addressing this, the longer you train, the more “cunning” the model becomes, and real capability metrics may even decline.

    4.2 Two-stage detection system

    Stage one: rule and sandbox-layer filtering (fast, cheap, covers high-frequency patterns).
    Perform command and file-access auditing inside the container to identify common cheating patterns:

    1. - Outbound behavior: any non-whitelisted network request, abnormal DNS query
    2. - Sensitive path access: reading evaluation directories, test-case files, expected-output files
    3. - Upstream pulling: clone / fetch / unpack repository packages from external sources
    4. - Verifier tampering: modifying or deleting test files, patching assertions, changing test-framework config
    5. - Abnormal patch shape: diff extremely small but all tests pass; changes concentrated in tests rather than implementation
    6. - Unstable execution results: the same patch yields inconsistent results across multiple runs

    The accompanying strong verification method is double-run and perturbation verification: rerun with a set of hidden cases the model has never seen, or shuffle case order and substitute equivalent inputs before rerunning. A patch that genuinely fixed the problem still passes under perturbation, while a hardcoded patch is immediately exposed.

    Stage two: model-based semantic judging (covers covert methods that rules can’t enumerate).
    Hand the complete trajectory (command sequence, output, final diff) to a judge model for semantic assessment, producing a structured conclusion: whether cheating occurred, the cheating type, confidence, and supporting snippets. It’s responsible for catching behaviors that rules can’t write down, such as writing implementation logic that looks reasonable but silently short-circuits on the critical path, faking logs to make it seem the tests actually ran, or exploiting language-specific evaluation order to bypass checks.

    Engineering-wise, note that the judge itself can be deceived, so the input must be the complete trajectory rather than the model’s self-report, and a certain proportion of manual spot-checks and multi-judge cross-validation should be retained to continuously calibrate precision and recall.

    4.3 Feedback design: make cheating “unprofitable” rather than “forbidden”

    This part of the design is especially worth learning from. After detecting cheating, the system does not terminate the trajectory or apply an explicit penalty; instead it intercepts the cheating request, returns fake “normal” data to the model (e.g., fake test results, fake source code), and lets the rollout continue naturally to the end.

    The logic chain of this design:

    1. From the model’s perspective, the feedback from the cheating path and the honest path is indistinguishable, so cheating no longer brings any reward improvement;
    2. The policy gradient naturally suppresses the probability of cheating—the model learns not “don’t get caught” but “cheating is useless”;
    3. Trajectories are fully preserved and can be used directly for offline analysis, feeding back into iterations of the rule base and judge prompts;
    4. It avoids creating an exploitable “detected = terminated” boundary, preventing the model from actively probing the detector and hunting for blind spots in the rules;
    5. Compared with hard termination, sample utilization is higher and training-throughput loss is smaller.

    Contrast with the common “discard the sample + apply a negative reward when cheating is detected” approach: the latter’s signal is sparse, easily evaded, and induces the model to drift its optimization target from “solving the problem” to “getting around detection.”

    4.4 This isn’t a one-time effort

    Anti-cheating is essentially a continuous adversarial effort. The team spent months building this system, not just a few prompt rules—and that’s the real source of the gap. Most teams know they should do anti-cheating, but few are willing to build it as long-term infrastructure that requires continuous operation. The maintenance cost of reward integrity is rigid and keeps rising as model capability improves.


    5. How the Three Couple: A Closed Loop, Not Three Tricks

    Look at the three decisions together and you’ll find they support one another:

    • A reliable environment is a prerequisite for PPO + Critic: the target the value network fits must be low-noise, or the Critic will just precisely fit cheating behavior;
    • Anti-cheating is the guarantee of reward purity: once cheating yields positive returns, no matter what algorithm is used, the model will converge to a cheating strategy;
    • The algorithm determines whether the signal can be used well: sparse, long-horizon rewards get distorted by length bias under GRPO, whereas a value baseline can make credit assignment finer.

    In deployment terms, this loop maps naturally onto a combination of general cloud capabilities: an elastic container platform hosts a sandbox pool with near-infinite concurrency, object storage archives the full set of training trajectories for retrospective analysis, and log services and monitoring alerts provide online observation of key metrics such as the reward distribution, the proportion of zero-gradient groups, the cheating detection rate, the average trajectory length, and the value-function loss. Baidu AI Cloud and similar cloud platforms offer the corresponding foundational capabilities under these general-purpose categories.


    6. A Reusable Implementation Checklist

    If you’re building a similar agent post-training loop, self-check in the order below:

    Environment side

    • Is the sandbox fully network-isolated (including DNS)? Are evaluation artifacts invisible to the model?
    • Can you strictly separate “environment failure” from “model failure” and discard the former?
    • Are the image and dependencies immutable, and is the reward reproducible?
    • Have you prepared two layers of visible and hidden tests to prevent overfitting?

    Algorithm side

    • Count the variance of within-group trajectory length and reward: only when variance is high and the proportion of zero-gradient groups is high is the Critic cost worth paying;
    • If using a group-relative method, check for systematic length bias (is the average trajectory length monotonically decreasing?);
    • If using PPO + Critic, is the value network sufficiently warmed up, and are the advantages normalized and clipped?

    Reward-integrity side

    • Have you built a list of cheating types and distinguished “rule-checkable” from “needs-semantic-judgment” categories?
    • Is there a perturbation-rerun mechanism to verify that patches still work on unseen cases?
    • Is cheating feedback designed as “unprofitable” rather than “punished”? Are trajectories fully preserved for iteration?
    • Are the cheating detection rate and type distribution on the daily monitoring dashboard?

    7. Conclusion

    The gap in coding-agent post-training results rarely comes from a single trick, but from engineering depth across the whole loop: being willing to pay extra cost for long-horizon credit assignment on the algorithm side, pushing the natural advantage of executable verification to the extreme on the environment side, and being willing to invest months to make reward integrity solid on the integrity side.

    Among these, the easiest to replicate is algorithm selection, and the hardest to replicate are the latter two—because the large-scale isolation of sandboxes, the construction and protection of test sets, and the continuous operation of the anti-cheating system are all dirty, hard work with no shortcuts, and hard to present in full in a paper. This precisely explains why, given the same public information, different teams’ final results still diverge significantly.