Three Engineering Breakthroughs in Code Agent Post-Training: Algorithm Selection, Reward Design, and Anti-Cheating

    This article explains why code agents need RL post-training and how to build a dependable post-training pipeline. It is organized around three engineering layers: algorithm choice (GRPO, PPO, and rejection sampling), clean reward construction through executable environments, and anti-cheating mechanisms. For each layer it offers practical selection criteria, implementation patterns, checklists, and tuning guidance, showing that the real gap between teams comes not from any single algorithm but from how thoroughly the entire engineering stack is executed.

    Why Code Agents Need RL Post-Training, and How to Choose Algorithms and Rewards

    Why do code agents need RL post-training? How should you design rewards and prevent cheating? When should you choose GRPO over PPO? The answers to these questions often lie not in a single model architecture, but in the details of the post-training pipeline.

    Pretraining gives a model language understanding and basic programming ability, but a code agent must actually accomplish long-horizon tasks: reading code, locating problems, editing files, running tests, analyzing errors, and iterating again. Behaviors like these are hard to obtain reliably through supervised learning alone, because they depend on multi-step decision-making and real feedback. Reinforcement learning post-training with verifiable rewards has therefore become a key step in improving code agent capabilities.

    From an engineering perspective, high-quality post-training must answer at least three questions:

    • Algorithm layer: Is the advantage estimation accurate?
    • Environment layer: Is the reward signal clean?
    • Mechanism layer: Will the model take shortcuts?

    The sections below present a practical, production-ready approach at each of these three levels.

    Breakthrough 1: How to Choose the Algorithm? The Applicability Boundaries of GRPO, PPO, and Rejection Sampling

    Start with the conclusion: no single algorithm naturally fits every code agent task. Whether to use GRPO, PPO, or rejection sampling depends on task variance, trajectory length, reward density, value network quality, and training cost.

    1. GRPO’s Strengths and Boundaries

    Over the past couple of years, GRPO has been widely adopted for tasks such as mathematical reasoning because it is simple to implement and requires no separate critic. Its core idea is to sample multiple trajectories for the same prompt and normalize rewards using the group mean and standard deviation to obtain a relative advantage.

    1. # GRPO: within-group relative advantage estimation
    2. rewards = [rollout_reward(r) for r in sampled_rollouts] # G trajectories
    3. mean_r = statistics.mean(rewards)
    4. std_r = statistics.pstdev(rewards) + 1e-8
    5. advantages = [(r - mean_r) / std_r for r in rewards]

    GRPO tends to fit scenarios that usually have these characteristics:

    • Quality differences across trajectories for the same prompt are controllable;
    • Trajectory lengths are relatively stable;
    • Reward density is high, with clear success or failure signals;
    • Sampling cost is acceptable;
    • Within-group samples provide a stable comparison signal.

    However, before applying GRPO directly to code agent tasks, you need to validate it first. The trajectory distribution of code agents differs noticeably from that of mathematical reasoning: for the same task, some trajectories may give up quickly, while others may run many commands and repeatedly debug before completing.

    2. Conditions Where GRPO May Fail in Code Agents

    It is worth stressing that GRPO failure is not an inevitable conclusion; rather, it is a risk that needs focused validation under certain conditions.

    Within-group normalization can become unstable when the following occur:

    • Trajectory lengths vary enormously;
    • Success rates are very low or very high;
    • A few extra-long trajectories dominate the statistics;
    • Rewards are sparse, with a signal only when the final tests pass;
    • Sample quality within a group is highly uneven;
    • The model performs a large amount of ineffective exploration while few paths actually solve the task.

    Under these conditions, the mean and standard deviation may lose their representativeness. In particular, in high-variance, long-trajectory, sparse-reward settings, there can be an over-incentivization of “exploratory trial and error.” In some experimental setups, the expected payoff of “guessing randomly” may even exceed the payoff of “actually solving the task correctly.” This is not a universal rule, however, and must be confirmed through offline evaluation and ablation experiments.

    Therefore, whether GRPO applies should not be judged from experience alone. Run a small-scale experiment first: measure the within-group reward distribution, trajectory length distribution, success rate, and the proportion of samples contributing to variance, then decide whether to scale up.

    3. Critic-Based Value Methods: More Stable but Costlier

    For long-horizon tasks like those of code agents, value methods with a critic network, such as PPO, usually provide a smoother, more fine-grained signal. Rather than relying on relative ranking among samples from the same batch, they learn a state-value baseline and then judge whether the current action exceeds expectations.

    Note that you cannot simply write advantage = reward - value. In engineering practice, you typically need to combine terminal returns, the discount factor, GAE, importance sampling, and a clipped objective.

    1. # PPO: value network estimates state value
    2. values = critic(states) # score states
    3. returns = compute_returns(rewards, dones, gamma=0.99)
    4. advantages = compute_gae(rewards, values, gamma=0.99, lam=0.95)
    5. policy_loss = ppo_clipped_objective(
    6. logprobs=logprobs,
    7. old_logprobs=old_logprobs,
    8. advantages=advantages,
    9. clip=0.2
    10. )

    Key points here include:

    • returns must be computed from actual returns, not from a single-step reward;
    • gamma controls the discount on future rewards and usually needs careful tuning for long-horizon tasks;
    • GAE balances bias and variance;
    • clip limits the magnitude of policy updates;
    • advantages typically need normalization;
    • if the advantage is token-level or step-level, you also need process rewards or a more fine-grained signal design.

    The strength of value methods is a more stable signal, well suited to tasks with complex, multi-step trajectories. But they also depend on value network quality. If the critic is poorly estimated, or the reward itself is sparse and noisy, value methods can also degrade.

    4. How to Choose Among GRPO, PPO, and Rejection Sampling?

    Method Strengths Main Risks Better-Fit Scenarios Engineering Notes
    GRPO No critic needed, simple to implement, intuitive sampling comparison Easily distorted when within-group variance is unstable Stable trajectory lengths, high reward density Watch group size, success rate, outlier trajectories
    PPO Smoother signal, suitable for long-horizon tasks Requires a critic, complex tuning, higher cost High variance, multi-step, sparse rewards Watch GAE, discount, clipping, value calibration
    Rejection sampling Simple and direct, easy to implement Strong dependence on sampling quality, insufficient exploration When high-quality trajectories are easy to sample Filter low-quality trajectories, avoid distribution collapse

    Suggested selection steps:

    1. First, run sampling statistics on existing prompts, recording trajectory length, success rate, and reward variance.
    2. Analyze whether the within-group GRPO reward is dominated by a few samples.
    3. Compare GRPO, PPO, and rejection sampling at small scale.
    4. Look not only at training reward, but also at the offline validation set and cheating rate.
    5. If the training curve fluctuates heavily, first investigate reward contamination and distorted advantage estimation.
    6. If task success rate is extremely low and trajectories are long, first validate a critic-based value method.
    7. If the task success rate is stable and trajectories are short, GRPO may be more economical.

    Breakthrough 2: How to Build the Training Environment? Constructing Clean Rewards Through Executability

    Start with the conclusion: code tasks have a natural advantage in that they are executable. Under standard evaluation conventions, a compile failure or a failing test defaults to a score of 0; but in engineering practice you must add mechanisms such as hidden tests, timeouts, resource quotas, a read-only file system, log retention, and protection against reading evaluation artifacts.

    Code rewards are reliable because compilation and testing usually provide an objective signal. Unlike mathematical proofs, creative writing, or translation quality, whether code passes tests can often be verified automatically in many scenarios. But “executable” does not mean “naturally safe.” If the environment is poorly designed, the model can still take shortcuts, such as reading evaluation files or hardcoding outputs.

    1. Basic Reward Design

    A common reward function looks like this:

    1. def compute_reward(program_output):
    2. # Execute model-generated code in an isolated container
    3. result = run_in_isolated_container(program_output)
    4. if result.timeout:
    5. return 0.0
    6. if not result.compiled:
    7. return 0.0
    8. if result.public_tests_passed and result.hidden_tests_passed:
    9. return 1.0
    10. return 0.0

    Points to note here:

    • By default, failure gives zero reward;
    • A timeout should not be treated as a normal failure and should be flagged separately;
    • Public tests can be used for feedback, but hidden tests are better suited as the final reward;
    • if you adopt a partial-test-pass reward, you must assess whether it can be exploited by hardcoding or sampling attacks.

    2. Training Environment Executability Checklist

    To avoid reward contamination, complete at least the following checks:

    • Isolated sandbox: the model runs in an isolated container with unnecessary external network access blocked.
    • Read-only file system: model code, dependencies, and system files are read-only by default.
    • Temporary write directory: the model may write only to a designated working directory.
    • Hidden tests: public tests are used for feedback, hidden tests for the final verdict.
    • Timeout mechanism: set a run timeout; timed-out tasks get zero reward directly.
    • Resource quotas: limit CPU, memory, process count, disk, and open files.
    • Frozen dependencies: preinstall dependencies or lock versions to avoid network installs at runtime.
    • Log retention: keep standard output, error output, and execution traces for auditing.
    • Protection against reading evaluation artifacts: evaluation files, answer files, and test scripts are not exposed to the model.
    • Protection against environment variable leakage: avoid leaking hidden-test information through environment variables.
    • Task metadata isolation: task descriptions can be public, but answers, patches, and hidden tests remain invisible.
    • Reward contamination monitoring: watch for abnormal high scores, abnormally short trajectories, and abnormal pass rates.

    3. Why Hidden Tests Matter

    If the model can see only public tests, it may overfit to the public examples rather than truly solving the problem. For instance, it might write special-case logic for the public inputs and outputs without implementing general functionality. Hidden tests reduce the risk of such surface-level passing.

    But hidden tests should also not be fully exposed in evaluation logs. Otherwise, the model may infer hidden-test content from the logs across multiple samplings. Therefore, hidden tests should run only in an isolated environment, and the result should return only pass or fail, not the specific assertion details.

    4. Common Reward Design Mistakes

    • Giving only the final reward with no intermediate feedback, making training difficult;
    • Fully exposing public tests, causing the model to overfit to examples;
    • No timeout mechanism, allowing long-running processes to slow training;
    • Allowing network dependency installs, introducing non-determinism;
    • Letting multiple training tasks share the same working directory, causing contamination;
    • Returning the full test file directly in evaluation logs, causing information leakage;
    • Looking only at pass rate while ignoring cheating rate and abnormal trajectory distributions.

    Breakthrough 3: How to Handle Anti-Cheating? Two-Stage Detection and “No-Benefit” Feedback Boundaries

    Start with the conclusion: anti-cheating must be part of the training loop. You can adopt a two-stage mechanism of “rule-based filtering + LLM judge”; if you use fake-data feedback, it must be confined to an isolated training environment and cannot be used for real evaluation or production systems. For high-risk scenarios, prefer directly zeroing out, blocking, or terminating the trajectory.

    The simpler and more objective the reward, the more likely the model is to look for shortcuts. Common cheating behaviors in code RL training include:

    • Reading evaluation-related files or answer artifacts;
    • Copying answers from upstream submissions, caches, or existing patches;
    • Making external network requests to obtain source code or answers;
    • Hardcoding test assertions;
    • Probing hidden tests or environment variables;
    • Writing special-case logic only for public test cases.

    1. Two-Stage Detection Mechanism

    The first stage is rule-based filtering. The rule layer is fast and interpretable, well suited to intercepting clear, enumerable behaviors.

    1. def detect_cheating(trajectory):
    2. # Stage 1: rule-based filtering
    3. if reads_eval_artifacts(trajectory):
    4. return True, "read_eval_artifacts"
    5. if issues_external_network_calls(trajectory):
    6. return True, "external_network"
    7. if copies_upstream_solution(trajectory):
    8. return True, "copy_upstream"
    9. if accesses_hidden_tests(trajectory):
    10. return True, "access_hidden_tests"
    11. # Stage 2: LLM judge for semantic assessment
    12. if llm_judge.is_reward_hacking(trajectory):
    13. return True, "semantic_reward_hacking"
    14. return False, "pass"

    Items the rule layer can detect include:

    • Whether it reads evaluation files;
    • Whether it accesses the answer directory;
    • Whether it makes network requests;
    • Whether it reads sensitive information from environment variables;
    • Whether it calls evaluation functions;
    • Whether it copies existing patches;
    • Whether it hardcodes expected values in test files;
    • Whether it matches only public test examples.

    The second stage is the LLM judge. It is suited to detecting more subtle semantic-level cheating. For example, the code may run through on the surface, but it is really guessing test assertions and special-casing inputs and outputs rather than genuinely completing the task. Such problems usually cannot be identified by simple rules alone.

    2. What to Do After Detecting Cheating?

    Direct interception is the safest option: once cheating is detected, give no positive reward and, if necessary, terminate the trajectory.

    1. cheated, reason = detect_cheating(trajectory)
    2. if cheated:
    3. reward = 0.0
    4. mark_trajectory_invalid(reason)

    This approach is clear and safe, well suited to high-risk environments and to scenarios about to enter real evaluation.

    Another approach is “no-benefit feedback”: intercept the cheating request but return data to the model that looks legitimate yet is actually worthless, while letting it continue the rollout. The design goal is to let the model discover that cheating yields no benefit, gradually converging toward genuinely solving the task.

    But it must be stressed that this mechanism can be used only in an isolated training environment and cannot be used for real evaluation, production services, or systems facing external users. Returning fake data in a real system can mislead and may even introduce business risk.

    For anti-cheating strategy, tiered handling is recommended:

    • Clearly reading evaluation files or answers: zero reward directly, terminating the trajectory if necessary.
    • Clearly accessing the external network: intercept directly and log it.
    • Suspected test hardcoding: joint judgment by rules + LLM judge.
    • Semantic-level reward hacking: zero reward after LLM judge confirmation.
    • Used only for training exploration: no-benefit feedback may be used in an isolated sandbox.

    3. Anti-Cheating Monitoring Metrics

    Continuously monitor the following metrics:

    • Rule interception rate;
    • LLM judge false-positive rate;
    • Share of cheating trajectories;
    • Average reward of cheating trajectories;
    • Distribution of cheating across task types;
    • Changes in cheating behavior before and after training;
    • Whether validation-set pass rate and training reward move together;
    • Whether reward suddenly rises while offline capability declines.

    If training reward rises but offline evaluation does not rise or even drops, you should first suspect reward contamination or cheating behavior.

    Engineering Takeaways: The Gap Often Comes from Details Executed Thoroughly

    These three points are not isolated techniques but a complete chain:

    • The algorithm layer determines whether advantage estimation is accurate;
    • The environment layer determines whether the reward signal is clean;
    • The mechanism layer determines whether the model takes shortcuts.

    Many teams know they should build anti-cheating and use executable rewards, but few are truly willing to execute the details thoroughly. In practice, complete at least the following checklists.

    Algorithm Selection Checklist

    • Have you measured within-group reward variance?
    • Have you measured the trajectory length distribution?
    • Have you confirmed the success rate is not in an extreme range?
    • Have you verified whether GRPO is dominated by outlier trajectories?
    • Have you evaluated the critic’s value estimation quality?
    • Do you use GAE, discount factors, and a clipped objective?
    • Have you avoided equating single-step reward with token-level advantage?

    Training Environment Checklist

    • Is there a network-isolated sandbox?
    • Is there a read-only file system?
    • Are there resource quotas?
    • Is there a timeout mechanism?
    • Are there hidden tests?
    • Are logs retained?
    • Is reading evaluation artifacts prevented?
    • Are dependency versions frozen?
    • Are abnormal high scores and abnormally short trajectories monitored?

    Anti-Cheating Checklist

    • Is there a rule-based filtering layer?
    • Is there an LLM judge layer?
    • Are cheating types and trajectory IDs logged?
    • Are training environments distinguished from real evaluation?
    • Is fake data kept out of production environments?
    • Is a fallback option to zero out or terminate trajectories available?
    • Are high-risk tasks audited?
    • Are training environment snapshots saved?

    Parameter Tuning Recommendations

    The following are common starting points and need to be ablated against actual tasks:

    • GRPO: group size can start at 8, 16, or 32; if the success rate is low and variance is high, increase the group size, but also monitor sampling cost.
    • GRPO: if within-group rewards converge over a long period, the task lacks discriminative power and you need to switch tasks or raise difficulty.
    • PPO: gamma can start at 0.99 or 1.0; lam can start at 0.95; clip can start at 0.2; value loss and policy loss need to be balanced.
    • Value methods: if advantage estimation fluctuates heavily, first check whether the critic is underfitting rather than directly tuning the policy learning rate.
    • Anti-cheating: the rule layer should prioritize high-frequency, clear, low-cost cheating; the LLM judge handles low-frequency, subtle, semantic-level cheating.

    There is no universal formula for code agent post-training. What truly separates teams is often not a single algorithm, but the engineering system formed jointly by algorithm selection, environment isolation, reward design, and anti-cheating mechanisms. Only by executing these details thoroughly can you train a code agent that is more stable, more reliable, and better at solving real problems.