Code LLM Post-Training End-to-End: From Executable Feedback to Stability
Code capability post-training can be organized into a reproducible five-stage closed loop: an executable task pool, engineered sandboxes and execution feedback, cold-start supervised fine-tuning, reinforcement learning with verifiable rewards, and stability engineering with data backflow. This article explains why code tasks are uniquely suited to machine-verifiable feedback, how to define stability as optimizable metrics beyond pass@1, how to design layered reward signals and guard against reward hacking, and how evaluation gates plus failure backflow turn one-off training into a repeatable pipeline. It includes task manifest and code examples, a troubleshooting table for common failure modes, and three questions for judging whether a team has truly mastered the methodology.
Before judging whether a code model is “good,” you need to answer one question first: good on which dimension? In real engineering use, the differences developers perceive are usually not “can it get it right once,” but “out of ten consecutive attempts, how many are usable,” “can it read and understand a repository with tens of thousands of lines,” and “when it fails, does it stay put or iterate on its own to fix it.” Most of these experience differences come not from scale differences in the pre-training stage, but from how complete the post-training pipeline is.
This article breaks down code post-training into a reproducible five-stage closed loop, explaining what each stage solves, how to verify that it works, and where the common pitfalls lie.
1. Why Code Capability Is Naturally Suited to Post-Training Amplification
Code tasks differ fundamentally from open-ended writing or chat tasks: their correctness can be automatically determined by a machine.
Whether a piece of Python code is correct doesn’t rely on a human annotator’s subjective judgment—run the tests and you know. Whether a C++ file compiles is something the compiler answers directly. Whether a refactor introduced a performance regression can be quantified by benchmark scripts. This means we can obtain massive, objective, automatically verifiable feedback signals at very low cost.
This determines the methodological backbone of code post-training: replace “human preference alignment” with “executable environment judgment” as much as possible. Whenever something can be automatically determined, don’t introduce fuzzy human scoring, because fuzzy rewards lead to two consequences—the model learns to please the reward function rather than solve the problem, and the training curve becomes too noisy to attribute.
The second characteristic is long-range dependency. Real engineering tasks are often “read thirty files → locate the problem → change three places → run tests → fix again based on the failure output.” Such tasks cannot be verified in a single turn and require multiple rounds of interaction. Therefore the data form for post-training can’t just be “problem-answer” pairs; it must be four-tuples of “task-environment-action trajectory-result.”
The third characteristic is highly structured format. Edit operations are presented as diffs, tool calls have explicit schemas, and test results have definite output formats. Structured output is training-friendly, but it also brings a unique risk: the model may learn only the format and not the intent (see Section 11).
2. First, Define “Stability” as Optimizable Metrics
When discussing stability in engineering, you should at least distinguish three levels of metrics, which correspond to completely different optimization approaches:
- Single-attempt correctness (pass@1): the probability that the model passes all tests in a single sample. This is the most commonly used metric, but it is sensitive to sampling temperature and fluctuates considerably.
- Capability ceiling (pass@k): the probability that at least one of k samples of the same task passes. It measures “whether the model knows how to do it,” which is a different thing from “whether the model reliably does it right.” A model may have high pass@k but low pass@1, meaning the answer exists in the distribution but simply isn’t sampled with high probability.
- Stability metrics: including the variance of pass rates across multiple samples, the proportion of repeated failures with fixed patterns, the average number of retry rounds, and the token consumption and sandbox execution count per task.
Only when all three types of metrics are observed simultaneously can you judge whether a strategy update actually “improves capability” or merely “narrows the output distribution.” Many teams see pass@1 rise after training and declare success, when in fact the model has learned to spin within a few templated writing styles and collapses entirely when facing tasks that deviate even slightly from the training distribution.
A practical minimal observation panel should include:
Task level: pass@1 / pass@4 / average retry rounds / first-pass rateDistribution: output length quantiles / distribution of files edited / distribution of tool-call roundsStability: result histogram of running the same task 5 times / clustering of failure reasonsCost level: token consumption per task / sandbox execution count and total duration
3. Overview of the Five-Stage Closed Loop
[1] Executable task pool → [2] Sandbox & execution feedback → [3] Cold-start SFT↓[5] Eval gate & version comparison ← [4] Verifiable-reward RL + rejection sampling backflow↓Backflow as new tasks / failure samples (back to stage 1)
The key is that last backflow edge. A pipeline that does one round of training and stops cannot deliver continuous, stable improvement; teams that can iterate continuously essentially convert “where the model fails” automatically into “the next round of training data.”
4. Stage One: The Executable Task Pool
The quality of the task pool determines the ceiling of the entire pipeline. Three principles should be upheld when building it.
Principle one: tasks must carry their own judgment conditions. Don’t accept descriptions like “implement a sorting function” that cannot be automatically judged. Each task must have at least: an initial code state, an executable command, and a judgment script (usually test cases).
Principle two: sources should be layered. It is recommended to mix at least three types of sources, with proportions adjusted to the business scenario:
| Source | Characteristics | Main role |
|---|---|---|
| Problems and fix records from real business repositories | Close to real distribution, natural difficulty | Improves repository-level task performance |
| Synthetic / variant tasks | Batch-generatable, controllable difficulty | Covers long-tail scenarios, enables decontamination |
| Public, verifiable algorithm problems | Clean judgment, low ambiguity | Solidifies basic correctness and unit-test understanding |
Principle three: strict decontamination. The tasks used for the evaluation set and their variants must be completely excluded from the training pool. The practical difficulty here lies in judging “variants”: variable renaming or minor boundary-condition tweaks of the same problem are enough to constitute leakage. It is recommended to maintain task fingerprints (a semantic hash of the problem description plus a hash of the test assertions) and filter bidirectionally at both the ingestion and evaluation entrances.
A readable task manifest example:
task_id: repo-fix-000173repo: internal/service-acommit_base: 3f2c1abproblem: "Concurrent writes cause a race condition in the counter, losing updates under load"language: gobuild_cmd: go build ./...test_cmd: go test -race ./internal/counter/ -count=1timeout_sec: 180allowed_files: # Restrict the modifiable scope to prevent the model from bypassing the task- internal/counter/*.gohidden_tests: true # Visible during training, hidden during evaluation, prevents gamingmetadata:difficulty: mediumtags: [concurrency, race-condition]
allowed_files and hidden_tests are two easily overlooked but very important fields. The former prevents the model from “cheating” by modifying tests or unrelated modules; the latter prevents the model from hardcoding against specific assertions.
5. Stage Two: Engineering the Sandbox and Execution Feedback
This is the most underestimated cost part of the whole pipeline. A single RL iteration often requires tens of thousands to hundreds of thousands of code executions; if the sandbox is poorly built, training gets stuck waiting in a queue.
Image layering and preheating. Split “base runtime + dependency installation + repository source” into three image layers, make the first two long-term cached base images, and let the task image contain only the source-level difference. Combined with image preheating on the nodes, this can compress the startup time of most tasks down to the second level.
Resource isolation and security. The execution environment must default to no external network access, read-only mounts for all paths except the workspace, limits on process count and memory, and forced termination on timeout. Untrusted code is the norm in a training environment; inadequate isolation leads to hard-to-trace resource exhaustion.
Layered feedback collection. Don’t return only the boolean “pass/fail.” It is recommended to collect and return in structured form: compile diagnostics, failed test-case names with assertion differences, key stderr snippets, and resource usage plus elapsed time. This information directly becomes the model’s next input, and its quality determines whether self-repair capability can form.
def run_task(task, workspace) -> Result:"""Execution order: build → visible tests → static check → hidden tests.Early stopping significantly reduces cost, but you must tag each stageto distinguish two kinds of signals: 'build failure' versus 'logic error'."""build = sandbox.exec(task.build_cmd, workspace, timeout=120)if build.returncode != 0:return Result(stage="build", passed=False, feedback=parse_diag(build))visible = sandbox.exec(task.test_cmd, workspace, timeout=task.timeout_sec)if visible.returncode != 0:return Result(stage="test", passed=False,feedback=parse_failures(visible),pass_ratio=visible.passed / visible.total)lint = sandbox.exec("static_check", workspace, timeout=60)hidden = sandbox.exec(run_hidden_tests, workspace, timeout=task.timeout_sec)return Result(stage="hidden", passed=hidden.returncode == 0,lint_warnings=lint.warnings,feedback=parse_failures(hidden))
Combining on the platform side. This kind of large-scale execution cluster is usually built from general cloud capabilities: a container platform hosts the elastic sandbox pool and image cache, object storage holds task packages, trajectory logs, and model checkpoints, a message queue handles task distribution and backpressure control, and a logging service plus monitoring and alerting provides pipeline observability. For example, on Baidu AI Cloud this combination can be implemented without any proprietary interfaces; the key is queue priority design and image preheating strategy, not which vendor you choose.
Backpressure and priority. The sandbox is a finite resource, so priority scheduling is essential: evaluation tasks > RL sampling > rejection-sampling generation. Otherwise a full evaluation run will starve the training samples.
6. Stage Three: Cold-Start Supervised Fine-Tuning
Before introducing RL, you usually need a cold-start SFT whose purpose is to make the model learn format and basic interaction protocols, not to raise the capability ceiling. The positioning of this step must be clear: it provides a usable behavior distribution for the subsequent RL; it is not the endpoint.
There are several key points on data proportions:
- Keep multi-turn trajectories complete. This includes the failed intermediate steps and the final fix. Keeping only successful trajectories deprives the model of the learning signal for “how to adjust after seeing an error,” and that part is precisely the key to improving stability.
- Control the proportion of long samples. Too high a share of ultra-long, repository-level samples makes training time explode and also pushes the model toward lengthy, irrelevant explanations. It is recommended to bucket by length and sample, rather than mixing uniformly by count.
- Match language and framework proportions to the business distribution. If the business is a Java backend but 80% of the mix is given to scripting languages, the gains from subsequent RL will be diluted.
- Watch out for format overfitting. It shows up as the model producing perfect format within the training distribution but unparseable output on new tasks. Mitigation is to keep a small number of format variants in the data and to include cross-format tasks in the evaluation set.
7. Stage Four: Reinforcement Learning with Verifiable Rewards
This is the core of the whole pipeline, and also the watershed for whether the methodology is truly mastered.
7.1 Designing the Reward Signal
The recommended reward function should be layered and monotonic:
def reward(result, task):if result.stage == "build":return -0.5 # Build failure: negative, but not zero, to keep a gradientr = 0.2 * result.pass_ratio # Dense signal: test pass ratioif result.stage == "test":return rr += 0.3r -= 0.05 * result.lint_warnings # Code-quality constraint, weight must be smallif result.stage == "hidden" and result.passed:r += 0.5 # Sparse but strong: all hidden tests passreturn r
Three design points:
- Sparse rewards must be paired with dense intermediate signals. A binary “all pass / fail” reward alone is nearly impossible to learn from on hard tasks, because there is no gradient to point toward. The test pass ratio and whether the build stage advanced are all cheap dense signals.
- Preventing reward hacking matters more than the reward itself. The model will find loopholes in the reward function: deleting test files, hardcoding inputs and outputs, rewriting assertions to be always true, using exceptions to swallow failures. Therefore hidden tests, limits on the modifiable file scope, and explicit penalties for changes like “deleting or weakening tests” are required items, not optional ones.
- Constraint terms should be weak. Preference-type rewards such as style, comments, and lines of code must carry far lower weight than correctness rewards; otherwise the model will sacrifice logical correctness to earn style points.
7.2 Key Controls in the Training Loop
When using policy-gradient-based RL methods, three things mainly determine stability:
- Advantage normalization. Sampling a group of trajectories for the same task and normalizing with in-group relative scores automatically offsets task-difficulty differences, preventing simple tasks from dominating the gradient.
- Deviation constraint between old and new policies. Limit the per-step update magnitude through clipping or a KL penalty. Too loose a constraint causes output collapse (the model converges to a few templates); too tight one hurts sample efficiency.
- Entropy control. A rapid entropy drop is a precursor to collapse. It is recommended to treat entropy as a first-class monitoring metric rather than an after-the-fact analysis item.
The following is only illustrative; specific hyperparameters must be tuned through repeated experiments on your own task distribution—there is no universally optimal value:
for step in range(total_steps):tasks = sampler.sample(batch_size) # Difficulty-stratified task samplinggroup = []for t in tasks:trajs = [policy.rollout(t) for _ in range(group_size)]scores = [reward(run_task(t, tr.workspace), t) for tr in trajs]group.append(normalize(trajs, scores)) # In-group advantage normalizationloss = policy_loss(group, clip_ratio=0.2, kl_coef=0.02)loss.backward()optimizer.step()if entropy() < entropy_floor or kl() > kl_ceiling:rollback_and_reduce_lr() # Stability guardrailif step % eval_interval == 0:metrics = evaluate(holdout_set) # Private holdout, never enters traininggate(metrics) # Block release if the gate fails
Sampling-difficulty stratification is equally important: if the task pool is dominated by extremely simple tasks, RL saturates quickly; if it is dominated by extremely hard tasks, most samples fail entirely, reward variance becomes zero, and the gradient carries no information. In practice, tasks are dynamically stratified by historical pass rate, keeping the model always in the “partially able to get it right” region.
8. Stage Five: Stability Engineering and Data Backflow
After RL, further gains in stability come mainly from three places.
Rejection-sampling backflow (self-distillation). Sample each task multiple times with the current policy, keep only the trajectories that pass the hidden tests, mix them with newly collected failure-fix trajectories, and run another round of SFT. Iterating repeatedly forms a positive feedback loop. Note two things: first, each iteration must redo the decontamination check; second, avoid keeping only the “prettiest” trajectories, or the model loses diversity.
Targeted data supplementation for failure samples. Cluster the failures seen in evaluation by cause—dependency version conflicts, insufficient context, unhandled boundary cases, cross-file change omissions—and construct targeted training data for each category. This is the only effective path for converting evaluation results into training assets.
Inference-side stabilization. Beyond post-training, the engineering side can also significantly improve stability: multi-sample and then vote by test results, automatically send failures into a repair loop (with max-round and budget caps), and run a static self-check before execution. These measures don’t change model capability but can markedly improve the user-perceived success rate.
A repair loop with budget control:
def solve_with_repair(task, policy, max_rounds=4, budget=None):ctx = build_context(task)for rnd in range(max_rounds):patch = policy.generate(ctx)result = run_task(task, apply_patch(task.repo, patch))if result.passed:return success(patch, rounds=rnd + 1)if budget and budget.exhausted():break# Key: feedback must be structured and trimmed; overly long logs crowd out the contextctx.append(patch, compact_feedback(result.feedback, max_tokens=2000))return failure(ctx, last_result=result)
Feedback trimming is an underestimated part. Full test logs are often tens of thousands of lines; stuffing them straight back into the context crowds out the original problem description, causing the model to “forget what it was supposed to fix.” Keeping the failed test-case names, the differences between assertion expected and actual values, and the most recent build diagnostics is usually more effective than the full log.
9. Evaluation and Regression Gates
The design goal of the evaluation system is to answer two questions: did this change make things better? And where specifically did it get better or worse?
It is recommended to use three layers of evaluation:
- Unit level: algorithm problems and function-level tasks, with clean judgment, used for fast regression.
- Repository level: multi-file defect fixes, dependency upgrades, test completion—close to real usage.
- Business level: a sampled set of internal real tickets, used only for relative comparison between versions.
Beyond accuracy, you must include the cost dimension: average token consumption per task, average sandbox execution count, and P95 completion time. Looking only at accuracy rewards strategies like “unlimited retries” that can never go to production.
The purpose of the gate mechanism is to prevent metric regressions from being silently merged. It is recommended to base it on a private holdout set, set hard constraints that correctness must not drop and that cost and latency must not rise significantly, and block release if any item fails. Also periodically rotate the holdout tasks to avoid implicit overfitting from long-term evaluation.
10. Common Failure Modes and Troubleshooting
| Symptom | Common root cause | Handling direction |
|---|---|---|
| High training-set pass rate, sharp drop on the eval set | Task leakage or format overfitting | Bidirectional decontamination; expand format variants |
| High pass@k but low pass@1 | Answer exists in the distribution but isn’t sampled with high probability | Rejection-sampling backflow; check for entropy collapse |
| Passes tests but code quality is poor | Reward hacking; the model learns to bypass assertions | Hidden tests; restrict modifiable files; penalize test deletion/modification |
| Entropy collapses fast, output becomes templated | Constraint too loose or too high a share of simple tasks | Tighten update magnitude; stratify task difficulty |
| Performance worsens after adding feedback | Context crowded out by logs | Structurally trim feedback; limit rounds |
| Training stagnates for a long time | Tasks too hard, reward variance is zero | Dynamically adjust the sampling-difficulty distribution |
| Sandbox queue chronically saturated | Evaluation and sampling competing for resources | Priority scheduling; image preheating; cache reuse |
11. Conclusion
The reason code-model post-training can keep delivering returns is not any single-point trick, but that the backbone of “executable feedback” is fully connected: the task pool ensures the signal is trustworthy, the sandbox ensures the signal is cheap and safe, cold-start provides a usable behavior distribution, RL converts verifiable rewards into capability, rejection sampling and failure backflow make capability improvement a repeatable loop, and the evaluation gate ensures every step is a net gain.
To judge whether a team has truly mastered this methodology, you can ask three questions: after one training round, can they accurately state which category of tasks the capability change occurred in? Can they automatically convert the failures found in evaluation into the next round of training data? Can they guarantee releases without metric regression?
When all three questions can be answered clearly, “continuing to get stronger” no longer depends on luck—it becomes an engineering pipeline with a certain output.
