How to Evaluate the Post-Training Maturity of LLM Code Capabilities

    A vendor-neutral guide to assessing the post-training maturity of LLM code generation. It explains why evaluating code ability means more than benchmark scores — measuring the ceiling, floor, and variance of correctness — and why stability metrics such as pass@k, consistent-pass probability, and failure-mode distribution matter more than a single pass rate. It shows how verifiable rewards let models learn from real execution, walks through a complete post-training pipeline (data engineering and SFT, rejection-sampling self-training, reinforcement learning toward verifiable rewards, and end-to-end tool-execution training), and offers a practical evaluation workflow plus infrastructure considerations for developers and enterprises, along with common limitations and misconceptions.

    In discussions of LLM code capabilities, phrases like “a team has connected the post-training pipeline” are often treated as an important conclusion. Yet the statement is imprecise: does it mean the model happened to score well in a single evaluation, or that the team has a methodology it can keep iterating? For developers and enterprise users, judging this matters more in practice than comparing a handful of benchmark scores.

    This article takes a vendor-neutral view of LLM code post-training and attempts to answer three questions:

    • How should code ability actually be evaluated?
    • What stages does a complete post-training pipeline typically include?
    • How can a team tell whether a model’s capability gains are sustainable?

    1. Redefining “strong code ability”: from “writes something” to “reliably writes it correctly”

    In conversational coding, getting a model to produce code that “looks reasonable” is not difficult. What is difficult is producing, repeatedly across a real project, code that satisfies all of the following:

    • Functional correctness: it passes all test cases;
    • Style control: it stays consistent with the existing project’s conventions and dependencies;
    • Restrained changes: it only modifies what is necessary and does not spontaneously refactor unrelated code;
    • Clear boundaries: it states its assumptions and what it does not cover;
    • Behavioral stability: under the same inputs and parameters, the quality varies little across multiple generations.

    As a result, code ability should be evaluated along at least three dimensions: the ceiling, the floor, and the variance. The ceiling determines whether the model can complete complex tasks, the floor determines the loss when generation fails, and the variance determines whether it can be reliably integrated into an automated pipeline. For developers, variance often matters more than the ceiling. If a model fails 3 out of every 10 generations on average, the automated workflow must add extensive retries, manual review, and exception handling, and the end-to-end cost will quickly offset the model’s intrinsic advantage.

    2. Why stability matters: measuring code post-training beyond a single pass rate

    Traditional code-generation research often measures a model with the single-sample pass rate — sample once per problem and count how many pass the tests. This metric is intuitive but reflects only “one instance of luck.” Metrics closer to engineering practice include:

    1. Pass@k probability: sample each problem independently k times and measure the probability that at least one sample passes. It roughly captures “whether the task ultimately gets completed with a retry mechanism.”
    2. Consistent-pass probability: the probability that all k samples pass. This directly corresponds to whether an automated pipeline can run stably, and is a key indicator of stability.
    3. Variance in the number of passes: reflects whether the model’s performance swings wildly, i.e., how much results fluctuate across repeated runs of the same task.
    4. Failure-mode distribution: categorizes failures into compile errors, runtime exceptions, failing test cases, or resource-limit violations. This metric determines the direction of subsequent optimization.

    In practice, the same model can show a substantial gap between “passes at least once in k” and “passes all k times.” This shows that a single-sample pass rate is not enough to support a conclusion. Under controlled prompt formats, sampling parameters, and evaluation sets, an occasional lead by a particular model often says little; what really matters is whether repeated tests under the same evaluation conditions produce stable results, and whether that stability holds across different task types. This is the phenomenon most worth watching when assessing “post-training maturity.”

    It is worth stressing that a single-dimension comparison — especially an incidental advantage on a particular evaluation set — should not be the sole basis for technology selection. A more reliable approach is to run multi-round, multi-dimensional evaluations on a dataset that closely matches your own business.

    3. Looking at post-training through test cases: how verifiable rewards drive model learning

    To understand why code tasks are well suited to continuous improvement through post-training, first understand what “verifiable rewards” means.

    In reinforcement learning, the model generates several candidate answers for a prompt, a reward mechanism scores each candidate, and training adjusts the model to favor higher-scoring answers. For most open-ended generation tasks, “good” versus “bad” depends on subjective judgment or an additionally trained reward model, which easily introduces bias; the model may also learn to please the reward model rather than actually solve the problem.

    What makes code tasks unique is that, through compilation, execution, and running unit tests, “whether the result is correct” can be objectively and automatically determined. This means the reward signal comes from a real execution environment rather than the model’s own preferences. The training objective and the real objective are therefore highly aligned.

    Take “fixing a code defect” tasks as an example. The training process can be summarized as:

    • Prepare a large number of real or simulated code repositories, each containing a buggy version with failing test cases and the corresponding correct implementation;
    • For each buggy snapshot in a repository, prompt the model to locate and fix the problem;
    • Apply the model-generated patch in an isolated environment, compile it, and run the unit tests;
    • Compilation succeeds and all tests pass is recorded as success; a regression or test timeout is recorded as failure;
    • Successful samples are used to preserve or reinforce the corresponding generation behavior, while failed samples are used to analyze weak points and guide the next round of data and reward design.

    This “generate–execute–feedback” loop is the core of the code post-training methodology. It closely mirrors how a human engineer works: write code, run tests, read errors, and revise. Precisely because the feedback comes from real execution, the model learns not “text that resembles the correct answer” but “a solution that actually passes verification.”

    4. A complete post-training pipeline: from self-training to end-to-end execution

    If verifiable rewards are the starting point of the pipeline, what really determines the results is often whether the pipeline is complete. Public method research and practice in the industry suggest that a relatively complete pipeline usually includes the following stages, each providing a higher-quality starting point for the next.

    Stage 1: Data engineering and supervised fine-tuning (SFT)

    High-quality, diverse instruction data is the foundation, covering task types such as function implementation, defect repair, code explanation, unit-test generation, and refactoring suggestions. The data must undergo rigorous deduplication, denoising, and format normalization to ensure clear structure in both inputs and expected outputs. Supervised fine-tuning teaches the model the basic format and interaction patterns of tasks, but usually cannot reach a stable high correctness rate in one pass.

    Stage 2: Rejection-sampling self-training

    Using the model itself, batch-sample many candidates for the same prompt — for example, sample the same problem dozens of times — and then let the execution environment filter them automatically. Candidates that compile and pass all tests are treated as positive samples, used to construct new training data, and the model is fine-tuned again. This is often called rejection-sampling fine-tuning (R²FT) or self-training:

    • Moderately difficult tasks usually yield a stable supply of positive samples;
    • Hard tasks need more sampling, sometimes using a stronger model to generate reference solutions;
    • Data quality depends entirely on the reliability of the verifier.

    The underlying logic is: the model first “tries all possibilities,” then “learns the path it most often gets right.”

    Stage 3: Reinforcement learning toward verifiable rewards

    In this stage, the model batch-samples multiple candidates for the same prompt, scores each one, and adjusts its policy based on the scores. Compared with rejection sampling, a key advantage of reinforcement learning is that even if all candidates for a problem fail, a relatively better candidate can still receive a higher reward, providing a signal for improvement.

    Common algorithms include Proximal Policy Optimization (PPO) and various simplified policy-optimization methods. In engineering practice, teams often design both an “outcome reward” and a “process reward”: the former focuses on whether all tests eventually pass, while the latter gives partial credit for intermediate signals such as successful compilation, the proportion of unit tests passed, and reasonable modification scope, mitigating training stagnation caused by sparse rewards.

    Stage 4: End-to-end tool-execution capability

    Most of the above stages target single-step generation, whereas real development tasks are usually a multi-step process: reading project files, locating relevant code, making modifications, running tests, viewing errors, and revising again. The pipeline therefore also needs to train the model to complete tasks within a multi-round loop of “call a tool — observe the result — decide the next step.” Such training environments need capabilities like file-system read/write, command execution, and test running, and should work with an isolated runtime environment to reduce the accumulation of errors across multi-round interactions.

    From a static view, the outputs of each stage are roughly as follows:

    Stage Input Core output Typical failure risk
    Data engineering and SFT Cleaned instruction data Task format and interaction habits Data homogenization, insufficient generalization
    Rejection-sampling self-training Batch-sampled candidates + verifier High-correctness positive samples Lax verification lets noise into training
    Reinforcement learning toward verifiable rewards Executable environment + reward function Higher and more stable correctness Reward is “gamed”
    End-to-end tool-execution training Multi-step task environment Planning and iterative repair ability Error accumulation, getting stuck in loops

    5. The lasting value of the methodology: iterability over a one-time lead

    Back to the original question: how do you evaluate the post-training maturity of a model’s code ability? The key is not a ranking in some single evaluation, but whether a reusable feedback mechanism exists behind it. Having such a mechanism means the team possesses:

    • A code evaluation set that can be continuously accumulated and automatically expanded;
    • A reliable validation environment based on real execution;
    • A closed loop spanning data construction, sampling, training, and evaluation;
    • The ability to automatically attribute failed samples.

    Once these capabilities form, the effect is cumulative: the new data produced by each training round becomes the starting point for the next. Model capability can keep improving as data and environment quality improve, rather than depending on one incidental training configuration.

    For developers and enterprise users, the following approach can be taken when evaluating candidate models:

    1. Build a small evaluation set close to your own business, covering common scenarios such as code completion, defect repair, test generation, and cross-file modification;
    2. Fix the prompt format and sampling parameters, sample each task multiple times, and record both the single-sample pass rate and the consistent-pass rate;
    3. Analyze the failure-mode distribution to distinguish “insufficient capability” from “environment or configuration problems”;
    4. Prefer models with explainable and reproducible failure behavior over those that only lead in a single comparison.

    6. Engineering implementation: turning evaluation capability into product capability

    Whether building your own evaluation system or integrating model capabilities into the development workflow, supporting infrastructure is required. Below are several key elements.

    1. Isolated execution environment

    Code generated by the model must run in a controlled environment. Using a container platform, assign an independent sandbox to each task, limiting CPU, memory, and execution time so that malicious code or infinite loops cannot affect the host. This is also the most easily overlooked part of automated evaluation.

    1. Observability of evaluation results

    Record the prompt, candidates, execution logs, pass status, and time taken for each generation, making results easy to reproduce and attribute. Log services and monitoring/alerting can track the running status of evaluation tasks and issue timely warnings for problems such as sudden spikes in failure rate or execution timeouts.

    1. Data and result management

    Evaluation sets, training data, and historical results need version control and should be stored using general-purpose capabilities such as object storage, ensuring comparability across rounds. Only when data is traceable can conclusions be reproduced.

    1. Progressive integration

    Pilot first in low-risk areas such as code explanation, unit-test generation, and defect-repair suggestions, then gradually expand to changes involving core logic. Meanwhile, keep human review and automated testing as safety boundaries.

    Baidu AI Cloud provides a combination of general-purpose capabilities ranging from model services and container platforms to object storage, logging services, and monitoring/alerting, allowing enterprises to build the evaluation and integration workflows described above in their own environments. Note that capabilities vary continuously across models and versions, so any conclusion should be validated against your own data rather than directly copied from external evaluations.

    7. Limitations and common misconceptions

    While affirming the value of the post-training methodology, its boundaries should also be recognized.

    First, data and compute thresholds are high. A complete pipeline requires large-scale executable code tasks, a stable and reliable execution environment, and compute resources for multiple training rounds. Small and medium-sized teams can prioritize building evaluation and validation systems before gradually introducing the training stages.

    Second, evaluation sets carry bias. Public evaluation sets may be covered by training data, inflating scores. Self-built, private, and continuously updated evaluation sets are key to avoiding “benchmark gaming” distorting judgment.

    Third, stability does not mean zero errors. Even with a high pass rate, work involving critical business still requires engineering safeguards such as unit testing, code review, and canary releases.

    Fourth, code ability should be viewed in layers. Completion, single-file modification, defect repair, and cross-repository refactoring differ greatly in difficulty, so evaluation should state the specific task and conditions rather than reach sweeping conclusions.

    Fifth, cost needs to be assessed holistically. A high consistent-pass rate can significantly reduce retries and manual intervention, so the overall cost is not necessarily higher than a cheaper-but-less-stable option with a low per-run cost.

    Conclusion

    Evaluating a model’s code ability is, in essence, evaluating the completeness and sustainability of the post-training system behind it. “Writes something” can rely on data and scale, while “reliably writes it correctly” depends on a closed loop of execution-based validation, self-improvement, and end-to-end tool use. This path is not unique to any one team, yet it is indeed one of the more reliable technical directions for improving code ability today. For developers, rather than chasing a ranking in some single evaluation, it is better to build your own evaluation and validation system and use data to judge which model truly fits your engineering scenario.