Evaluating Post-Training Effectiveness for Code Large Language Models: A Practical Methodology

    This article lays out a methodology for judging whether post-training produces sustained, predictable gains in a code LLM's engineering ability, rather than relying on single benchmark scores. It breaks model quality into eight measurable dimensions—functional correctness, stability, instruction following, multi-turn fix convergence, repository-level understanding, tool use, security and engineering compliance, and cost—and explains why variance and stability matter as much as averages. It also covers the key elements of a reinforcement post-training pipeline that keeps producing gains, how to organize sandbox-based evaluation engineering, and how to roll out model upgrades through offline regression, shadow traffic, and canary releases, closing with common pitfalls to avoid.

    1. Reframing the Problem: Why Post-Training Is the Dividing Line for Coding Ability

    Pre-training solves the question of “what has been seen”—language distribution, grammatical structure, and the engineering conventions implied in public code. Post-training solves the question of “how to act”—given a context, how to choose an action, how to organize reasoning, and how to self-correct after a failure. For a highly constrained task like code generation, the marginal return from the latter is often steeper, for three capability gaps:

    • From fragment-level correctness to a runnable whole: A single function that is syntactically correct does not mean the type check passes, the dependencies resolve, or edge cases are handled.
    • From a single output to multi-turn convergence: Real development is a loop of “generate—compile—read the error—fix—try again,” and the capability ceiling is set by how fast this loop converges.
    • From passive completion to proactive tool use: Modern coding assistants need to search the codebase, run commands, execute tests, and adjust their approach based on feedback.

    As a result, the focus of evaluation should not be “the absolute score of one model at one version,” but “whether the post-training methodology can consistently produce gains.” The former is a single sample; the latter is a curve.

    2. Eight Dimensions That Must Be Measured Separately

    Breaking “is the model good or not” into measurable items is the first step to avoiding subjective judgment.

    1. Functional correctness: Determined by actually executed unit test results, not by the impression of reading the code.
    2. Stability and variance: The distribution of pass rates across repeated samples of the same task, which determines predictability on the engineering side.
    3. Instruction and format following: Whether it strictly adheres to constraints such as language version, output structure, forbidden dependencies, and code style.
    4. Multi-turn fix convergence: The fix success rate given error messages after an initial failure, and the average number of rounds required.
    5. Repository-level and cross-file understanding: Whether a change affects callers, whether related modules are missed, and whether the correct files can be located.
    6. Tool calling and agentic workflows: Correctness of command construction, error-recovery ability, and how it trades off on very long contexts.
    7. Security and engineering compliance: Whether it introduces unsafe patterns, hardcoded credentials, or dependencies with incompatible licenses.
    8. Efficiency and cost: First-token latency, token consumption per task, and the average number of attempts needed to pass.

    3. Stability: The Key Metric Hidden by Averages

    Imagine two models, each sampled 10 times on the same set of 200 tasks:

    Metric Model A Model B
    Average pass rate 0.71 0.70
    Standard deviation across tasks 0.31 0.12
    Share of fully passing (stable) tasks 41% 62%

    The average scores are almost identical, but Model A’s distribution is long-tailed: a few tasks almost always pass, while another batch almost never passes. Model B, by contrast, sits at a mid-to-high water level across most tasks. In scenarios that require “predictable results,” such as continuous integration and code-review assistance, Model B is clearly more usable in practice—because it reduces the number of manual fallbacks and pipeline jitter.

    A practical way to measure this: sample each task k times (k is typically 5–10), compute avg@k, pass@1, and task-level variance, and additionally track two engineering-oriented metrics:

    • Stability index: The share of tasks whose task-level pass rate falls within the [0.8, 1.0] range.
    • Convergence rate: With a fix budget of no more than n rounds, the share of tasks that eventually pass, along with the average number of rounds consumed per task.
    1. def evaluate(model, tasks, k=10, max_fix_rounds=3):
    2. per_task = []
    3. for task in tasks:
    4. trials = []
    5. for _ in range(k):
    6. trials.append(run_in_sandbox(model, task, max_fix_rounds))
    7. trials = [1 if t.passed else 0 for t in trials]
    8. per_task.append({
    9. "pass_rate": sum(trials) / k,
    10. "rounds_used": mean([t.rounds for t in trials]),
    11. })
    12. rates = [t["pass_rate"] for t in per_task]
    13. mean_rate = sum(rates) / len(rates)
    14. variance = sum((r - mean_rate) ** 2 for r in rates) / len(rates)
    15. return {
    16. "avg_pass_rate": mean_rate,
    17. "pass@1": mean(rates),
    18. "task_std": variance ** 0.5,
    19. "stability_index": len([r for r in rates if r >= 0.8]) / len(rates),
    20. "avg_fix_rounds": mean([t["rounds_used"] for t in per_task]),
    21. }

    Statistical rigor: When the task set is small and variance is large, a 1–2 percentage point gap between two models means nothing. You should apply bootstrap resampling to produce confidence intervals for the metrics, and only claim “better” when the intervals do not overlap.

    1. def bootstrap_ci(rates, n=2000, alpha=0.05):
    2. means = []
    3. for _ in range(n):
    4. sample = [rates[i] for i in randint(len(rates), size=len(rates))]
    5. means.append(sum(sample) / len(sample))
    6. means.sort()
    7. return means[int(n * alpha / 2)], means[int(n * (1 - alpha / 2))]

    4. Key Links in a Post-Training Reinforcement Chain That Keeps Producing Gains

    A pipeline that consistently produces gains usually contains the following elements; missing any one of them will cause repeated regressions in later versions:

    • Verifiable environments first: Tasks must be automatically judged for success or failure inside an isolated container, including pre-provisioned dependencies, network isolation, resource quotas, and timeout circuit-breaking. Tasks without a reliable verifier are not suitable as reinforcement training signals.
    • Layered reward signals: Execution-level (test pass rate), static-level (type checking, static analysis rules, complexity thresholds), and judgment-level (a model scoring readability and maintainability). The lower the layer, the more objective it is, and the higher the weight it should carry.
    • A combination of training paradigms: First use high-quality solutions for supervised fine-tuning to establish a base distribution; then expand correct solutions through rejection sampling and curate hard samples for curriculum-style training; finally use reinforcement learning to optimize long-sequence decisions. The three are complementary, not substitutes.
    • Difficulty stratification: Bucket tasks by their first-attempt pass rate, and concentrate training on samples in the 0.2–0.8 pass-rate range. Both fully-passing and fully-failing samples contribute little to the gradient.
    • Guarding against reward hacking: You must prevent the model from learning to game the tests, such as hardcoding expected return values, special-casing inputs, or tampering with test files. Defenses include hidden test cases, randomized parameters, additional unseen edge cases, and change-scope review of patches.
    • The data flywheel: Online failure samples (compilation errors, rejected reviews, large manual edits by users) that are de-identified and structured and fed back form the incremental data source closest to the real distribution.

    5. Organizing Evaluation Engineering

    Stratify the dataset to avoid representing overall capability with a single difficulty level:

    Level Typical tasks Main focus
    Function-level Complete/rewrite a single function Syntax correctness, edge-case handling
    File-level Add a module, refactor a single file Type system, interface design
    Repository-level Cross-file changes, defect localization Context retrieval, dependency reasoning
    End-to-end Fix a bug, implement a requirement Tool calling, multi-turn convergence

    Sandboxed execution is the trustworthy foundation of all metrics: each candidate solution runs in its own container with CPU/memory/time limits, external network access is forbidden, and the environment is destroyed after the run. Build artifacts and run logs are consolidated into object storage for easy reproduction and auditing.

    Contamination checks: Periodically compare the performance difference of old and new versions on the evaluation set, and keep a batch of “private, undisclosed task sets” as the final arbiter, to avoid the evaluation set itself being covered by training data and producing inflated scores.

    Regression baseline management: Fix the results of each model-version switch into a versioned baseline, and bring the metric definitions, task-set version, k value, and judging scripts all under version control. Otherwise, comparisons across time are not comparable.

    6. Going Live: Continuous Regression and Canary Rollout

    A model upgrade is essentially a change to online behavior and needs to be governed by software-release standards:

    1. Offline regression: The new version runs the full suite on the private task set, and the release is blocked if any core metric regresses beyond a threshold.
    2. Shadow traffic: The old and new versions process real requests in parallel, recording results without adopting them, to surface defects the evaluation set does not cover.
    3. Canary rollout: Open access gradually by user proportion, focusing on adoption rate, the extent of manual edits, and task-abandonment rate.
    4. Observability: Collect request-level metrics through a logging service, and use monitoring alerts with thresholds for sharp drops in pass rate, long-tail latency, and rising error rates.
    5. Feedback loop: Tag and feed back separately the samples of “adopted after large user edits” and “directly rejected,” forming high-value corpus for the next training round.

    In this system, what you typically need is a combination of general-purpose cloud capabilities: a container platform to host the sandbox and orchestrate evaluation tasks, object storage to hold build artifacts and logs, a message queue to decouple tasks and absorb traffic spikes, and a logging service plus monitoring alerts to provide observability. On Baidu AI Cloud, similar evaluation and release pipelines can be built using the same general-purpose components; the key is treating “evaluation” as a first-class engineering discipline that needs version management.

    7. Common Pitfalls

    • Looking only at leaderboards: Public leaderboards differ substantially from real business distributions and carry risks of contamination and overfitting.
    • Looking only at average scores: Ignoring variance means ignoring online jitter, and long-tail tasks are often precisely the samples that hurt users most.
    • Ignoring prompt sensitivity: The same model can differ by more than ten percentage points across different prompt organizations, so comparisons must fix the prompt template and record its version.
    • Ignoring coverage: Testing only mainstream languages and common frameworks overestimates a model’s usability on legacy systems and niche tech stacks.
    • Ignoring the cost dimension: Fix rounds and token consumption directly determine the cost per task, and a slightly stronger capability that doubles cost is not necessarily worthwhile.
    • Treating single-version conclusions as capability conclusions: Fluctuations between versions are the norm, and judgments should be based on multi-version trends and the evolution of the methodology.

    8. Conclusion

    Evaluating the post-training effectiveness of a code LLM is fundamentally answering one question: can this training methodology reliably, predictably, and sustainably improve the model’s performance on real engineering tasks? Only by upgrading evaluation from “scoring” to “stratified task sets + sandboxed execution + variance measurement + versioned baselines + online regression” can subjective impressions be turned into reproducible engineering conclusions. Once the methodology is clearly defined and baked into the pipeline, continuous improvement of model capability along version iterations becomes something expected rather than a matter of chance.