Stability-First Evaluation Framework for Code Model Post-Training

    Post-training for code models should be evaluated by distribution of outcomes, reproducibility, and controllability, not by peak performance. This article outlines a practical evaluation framework covering full training pipelines, multidimensional metrics, validation isolation, release gates, and failure-driven iteration to build reliable engineering processes.

    In code generation scenarios, post-training success is not determined by whether a model occasionally produces impressive snippets, but by whether it can consistently deliver compilable, testable, and maintainable results across repetitive tasks. Many teams are easily attracted by single demonstrations or public leaderboards during model selection, only to discover after deployment that the model performs inconsistently on similar requests, struggles with long contexts, or exhibits unstable tool invocation. For developers and enterprise users, the more valuable question is: has post-training evolved into a repeatable methodology, rather than relying on luck with individual samples?

    Shift the Evaluation Target from “Peak” to “Distribution”

    The cost of failure in code generation is typically higher than in casual conversation. Incorrect interfaces, missed edge cases, or silently altered behavior can propagate into CI/CD pipelines and production environments. Therefore, evaluation should focus on the distribution of outcomes, not the single best output.

    It is recommended to observe at least three categories of phenomena. The first is consistency across multiple samples of the same problem: with fixed temperature and sampling strategy, is the pass rate concentrated, rather than occasionally achieving full marks? The second is transfer to neighboring tasks: when function names, dependency versions, or constraints are slightly modified, does quality degrade sharply? The third is regression: while new post-training rounds improve performance on new problems, do old problems, legacy repository styles, and previously fixed defects regress? Reporting only average scores can easily obscure long-tail failures.

    Stability can also be broken down into reproducibility and controllability. Reproducibility means that under identical inputs, tool environments, and decoding configurations, key conclusions do not drift arbitrarily. Controllability means that through system prompts, test constraints, and tool permissions, the model can be confined within expected behavior. Models lacking both attributes are difficult to integrate into development workflows that require auditing.

    What a Complete Post-Training Pipeline Should Include

    The industry generally understands code post-training as using supervised data, preference data, and reinforcement learning or similar policy optimization based on execution feedback to steer the model toward “verifiably correct behavior.” A genuinely complete pipeline includes at least data, rewards, validation, training, and release gates. Omitting any of these components likely results in superficial improvements only.

    On the data side, distinguish between synthetic problems, real repository tasks, and failure sample replay. Synthetic problems are suitable for covering algorithms and API specifications, but an overly narrow distribution can make the model adept only at benchmark-style tasks. Real tasks should preserve dependencies, build systems, and directory constraints; otherwise, the model learns isolated functions rather than engineering modifications. Failed samples must be traceable to specific error types, such as compilation failures, test timeouts, or modifications to irrelevant files, rather than just being stored with a generic “failed” label.

    Rewards should not rely solely on whether final tests pass. A single sparse reward can encourage opportunistic behavior: deleting tests, relaxing assertions, hardcoding expected outputs, or introducing hidden issues in parts not covered by checkers. A more robust approach is to decompose the reward into interpretable components: build success, public test pass rate, hidden test pass rate, whether the diff is confined to permitted paths, and whether dangerous operations are introduced. The weights of these components should be set according to task risk, with manual spot-checking retained to prevent reward hacking.

    Validation must be executable. Static scoring can only identify a subset of issues; correctness ultimately depends on execution. Evaluation should run in an isolated environment: fixed base images, locked dependencies, minimal network and file system permissions, and clear timeout and resource quotas. Public tests for the same problem should be used for training feedback, while hidden tests are used for acceptance. These two sets must not be long-term mixed; otherwise, the model will memorize answers rather than learn to make modifications.

    The illustrative workflow below is pseudocode independent of any specific product, emphasizing phases rather than interfaces:

    1. for task in curated_tasks:
    2. patch = model.edit(task.repo, task.instruction)
    3. result = sandbox.run(build, public_tests, limits)
    4. reward = score(result, scope_check(patch), safety_checks)
    5. buffer.add(task, patch, reward, traces)
    6. policy_update(buffer)
    7. gate = replay(hidden_tests, old_regression_suite)
    8. release_if(gate.pass_rate and not gate.critical_regress)

    The key to this workflow is not the algorithm name, but whether feedback loops back into training and whether an independent gate exists before release. If optimization only scores on an offline set, cannot reproduce experiments, and cannot block regressions, it does not constitute full mastery of the methodology.

    Replace Single Pass Rate with Multidimensional Metrics

    The pass rate remains important, but should be interpreted with the following constraints added.

    • Difficulty stratification: Simple completions, cross-file modifications, and defects that require analyzing failure logs for localization should be statistically reported separately. An increase in average pass rate may result solely from easier problems.
    • Attempt budget: The number of rounds allowed for the model to read logs and revise must be fixed. Unlimited retries can turn “can actually fix” into “got lucky by guessing.”
    • Modification scope: Track statistics on changed lines, number of files touched, and whether prohibited directories are accessed. Patches that pass tests but involve large-scale rewrites are often unacceptable.
    • Tool behavior: Evaluate whether the sequence of retrieval, execution, and log reading is reasonable, and whether failure information is used effectively rather than repeatedly issuing the same command.
    • Style and contracts: Check whether naming, error handling, comments, and interface compatibility align with existing repository conventions. Patches that are correct but break compatibility are usually a net loss in enterprise repositories.
    • Security baseline: Flag whether the model suggests disabling validations, writing secrets, expanding privileges, or executing unverified scripts. Such issues should be treated as vetoes rather than averaged away by pass rates.

    When reporting results, provide confidence intervals or at least sample sizes, and specify decoding configurations, tool permissions, and timeouts. Comparisons without these conditions lack engineering relevance.

    How Training and Evaluation Can Avoid Self-Deception

    The most common bias in post-training occurs when the training distribution and acceptance distribution gradually converge to the same set of problems. Mitigation approaches include: splitting data by time or repository so that acceptance sets are invisible during training; periodically rotating hidden tests and dependency versions; incorporating human-written adversarial samples specifically covering edge cases and misleading error reports; and manually reviewing high-reward samples to exclude opportunistic patches.

    Another source of bias stems from the evaluator itself. If another model is used to score patches, its consistency with actual execution results must be spot-checked, and it should not have access to full reference answers. When the executor fails, record whether the cause is code error, environment error, or timeout, so infrastructure failures are not mistaken for model degradation or improvement.

    Logs and artifacts must be traceable: task identifiers, model versions, data batch IDs, reward configurations, container image digests, and test outputs should all be preserved. Object storage is suitable for storing patches and logs, container platforms for isolated execution, and monitoring and alerting for detecting sudden pass-rate drops, rising timeouts, and unusual commands. These are general-purpose capabilities that do not depend on any single vendor’s proprietary APIs. Teams using environments such as Baidu AI Cloud should also formalize images, permissions, and quotas as part of the evaluation specification, rather than recording only model scores.

    Release Gates and Continuous Improvement

    It is recommended to split releases into candidate, canary, and full phases. Candidate versions must pass hidden test sets and historical regression suites, with zero failures on critical safety items. During canary deployment, only low-risk repositories or assistive coding scenarios should be exposed, collecting data on human adoption rate, rollback rate, and human-editing distance. Before full release, re-examine long-tail cases: whether very large repositories, niche languages, or strictly compliant directories remain unusable. Unusability is acceptable, but boundaries must be documented so users do not misjudge capability.

    Continuous improvement relies on failure bucketing, not blanket statements like “train another round.” Samples should be categorized by compilation errors, logic errors, over-modification, tool misuse, and context truncation, then used to decide whether to supplement data, adjust rewards, or improve retrieval. If a particular error category only decreases after increasing sampling count for two consecutive rounds, it typically means the approach is not addressing the root cause, and simply adding more effort should be stopped.

    The outlook depends on whether this pipeline can be executed repeatedly: whether data can be updated, rewards audited, experiments reproduced, and regressions blocked by gates. With these conditions in place, code capability improvement can shift from occasional peaks to a manageable engineering process. When selecting models, prioritize asking whether evaluation is isolated, regression is independent, and failures are traceable. If only demo cases and a single score are provided, that should be treated as insufficient evidence.