How to Evaluate Code Post-Training Quality in Large Language Models
Single-pass demos do not prove strong code post-training. This article explains how to judge whether a model’s code post-training actually closed a verifiable loop: what signals are optimized, how to measure coverage and sampling stability, how to run isolated evaluation, and which signals indicate a transferable, reproducible method.
How to Evaluate Code Post-Training Quality in Large Language Models
Why “it works once” is not the same as high quality
A common community evaluation is to run a few problems and check whether the model produces runnable code on the first try. That samples one point in the capability distribution and then infers the quality of the entire post-training pipeline.
A stricter view is that code post-training quality is not whether a single sample succeeds. It is whether the same model, under multiple independent samples, consistently lands near a correct solution, and whether that stability is reproducible, attributable, and improvable. Several recent coding tools look similar in a one-shot demo. When you increase sampling to 5 or 10 attempts and require every attempt to be correct, the gap opens quickly. The former often reflects accidental alignment from the base model; the latter indicates that post-training actually injected a verifiable-correctness signal.
Evaluation should therefore shift from “who is stronger” to three engineering questions: what signal does post-training optimize, how is that signal measured, and is the method transferable and reproducible?
What the post-training pipeline is optimizing
Code is one of the few modalities with natural verifiability: whether a patch is correct can be judged nearly objectively from compile results and the repository test suite. Mature pipelines usually close the loop around that property:
- Verifiable reward construction: use unit-test pass rate, static checks, and build success as rewards, not only human scores or model-vs-model judging.
- Execution feedback: feed compile errors, stack traces, and failed assertions back as context so the model learns “read the error, then fix the code.”
- Difficulty curriculum and rejection sampling: schedule tasks by solvability, fine-tune on solutions that pass under multiple samples, and raise hit rate in the hard band.
- Tool use and multi-turn alignment: train correct use of retrieval, terminals, and file edits, and keep intent stable across turns.
Whether a team has “mastered the method” is not a single score. It is whether these four stages form a data–verification–evaluation loop, and whether each change can be validated by ablation. Stacking data or swapping the base model alone often raises the mean while variance stays high; that kind of gain is not sustainable.
Evaluation dimension 1: capability coverage
A single score is easy to overfit. Cover at least these six task types, with enough samples in each:
- Code generation and completion: function-level and file-level, including boundary cases and exception paths.
- Bug fixing: given failing tests and a stack trace, locate the issue and apply a minimal fix without breaking existing tests.
- Refactoring and migration: cross-file rename, interface extraction, framework or language upgrades.
- Test generation: assertions that actually catch defects, not just line coverage.
- Long-context, repository-level understanding: dependency analysis in contexts of hundreds of thousands of tokens.
- Engineering constraints: follow existing style, dependency allowlists, and files that must not be modified.
Bug fixing is the strongest probe of post-training depth: it requires localization, change, and regression together, and cannot be gamed by memorizing problems.
Evaluation dimension 2: stability and reproducibility
This is the divide between demo-grade and production-grade capability. Fix experimental conditions before comparing: temperature and top_p, no extra randomization, model version and prompt-template hash, and n independent samples per case.
Common metrics:
| Metric | Definition | What it reveals |
|---|---|---|
| pass@1 | Pass rate of a single sample | Average capability |
| pass@k | At least one pass in k samples | Ceiling and solvability |
| pass^k | All k samples pass | Stability and variance |
| Parseable-format rate | Share of outputs the toolchain can parse | Instruction following and format drift |
| Average turns / rework rate | Interactions needed to reach the goal | First-try success |
| Timeout and hallucinated-call rate | Timeouts and nonexistent APIs | Factuality and engineering discipline |
A typical inflated signal is high pass@1 with rapidly decaying pass^k. The model can solve problems but sampling variance is large, which produces unpredictable failures in CI and limited engineering value.
Aggregation sketch:
import mathdef pass_at_k(n: int, c: int, k: int) -> float:"""Unbiased pass@k estimate given n samples and c passes"""if n - c < k:return 1.0return 1.0 - math.comb(n - c, k) / math.comb(n, k)def stability(rows: dict[str, list[bool]], k: int = 5) -> dict[str, float]:"""rows: {case_id: [pass, pass, ...]}; returns pass^k per case"""return {cid: pass_at_k(len(v), sum(v), k) for cid, v in rows.items()}def summarize(rows: dict[str, list[bool]]):total = sum(len(v) for v in rows.values())passed = sum(sum(v) for v in rows.values())per_case = stability(rows, k=5)return {"pass@1": passed / total,"pass^5_mean": sum(per_case.values()) / len(per_case),"all_pass_ratio": sum(1 for v in per_case.values() if v == 1.0) / len(per_case),}
Evaluation engineering: sandbox, scoring, and contamination control
Evaluation is itself an engineering system. Trust depends on a controlled execution environment:
- Isolated execution: run each case in its own container, disable outbound network access, cap CPU, memory, and wall time so arbitrary model output cannot affect the host.
- Pinned dependencies: lock language versions and dependency lists so environment drift does not break comparability.
- Dual scoring: public tests for fast feedback, hidden tests for final scoring; also require build success and static checks.
- Diff constraints: patches must not edit test files, add skip markers, or change dependency declarations, so “passing tests” is not confused with “solving the problem.”
- Anti-contamination: build a private set from internal repos, recent merge requests, and real defect tickets; prefer samples after the model’s training cutoff and rotate items regularly.
Evaluation skeleton:
def evaluate(cases, model_client, n_samples=5, timeout=120):results = {}for case in cases:outcomes = []for _ in range(n_samples):patch = model_client.generate(case.prompt, case.repo_snapshot)if not parse_patch(patch):outcomes.append(False)continuereport = run_in_sandbox(case.repo_id, patch,build_cmd=case.build_cmd,public_tests=case.public_tests,hidden_tests=case.hidden_tests,timeout=timeout,)outcomes.append(report.build_ok and report.hidden_passed)results[case.id] = outcomesreturn summarize(results)
Four signals that the method has closed the loop
- Gains move together: generation, repair, and test writing improve in the same direction, without large capability trade-offs.
- Variance falls with capability: pass^k improves with pass@1, meaning training is shrinking the solution space rather than increasing lucky search.
- The recipe transfers: the same method yields measurable gains on a smaller or older base model, so the lift is from the method, not only base-model headroom.
- Experiments reproduce: with randomness fixed, retraining and re-evaluation stay within a controlled band, and ablations attribute each component’s contribution.
A practical validation path for enterprises
Model selection is only the start. Reliable delivery depends on engineering around the model:
- Three-stage validation: offline benchmark filter → shadow mode that does not affect production → limited rollout with rollback thresholds.
- Online metric loop: first-pass rate, build success, human rework, mean time to resolution, and regression defects in accepted code.
- Observability: log prompt versions, context assembly, tool-call chains, and failure reasons. Logging, metrics, alerting, containers, and object storage can carry collection and tracing.
- Ongoing context engineering: a stronger model does not automatically fix missing context; retrieval quality, dependency analysis, and prompt-template governance still need investment.
Conclusion
Evaluating code post-training is evaluating whether a verifiable loop is closed: objective signals, stable measurement, reproducible conclusions, and attributable gains. When evaluation moves from “a few examples run” to “repeated sampling in an isolated sandbox with variance-controlled numbers,” model capability becomes predictable engineering throughput—and you can tell a post-training path worth continued investment from a demo that is hard to reproduce.
