How to Read Repository-Level Coding Benchmark Scores

A viral 80.3% tells you surprisingly little on its own. A repository-level coding score is not a direct readout of how smart a model is — it is the combined result of the model + the scaffold + the budget + the task distribution. This article moves from how benchmarks evolved across generations, to how a verdict is actually produced, to statistical significance and cost control, and ends with a practical methodology you can use for model selection.

1. Ask three questions before celebrating a high score

Every time a leaderboard goes viral, the useful question is not “how impressive is this,” but these three:

1. Under what conditions was the score measured? How many tool-call rounds were allowed, what token budget, was parallel sampling and retry permitted, and could the model see the full test logs?
2. What was the pass criterion? “The target test turned green,” or “the target test turned green and the existing tests did not regress.” Only the latter counts as a legitimate fix in engineering terms.
3. Where did the tasks come from, and when were they frozen? The relationship between the task publication date and the model’s training cutoff directly determines how much of the score is reasoning and how much is memorization.

Fill in these three points and leaderboard numbers become comparable. Otherwise, two scores from different sources are describing two entirely different things.

2. Four generations of coding benchmarks

Generation 1: Function-level short code tasks

The canonical form is “given a natural-language description and a function signature, complete the implementation,” with problems running from a few lines to a few dozen lines. It maps to inline completion in an IDE and tests single-point syntax plus common algorithmic patterns.

The limitations are clear: short problems, closed context, no need to understand a codebase, no dependency docs, and almost no debugging. After several rounds of public exposure, these benchmarks have essentially lost discriminative power for current models — top models cluster near saturation, the score differences fall inside noise, and the selection value is close to zero.

Generation 2: Competitive programming and algorithm problems

New problems are continuously drawn from contest platforms and algorithm problem sets, testing algorithm design, boundary handling, complexity control, and length of the reasoning chain. This is better than Gen 1 at showing abstraction ability and has some contamination resistance (new problems can keep being added).

Its distortion is a distribution mismatch: in real engineering, writing an elegant dynamic programming solution is a low-frequency activity, while reading three thousand lines of unfamiliar code to locate a race condition is a high-frequency one. Skill at algorithm problems does not extrapolate directly to repository-level work.

Generation 3: Repository-level defect repair

This generation changes the task from “answering a question” to “taking an assignment”: the model gets a real repository snapshot and a defect description, and must locate the problem on its own, make cross-file changes, and pass the verification tests. The verdict is no longer text comparison but executing tests in an isolated environment.

The task set also subdivides along several axes, each addressing a different dimension of trustworthiness:

Sub-track	What it solves
Lightweight subset	Lowers the time and cost of a single evaluation run, enabling fast regression checks
Human-verified subset	Removes ill-specified, broken-test, or genuinely unsolvable problems to improve signal-to-noise
Multilingual subset	Keeps the benchmark from being tied to one language’s ecosystem and toolchain
Multimodal subset	Covers defects reported through UI screenshots, design mockups, or visual error states
Continuously updated subset	Suppresses inflated scores from data contamination by rolling in new problems

The “human-verified subset” and the “complex subset aimed at enterprise engineering tasks” are the two main rulers today: the former for fidelity, the latter for pushing the difficulty ceiling higher.

Generation 4: Realistic workflow-based benchmarks

Designed for agentic coding: the task is no longer “fix it and walk away” but includes requirement clarification, multi-round exploration, cross-repository changes, long-horizon planning, test execution, and iteration driven by failure feedback. This is closest to the real shape of R&D work, and also the hardest to evaluate — because there is no single path, one task has several correct solutions, and scoring has to expand from “did it pass” to “was the process reasonable and was the cost acceptable.”

The common pattern across all four generations: the closer a benchmark gets to real work, the higher the cost per run, and the more sensitive the score becomes to the execution framework. That is exactly why high scores need careful interpretation.

3. How a repository-level score is actually computed

Understanding the judging chain is what lets you judge the value of a score. A standard evaluation run goes roughly like this:

1. Environment fixing: dependencies are restored to the versions pinned by the repository and a reproducible image is built. Dependency drift directly causes false failures — “the code was changed but the tests won’t run.”
2. The model produces a patch: the model reads files, searches symbols, runs tests through its tools, and finally submits a diff.
3. Joint judgment from three kinds of tests:
   • FAIL_TO_PASS: must fail before the fix and must pass after — proves the problem was actually solved;
   • PASS_TO_PASS: must pass both before and after — proves nothing else was broken;
   • new or strengthened verification cases — suppress opportunistic solutions like hardcoding return values to satisfy the test.
4. Judgment applies to the patch only: other changes in the test execution environment are not counted, keeping environmental noise out of the score.

resolved = allFailToPassGreen(patch) and allPassToPassGreen(patch)

Any leniency or strictness at any link in this chain moves the final percentage significantly. This is also why a shift of a dozen points is routine when the same model runs under a different execution framework.

4. Translating 80% into engineering language

4.1 A high single-point score does not equal end-to-end usability

Success on a repository-level task is the串联 result of several sub-steps: accurate localization → sound approach → correct edits → verification passes. If the independent success rates of those steps are 0.92, 0.9, 0.93, and 0.95, the end-to-end rate is only about 0.73. The closer a score gets to saturation, the more this compounding loss dominates — which is why the gap between an 80 and a 60 often shows up in long-horizon stability rather than single-point intelligence.

4.2 Sample size and confidence intervals

At the scale of a few hundred tasks, a score is an estimate with variance, not a precise constant. Report a score together with its interval:

from math import sqrt
def wilson(k, n, z=1.96):
    """Confidence interval for a binomial proportion.
    More stable than the normal approximation at extreme rates."""
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (c - h) / d, (c + h) / d

At 500 tasks with an 80% resolution rate, the 95% interval is roughly ±3.5 percentage points; two models differing by 2 to 3 points usually cannot be declared genuinely different. A twenty-point gap is almost certainly a real difference — provided both ran on the same task distribution, the same judging criteria, and the same budget, otherwise the comparison does not hold at all.

5. The overlooked variables: scaffold and budget

The same model weights can score very differently under different execution frameworks — often by more than the gap between models. The main sensitive variables:

• Localization ability: does the model get structured code navigation such as symbol search, call-chain analysis, and dependency graphs, instead of being left to grep repeatedly?
• Context organization: stuffing everything in versus retrieving on demand makes a large difference; attention dilution from very long contexts can also shift localization off target.
• Tool set and feedback quality: whether the model can run tests directly and read a legible failure stack determines whether it can self-correct.
• Iteration rounds and budget ceiling: allowing more attempts effectively lowers the difficulty.
• Sampling strategy: a single submission and “parallel sampling plus best-of selection” are two different evaluation regimes. The latter measures the ceiling; the former corresponds to the stable behavior you get in real use.

So any regime built on “multi-sample, retry until pass” must be reported separately from pass@1, or it will systematically overestimate the everyday experience.

6. The “token-burning machine” problem: scores must be reported next to cost

High-budget execution frameworks buy high pass rates with large amounts of tokens. If you only look at pass rate, selection conclusions skew toward “the most expensive option,” which is not sustainable for enterprise adoption. The recommendation is a budget-equivalent comparison: put two models under the same token and call-count ceiling, or invert it and compare the cost needed to reach a fixed target resolution rate.

def report(records):
    n = len(records)
    resolved = [r for r in records if r.resolved and not r.regressed]
    lo, hi = wilson(len(resolved), n)
    total_cost = sum(r.cost for r in records)
    return {
        "resolution_rate": f"{len(resolved)/n:.1%} (95%CI {lo:.1%}~{hi:.1%})",
        "regression_rate": f"{sum(r.regressed for r in records)/n:.1%}",
        "cost_per_resolved_task": total_cost / max(len(resolved), 1),
        "avg_iteration_rounds": sum(r.steps for r in records) / n,
        "p95_latency": sorted(r.latency for r in records)[int(n * 0.95)],
    }

Three derived metrics matter more in practice: regression rate (fixes one thing, breaks another), average iteration rounds (cost and experience), and P95 latency (whether it can fit into CI or an interactive flow).

7. Contamination, leakage, and difficulty drift

Public leaderboard scores get inflated by three factors:

1. Corpus contamination: tasks and their fixes entered the training corpus before the model’s cutoff, showing up as an abnormally high share of “one-shot pass with zero iterations.”
2. Test overfitting: behavior alignment against a public test set inflates pass rates on particular problem patterns, which then visibly drop on a private codebase.
3. Difficulty drift: as leading models approach saturation, task sets keep adding harder long-horizon problems, so scores across versions are not directly comparable — a score can rise simply because the problems got harder.

The direct implication for consumers: public leaderboards are suitable for initial screening, not for the final decision. The final decision has to rest on an evaluation set drawn from the same distribution as your own codebase.

8. Building a high-fidelity evaluation: a reproducible recipe

8.1 Sampling

Sample 80 to 150 tasks from your own repositories and ticket system, stratified across:

• single-file small fixes (wrong logic, missing boundary cases)
• moderate cross-module changes (interface adjustments plus caller updates)
• localization tasks (symptoms only, the model must find the cause)
• dependency and build problems (version conflicts, configuration errors)
• test-writing tasks (add regression coverage for an existing defect)

One key constraint: keep a held-out set that is never published, used for periodic acceptance, so the evaluation set itself does not get “optimized.”

8.2 Automated judging

Each task fixes three things: the repository snapshot identifier, the problem description, and the set of verification tests. Judging must run in an isolated container and share the same images and dependency locks as the pipeline, so local evaluation and CI results agree.

def judge(task, patch):
    env = prepare_env(task.repo_snapshot)        # locked deps, reproducible results
    if not env.apply(patch):
        return False, False
    target_ok = env.run_tests(task.fail_to_pass)  # all must turn green
    no_break = env.run_tests(task.pass_to_pass)   # all must stay green
    return target_ok and no_break, (target_ok and not no_break)

8.3 Reading the results: four quadrants, not one number

Splitting results into four classes carries far more information than a single percentage:

• Fully resolved: all target tests green with no regressions — safe to adopt;
• Partially resolved: some target tests pass — usually a misunderstanding of the requirement, improvable with better context;
• Unresolved with zero regressions: safe but useless — common with conservative models; consider a larger budget or better tool feedback;
• Introduced regressions: highest risk — this determines whether the model can be enabled in stages that lack human review.

For selection, the regression rate often decides more than the resolution rate whether a model can enter the production flow.

8.4 Evaluation infrastructure

To keep evaluation sustainable, the pipeline itself has to be engineered: version the task sets and repository snapshots in object storage, run execution on a container platform for isolation and elastic scaling, load structured results into a database, collect failed-sample logs centrally for root-cause analysis, and attach monitoring, alerting, and trend dashboards for resolution rate, regression rate, cost, and latency. This kind of capability can be assembled from general-purpose cloud infrastructure — on Baidu AI Cloud, for example, the standard object storage, container platform, log service, and monitoring and alerting offerings cover it — without binding yourself to any particular framework.

9. Checklist

For any coding benchmark score you are handed, check it line by line:

• Does it state the budget ceiling (rounds, tokens, latency)?
• Single submission, or multi-sample best-of selection?
• Does the pass criterion include “no regressions”?
• What is the task set size, language distribution, and was it human-verified?
• Where does the task freeze date sit relative to the model’s training cutoff?
• Is a confidence interval given rather than a single number?
• Are cost metrics reported alongside?
• Are scaffold details published or a reproducible execution environment provided?
• Was the comparison baseline measured under exactly the same conditions?
• Are there validation results on a private codebase?

10. Conclusion

An 80-point result can legitimately say that this generation of models has crossed the practicality threshold for repository-level defect repair. On its own, it cannot say “you should pick it.” A score is the joint product of capability, framework, budget, and task distribution; when any one of those is opaque, the number’s comparability weakens.

The pragmatic posture: use public leaderboards to eliminate clearly unqualified candidates, use an in-house, same-distribution evaluation set for the final ranking, and bring resolution rate, regression rate, cost per task, and P95 latency into the decision together. Models iterate far faster than evaluation methods mature. Keeping your evaluation regime stable, judging automated, and the held-out set secret is the foundation for correct long-term technology selection.