How to Read Coding Benchmark Scores: Understand Benchmarks Before Choosing a Model

1. A leaderboard screenshot cannot support a model-selection decision

When a flagship model was released, a side-by-side comparison table quickly spread across developer communities. On a mainstream repository-level coding benchmark, the new model self-reported 80.3%, while another model previously regarded as the benchmark scored 58.6%. The gap of more than 20 percentage points looked like a generational leap.

Before asking whether the new model is really stronger or whether its inference cost is acceptable, three more basic questions should come first:

• Under what task distribution and grading criteria was the score measured?
• What was the reporting protocol: a single attempt that had to pass on the first try, or multiple attempts with the best result selected?
• Can the score map to your codebase and your team’s development workflow?

A benchmark is essentially an estimate of a capability distribution based on a limited sample. As such, it inevitably carries sampling error, distribution shift, and the risk of targeted optimization or data contamination. A report that provides only a single number without exposing the harness or budget constraints carries almost no decision-making value. A report that discloses the task list, execution traces, and cost limits is far more useful for decisions, even if the score is not high.

2. Four generations of coding evaluation: from writing one function to delivering one fix

First generation: function-level short coding tasks

The earliest coding evaluations were designed for code completion in IDEs. The typical format was: given a function signature and comments, write the function body. Representative benchmarks were early function-level sets, mostly around a hundred tasks, with larger supplementary sets and extensions closer to real dependency usage.

Their limitations are clear:

• The prompts are short and the context is small; the model does not need to understand a codebase or read documentation.
• The benchmark questions are old, and both questions and solutions are likely to have appeared in training data, so contamination is severe.
• Grading usually runs only a few example tests, leaving insufficient coverage of edge cases and exception paths.

For current frontier models, this type of evaluation is basically saturated and offers little discriminative power. Using it to choose a model has almost no reference value.

Second generation: competitive programming and algorithm problems

The second generation moved toward public competitive programming platforms and algorithm problem sets. The focus shifted to algorithm design, logical reasoning, edge-case handling, and complexity control.

This generation can separate models better than the first, but it still remains far from engineering practice:

• The problems are self-contained; they do not depend on an existing codebase and do not require understanding module boundaries.
• Solutions are highly template-driven. Models can improve by practicing similar problems, which has little to do with whether they can work effectively in a real repository.
• Scoring often only checks whether all test cases pass and does not measure maintainability or compatibility with existing code style.

Third generation: repository-level real issue resolution

To move away from exam-style coding, the third generation constructs tasks from real issues and corresponding fix pull requests on open-source code hosting platforms. Given an issue description and a full repository snapshot, the model must produce a patch, and the repository’s own tests determine whether the issue is resolved. The scope expands from “answering one question correctly” to “understanding the codebase → locating the problem → modifying across files → passing verification.”

The original version collected more than 2,000 real issues. A family of variants later emerged:

Variant direction	Scale	Problem addressed
Lightweight subset	Hundreds of tasks	Lower evaluation cost and faster regression checks
Human-validated version	Hundreds of tasks	Filter out unclear descriptions, incorrect tests, and unsolvable tasks
Multimodal version	Hundreds of tasks	Add screenshots, design mockups, and visual error information
Multilingual version	Hundreds of tasks	Avoid testing only a single language ecosystem
Continuously updated version	Rolling growth	Reduce contamination risk
High-difficulty enterprise version	Hundreds of tasks	Longer-horizon, more complex, and closer to engineering tasks

Among these, the human-validated version and the high-difficulty enterprise version are currently the main reference standards for coding capability.

Fourth generation: real workflow-oriented evaluation

The fourth generation is designed for agentic coding. The task is no longer “submit a patch,” but “complete a delivery goal inside a sandbox.” The model is allowed and expected to repeatedly search the codebase, execute commands, read test output, and self-correct. Evaluation is also charged and budgeted by rounds.

Scoring expands from a single test-pass rate to whether the model follows constraints, whether it introduces regressions, and whether it needs to ask a human mid-task.

The main line of evolution is clear: from testing knowledge to testing the ability to complete a task loop under real constraints.

3. What a repository-level score actually measures: dissecting one task

Using the third generation as an example, a task usually has four parts:

task_id           : unique identifier for the task
problem_statement : issue body (may include stack traces, reproduction steps, and expected behavior)
repo_snapshot     : complete repository snapshot before the fix (specified baseline commit)
fail_to_pass      : tests that fail before the fix and must pass after the fix
pass_to_pass      : tests that must pass before and after the fix (regression guard)

The evaluation loop is roughly as follows:

1. Start an isolated container from the baseline snapshot and install dependencies.
2. Give the problem statement (and optional retrieval results) to the model, allowing it to read code and execute commands inside the container.
3. Collect the patch produced by the model and apply it to a fresh container.
4. Run fail_to_pass: only when all tests are green is the task considered correctly located and fixed.
5. Run pass_to_pass: if any test turns from green to red, the patch is judged to introduce a regression and the task also fails.
6. Calculate the resolved rate (number of passed tasks / total number of tasks).

Two facts must be clear:

First, the score measures “the probability of passing hidden tests,” not “code quality.” A patch counts as successful if it makes the specified tests green and does not break regression tests. Whether the naming is reasonable, whether duplicate code is introduced, whether unrelated modules are refactored, or whether tests are “fooled” by commenting out assertions (if assertions are not strictly isolated) are not part of scoring.

Second, the judgment is binary and all-or-nothing. If a patch gets 90% of the semantics right but fails one hidden test, the score is zero. This kind of “steep” scoring function is extremely sensitive to sampling variation. It is also one reason why the same model can differ by several percentage points across different reports.

4. Six common sources of inflated scores

Before treating a number as a basis for selection, check the following items.

1. Data contamination and training overlap

Repository-level tasks come from public code repositories. Fix pull requests, issue discussions, follow-up blog posts, and tutorials may already have entered the training corpus. Even if the official report claims temporal splitting, the “solution pattern” may still have been learned. As a rule of thumb, the older and more popular the tasks are, the less credible the score becomes. This is precisely where continuously updated variants add value.

2. Subset selection and filtering bias

Different reports may use different subsets. A human-validated version is already a cleaner and more solvable set after filtering, so it is naturally easier than an unfiltered full set. Some studies also actively remove tasks that time out or fail during environment installation, further improving the quality of the evaluated set and inflating pass rates. When comparing scores across reports, the first step is to confirm whether the task sets are the same.

3. Prompt information leakage

Stack traces, logs, fix suggestions in maintainer comments, and even guesses pointing to specific function names in the issue body can greatly reduce the difficulty of localization. If the evaluated system can also retrieve the PR content corresponding to the issue, the task degrades into “information recall.” Whether internet access and repository history retrieval are allowed directly changes the meaning of the score.

4. Sampling and retry policies

This is the most easily overlooked factor, yet it has the largest impact:

• pass@1: only one attempt is allowed, which is strict;
• pass@k: k independent attempts are allowed, and the task is considered solved if any one passes; the score rises monotonically with k;
• best-of-n with a verifier to select candidates: the score is higher, but additional judging cost is required;
• unconstrained agentic loops: unlimited rounds of trial and error are allowed, pushing the score even higher.

Therefore, numbers such as 58.6 and 80.3 are not comparable at all if one side uses strict pass@1 while the other side includes retries or a stronger harness.

5. Harness and tool differences

The same model can score more than ten percentage points differently under a different harness. Influencing factors include: retrieval method (full-text search, dependency graph, or directly injecting the full file tree), how much code is placed in context, whether free shell execution is allowed, whether CI logs can be read, whether patch format constraints are strict, and whether the model gets one retry when applying a patch fails. A harness is an amplifier for the model. A strong model can mask harness weaknesses; conversely, a strong harness can mask model weaknesses.

6. Cost and time budget

For long-horizon tasks, a model may consume dozens of calls and hundreds of thousands of tokens to solve one task. If a report does not disclose average turns, average tokens, and wall-clock time, the “pass rate” loses engineering meaning. Pass rate must be viewed together with cost per task.

5. Minimum checklist for reproducible coding evaluation

To make a coding evaluation result usable for decisions, at least the following information must be public:

• Model side: exact model version/snapshot, sampling parameters (temperature, top_p), maximum output length, and the full system prompt;
• Harness side: harness code or pseudocode, available tool list, retrieval strategy, context trimming rules, and patch submission format;
• Budget side: maximum turns, maximum tokens, maximum wall-clock time, and concurrency level;
• Metric side: resolved rate, pass@1, pass@k with the value of k, regression breakage rate (the proportion of tests turning from green to red), average turns, and average tokens;
• Raw artifacts: full per-task execution traces and test logs, allowing manual review of whether the task was truly fixed or the assertions were bypassed.

If these are missing, the reasonable interpretation is: treat the number as an upper-bound claim, not a measurement.

6. What kind of internal evaluation set should enterprises build

Public leaderboards are useful for “eliminating obviously unqualified candidates,” but the final decision must be based on a proprietary codebase. A practical way to build this is as follows.

1. Task construction: automatically pair historical commits

From your own repositories, filter merge requests that are linked to issues or defect tickets and have a moderate amount of code change. Use the parent commit as the baseline snapshot, use the tests added or modified in the merge request as fail_to_pass, and use the tests that were already running before the merge request as pass_to_pass. This allows low-cost automatic task generation at the scale of thousands, followed by manual quality sampling.

2. Layering: design by engineering difficulty rather than language distribution

Level	Task type	What it tests
L1	Single-file bug fixes, exception-branch completion	Localization accuracy and local editing
L2	Cross-module bugs, interface signature changes	Dependency analysis and synchronized call-site updates
L3	Dependency upgrades, build configuration migration	Environment understanding and use of error messages
L4	Local refactoring, performance optimization	Behavior-preserving transformations and regression risk control
L5	Long-horizon tasks: module decomposition, feature delivery	Planning, multi-turn self-correction, and constraint compliance

3. Metrics: go beyond pass rate

resolve_rate         : proportion of tasks where all hidden tests pass
regression_rate      : proportion of tasks where pass_to_pass turns from green to red (lower is better)
avg_turns            : average number of interaction/tool-call turns
cost_per_resolved    : average tokens and cost per resolved task (core cost-effectiveness metric)
human_takeover_rate  : proportion of tasks requiring human intervention to finish

cost_per_resolved and regression_rate are often the real decision points during model selection. Two models may differ by only 2–3 percentage points in resolved rate, but if one has twice the regression rate and twice the unit cost, the engineering choice should exclude that model.

4. Skeleton of the evaluation loop

def evaluate(model, harness, tasks, budget):
    results = []
    for task in tasks:                      # Each task runs in an isolated container
        for attempt in range(budget.max_k):
            env = Sandbox(task.repo_snapshot)   # Rebuild from the baseline snapshot
            patch = harness.run(model, env, task.problem_statement,
                                tools=harness.tools,
                                max_turns=budget.max_turns,
                                max_tokens=budget.max_tokens)
            report = env.apply_and_test(patch,
                                        fail_to_pass=task.fail_to_pass,
                                        pass_to_pass=task.pass_to_pass)
            record = {
                "task": task.id, "attempt": attempt,
                "resolved": report.fail_all_green and report.no_regression,
                "turns": harness.turns, "tokens": harness.tokens,
                "trajectory": env.dump_trace(),     # Must be archived for review
            }
            results.append(record)
            if record["resolved"]:
                break
        # Output: pass@1, pass@k, regression rate, average turns, and cost per resolved task
    return summarize(results)

Key constraints: each task must be rebuilt from a clean baseline snapshot; model and harness versions must all be recorded; traces and logs must be fully archived.

5. Common capabilities required to support this system

The infrastructure requirements for this process are generic and do not require special components. Use object storage to archive repository snapshots, patches, and execution traces. Use a logging service to collect build and test output inside containers for failure attribution. Use a container platform to provide isolated sandboxes and concurrent scheduling, ensuring no side effects between tasks. Use monitoring and alerting to track drift in resolved rate, regression rate, and unit cost, and to detect hidden regressions caused by model or harness upgrades. Use a model serving platform to support batch inference, multi-version canary deployment, and A/B comparisons.

On Baidu AI Cloud, these capabilities can be combined into a pipeline covering task creation, execution, scoring, and dashboards, with evaluation results automatically written back as a time-series metric stream.

7. When leaderboards can be trusted, and when internal testing is required

• Code completion and unit-test generation: older function-level benchmarks are saturated, so leaderboard scores have little meaning. Instead, look directly at completion acceptance rates on your own code and the effectiveness of generated tests.
• Single-file bug fixes and localized refactoring: scores from human-validated repository-level benchmarks have some reference value, but a small-scale A/B calibration on your own set is still needed, with a focus on regression rate.
• Cross-module changes, dependency upgrades, and long-horizon feature delivery: public leaderboards have almost no reference value. Success depends on understanding your proprietary codebase structure, build system, and internal conventions. Only internally built L3–L5 evaluations can provide credible answers.
• Cost-sensitive or high-concurrency scenarios: you must measure per-task token consumption and latency distribution yourself; leaderboards do not contain this information.

8. Conclusion

The model that self-reports 80.3% may indeed be strong, but the number itself is not enough to support any decision. A lead of more than 20 percentage points may come from a real generational capability jump, but it may also come from a looser sampling protocol, a more favorable task subset, a stronger harness, or simply the fact that the issue text leaked the answer. Until these questions are clarified, it can only be regarded as an upper-bound claim.

Three actionable recommendations for practitioners:

1. Ask about the measurement protocol before looking at the score. Confirm the task set, whether it is pass@1 or pass@k, and whether the harness and cost budget are disclosed. Results lacking these three elements should not be compared horizontally.
2. Downgrade “pass rate” to one metric. Put it side by side with regression breakage rate, average turns, and cost per resolved task to form a four-dimensional view.
3. Return the selection decision to your own evaluation set. Use historical commits to generate tasks automatically, layer them by engineering difficulty, and turn the evaluation into a repeatable and observable pipeline. Leaderboards should filter out failing candidates; your own data should make the final choice.

Model iteration is much faster than public benchmark updates. The truly stable capability is not remembering a model’s score, but owning a reproducible measurement method that fits your business.