How to Read AI Coding Benchmarks: What a Doubling in Long-Horizon Pass Rate Actually Means
A recent AI coding benchmark aimed at mobile application development published its 2.0 leaderboard, adding results from several new models. This article walks through what the numbers mean: how to interpret the four core metrics (pass rate, average task completion, single-run latency, and average cost), why a jump from 12.0% to 24.0% end-to-end pass rate matters more than it first appears, and where a lightweight low-cost model wins on cost-efficiency. It then explains why long-horizon coding agents are hard to build and to evaluate, and lays out the key pieces of an in-house evaluation system, from task-set design and reproducible sandboxes to multi-dimensional acceptance, cost guardrails, and pipeline integration, so teams can turn leaderboard capability into measurable engineering productivity.
A recent AI coding benchmark for mobile application development released its 2.0 leaderboard, adding results from several newly evaluated models. On the long-horizon task leaderboard, the top model reached an end-to-end pass rate of 40.0%. One model improved its pass rate from 12.0% in the previous generation to 24.0% — a doubling — with an average task completion of 82.3%. Its single-run time dropped from 66.2 hours to 27.4 hours, and its average cost fell from $418.3 to $317.4. Another model, positioned as lightweight and low-cost, achieved a 17.3% pass rate, a 12.0-hour single-run time, and a cost of just $9.9.
The specific products are not the point of this article. What these numbers invite us to discuss is how to read leaderboard metrics, what makes long-horizon agent tasks difficult, and how enterprises can convert “leaderboard capability” into “R&D efficiency.”
1. First, Read the Leaderboard: What Four Categories of Metrics Each Answer
Judging by the metric composition in the 2.0 release, this leaderboard measures at least four dimensions at once:
- Task pass rate: end-to-end success rate. A task counts as “completed” only when the final artifact passes full acceptance. This typically corresponds to the probability of success on a single run, i.e., pass@1.
- Average task completion: partial credit. For tasks that don’t fully pass, key completed steps are weighted and scored, reflecting “how far from success” the task is.
- Single-run end-to-end latency: the time it takes for a full task set to run from start to finish, which determines whether a round of evaluation can finish within a day or even a single shift.
- Average cost: the average spend to complete a full round of evaluation, driven mainly by the input and output token consumption of model calls, plus the extra overhead from tool calls and retries.
These four categories of metrics answer four different questions:
- Pass rate answers “can it deliver a usable result”;
- Completion answers “how far is it from the goal”;
- Latency answers “how long do I have to wait”;
- Cost answers “how much does it cost.”
An example illustrates the difference between the first two. In one task, the model generated correct code that compiled, but it didn’t handle the edge case of an empty input, so the unit tests still failed. Under strict acceptance criteria, it fails; under completion scoring, it can earn a fairly high score.
Also note that this leaderboard measures long-horizon tasks: a single task often requires dozens of tool-call rounds and takes hours. The more rounds there are, the longer the context, the larger the accumulated deviation, and the more pronounced the variance in cost and latency. An advantage on short tasks does not necessarily carry over to long ones.
2. Beyond “Doubling”: The Real Progress Behind the Pass-Rate Gain
Breaking down the data, this generational update delivers at least four benefits.
1. Success rate multiplies. The pass rate rose from 12.0% to 24.0%, doubling. At a sample size of roughly 20-plus tasks, 12.0% corresponds to about 2–3 tasks passing, and 24.0% corresponds to around 5. In other words, the number of tasks that can be completed end-to-end roughly doubled. For workflows that rely on large volumes of repetitive changes, this gain shows up directly in delivery throughput.
2. Failure cycles shrink dramatically. Single-run latency dropped from 66.2 hours to 27.4 hours, a reduction of about 58.6%. Measuring output per unit time as “passed tasks / single-run latency”:
- The previous generation was roughly 0.045 tasks/hour;
- The new version is roughly 0.175 tasks/hour, about a 3.8x improvement.
The more practical significance lies in feedback speed. Hour-scale runs can be batched overnight or over a weekend, whereas tens-of-hours runs make it hard to close the “fail — fix — rerun” loop within a single workday.
3. Overall cost declines. Single-run cost fell from $418.3 to $317.4, a drop of about 24.1%. More important is the cost per unit of output. Measured as “single-run cost / passed tasks,” the previous generation was about $139.4 per task, and the new version is about $52.9 per task, a decline of about 62.1%. This is the most easily overlooked bonus value of “doubling the pass rate.”
4. A gap remains between completion and pass rate. The new version’s average task completion is 82.3%, while its pass rate is 24.0% — a difference of about 58 percentage points. The model can walk through most of the key steps on the majority of tasks, yet a large number of tasks still fail at the final stage. This kind of “long-tail failure” usually comes from edge cases, cross-module dependencies, implicit constraints, or differences in test environments. It also shows that today’s long-horizon coding agents are closer to a “highly efficient senior assistant” than to a fully hands-off producer.
3. The Cost and Efficiency Dimension: Another Answer on the Leaderboard
On the cost-efficiency dimension, a lightweight low-cost model offers a different answer:
| Model Type | Pass Rate | Single-Run Latency | Single-Run Cost | Equivalent Passed Tasks | Cost per Task |
|---|---|---|---|---|---|
| Top model | 40.0% | — | — | 10.0 | — |
| Long-horizon workhorse (new) | 24.0% | 27.4 hours | $317.4 | 6.0 | ~$52.9 |
| Lightweight low-cost model | 17.3% | 12.0 hours | $9.9 | 4.3 | ~$2.3 |
| Long-horizon workhorse (previous) | 12.0% | 66.2 hours | $418.3 | 3.0 | ~$139.4 |
Note: “Equivalent passed tasks” is estimated on a 25-task set; the single-run latency and cost for the top model and the lightweight model are not given in the leaderboard and are shown as “—.”
The comparison shows that the lightweight model’s cost per task is about 1/23 of the new long-horizon model and about 1/60 of the previous generation. Its pass rate is about 6.7 percentage points lower than the new version, but it gains roughly an 18x cost advantage and more than a 2x speed advantage. For tasks with clear change boundaries, well-defined specs, and batch-distributable work, this combination is often more cost-effective than chasing the highest pass rate alone. Selection should therefore be decided per scenario, rather than by leaderboard rank alone.
4. Why Long-Horizon Coding Agents Are Hard to Build and Hard to Test
The difficulties of long-horizon tasks concentrate in four areas.
1. Error accumulation over long chains. A slight misreading of requirements gets amplified at every subsequent edit, and the final artifact may be locally correct but deviate from the goal overall. In addition, a single generation is inherently stochastic, so the pass rate of the same task varies across runs, and a single result is not enough to represent the true level.
2. Context and memory management. In a long task, the codebase, logs, errors, and conversation keep accumulating, and even the largest context window has a limit. Which information is retained, which is compressed, and which is written to external files for later retrieval directly determines whether the model still remembers its original goal after dozens of rounds.
3. Tool use and environment dependencies. The agent must repeatedly build, run tests, view logs, and so on. Whether the format of tool return values is stable, whether environment dependencies can be reproduced, and whether permission boundaries are clear all affect the success rate and cost.
4. Result verification. A model claiming “fixed” is not the same as the problem being truly resolved; it takes independent testing and inspection mechanisms to decide. This is also the most underestimated part of evaluation.
5. From Leaderboard to Team: Key Pieces of an In-House Evaluation System
A leaderboard only reflects performance on a specific task set. Enterprises need an internal evaluation system that stays close to their own codebase and can run continuously. The following areas are worth focusing on.
1. Task-set design. Tasks should come from real to-do items, bug records, and refactoring needs, stratified by difficulty: single-file fixes, cross-module changes, legacy code maintenance, and architecture-level adjustments. The sample size must be large enough that small-sample variance doesn’t mask real differences. It’s advisable to keep a portion of private tasks and rotate them periodically, reducing the risk of overlap between evaluation data and training data.
2. Isolated and reproducible execution environments. Each task runs in an independent container with pinned dependency versions; the codebase is restored before and after execution via snapshots or version control; only the minimum permissions the task needs are granted; and caps are set for per-step latency, total cost, and total rounds.
3. Multi-dimensional acceptance. The verdict should include at least the following combination:
- Build or compile;
- Unit tests and integration tests;
- Static checks;
- Regression tests written specifically for that task.
High-risk tasks can add human review.
4. Metrics system. Beyond pass rate, also record average rounds, end-to-end latency, token consumption, converted cost, distribution of failure reasons, and the amount of manual rework.
5. Statistics and observability. Sample each task multiple times and use pass@k to assess potential; record the trajectory, cost, and artifacts of each run in a structured way to enable after-the-fact attribution; and codify the statistics logic into scripts to avoid manual calculation errors.
The following sketch code shows a skeleton for task-level execution and result aggregation:
import jsonimport mathfrom dataclasses import dataclass@dataclassclass TaskResult:task_id: strpassed: boolsteps: int # number of tool-call roundscost_usd: float # cost consumed by this task runfailure_stage: str # setup/build/test/timeout/review; empty when passeddef run_task(task: dict, agent, sandbox) -> TaskResult:"""Run a single evaluation task in an isolated sandbox and return a structured result."""env = sandbox.create(image=task["image"], limits=task["limits"])try:repo = task["repo_factory"].make() # create an independent working copy per task via version controlenv.upload(repo.path, "/workspace/repo")env.exec(task["setup_command"]) # install dependencies, prepare test datasteps = 0for step in range(task["max_steps"]):action = agent.propose(task["prompt"], env.snapshot("/workspace/repo"))if action.kind == "finish":breakresult = env.exec(action.command, cwd="/workspace/repo",timeout=task["step_timeout"])agent.observe(result)steps += 1checks = [env.exec(cmd, cwd="/workspace/repo").exit_code == 0for cmd in task["verify_commands"]] # build, test, static checkspassed = all(checks)return TaskResult(task["id"], passed, steps, env.usage.cost_usd, "")except sandbox.StepTimeout:return TaskResult(task["id"], False, steps, env.usage.cost_usd, "timeout")finally:env.destroy()def summarize(results: list[TaskResult]) -> dict:"""Summarize the pass rate, cost, and failure distribution of a single run, with a pass@k estimate."""n = len(results)c = sum(r.passed for r in results)return {"tasks": n,"pass@1": round(c / n, 4),"avg_steps": round(sum(r.steps for r in results) / n, 2),"total_cost_usd": round(sum(r.cost_usd for r in results), 2),# if the task set is large enough and can be treated as independent sampling,# use the combinatorial unbiased estimator for pass@k"pass@k": round(1 - math.comb(n - c, k) / math.comb(n, k), 4) if n - c >= k else 1.0,"failure_dist": {}, # count failures by failure_stage to pinpoint weak spots}
6. Closed-loop iteration. After every model change, prompt tweak, or toolchain improvement, first run a regression on a small task set to confirm the changes in pass rate and cost, then gradually widen the scope. The evaluation set itself should be version-controlled just like test cases.
6. Enterprise Adoption: Making Evaluation Results Serve the R&D Process
Route tasks by type. Tasks with clear requirements, localized changes, and high repetition go to a low-cost model for batch processing; cross-module, high-risk, or long-horizon tasks go to a more capable model, paired with human review.
Bring evaluation into the R&D pipeline. Add automated regression checks to the CI process, and uniformly run build, test, and static checks on agent-generated changes before they move to human code review. This is what distinguishes “the model claims it’s done” from “it’s actually done in engineering terms.”
Establish cost guardrails. Set invocation budgets per project or team, trigger alerts and downgrade to a more economical model when thresholds are hit, and reuse historical changes and test cases for similar tasks to reduce repeated consumption.
Build evaluation infrastructure. Taking general-purpose technology components as an example:
- Use a container platform to provide isolated environments for each task;
- Use object storage to archive evaluation task sets, run trajectories, and artifacts;
- Use a message queue to schedule large-scale concurrent evaluations;
- Use logging and monitoring/alerting services to track failure rates and cost anomalies.
At the infrastructure level, general-purpose product capabilities from Baidu AI Cloud — such as containers, object storage, message queues, logging, and monitoring/alerting — can provide the underlying support for this kind of evaluation and agent-execution platform.
Connect evaluation metrics to engineering metrics. Beyond pass rate, also track metrics such as review feedback on agent-generated code, rework rate, and defect escape rate. Only when these metrics move in line with the leaderboard trends does a model upgrade truly translate into R&D efficiency.
7. Conclusion
The signal from the new leaderboard is clear: the pass rates of long-horizon coding agents are rising fast, and the cost per unit of output is falling significantly. Yet even the top-ranked model still fails strict acceptance on about 60% of tasks. The point of evaluation is not to memorize rankings, but to understand what the metrics mean behind them and to make judgments suited to your own situation using a task set close to your own business. Only by getting isolated execution, multi-dimensional acceptance, cost guardrails, and pipeline integration right can coding agents truly become a measurable, governable, and sustainable form of productivity.
