Coding Agent Benchmarks: How to Weigh 24% vs 17% Pass Rates

    A public long-horizon coding agent benchmark compares two agent shells — one at a 24.0% pass rate (82.3% average completion, 27.4 hours per round, $317.4 per run) and one at 17.3% (71.2%, 12.0 hours, $9.9). This article explains how to read the four key metrics (pass rate, completion, elapsed time, and cost), why a 6.7-point pass-rate gap is not statistically decisive on a 30-task sample, and how to turn benchmark numbers into engineering decisions — from cost-per-pass metrics and tiered routing to budget caps, acceptance scripts, and a repeatable in-house evaluation pipeline.

    From “Writing One Correct Function” to “Finishing an Engineering Project”

    Early coding agent evaluations focused mostly on whether a single generated function was correct. As agents became capable of running multi-step loops — read code, modify files, run commands, inspect errors, and iterate — the evaluation target shifted from isolated code generation toward the ability to complete long-horizon engineering tasks. Mobile app development is a typical scenario for testing this ability, for three reasons:

    • Long task chains. Creating a runnable app, building a complete feature, or performing a code migration usually involves project scaffolding, dependency and permission configuration, coordination across multiple modules, and repeated compile, install, and test cycles.
    • Highly dispersed context. Build scripts, resource files, manifest configuration, dependencies from multiple sources, and runtime logs all affect the outcome. The model must rely on retrieval and reasoning to locate problems; a single code snippet is not enough.
    • Verification depends on a complete environment. Judging whether a task is truly complete requires an isolated runtime environment and automated acceptance checks.

    For this reason, a public long-horizon benchmark for coding agents released by a mobile OS developer team is valuable not just for its rankings, but because it presents four dimensions at once — pass rate, completion, per-round elapsed time, and cost per run — offering a reference sample for how to trade off capability, speed, and cost.

    Reading the Scores: Two Routes Compared

    The public results show a sharp contrast between two routes:

    Metric Route A (vendor self-developed shell) Route B (third-party shell)
    Agent shell under evaluation Vendor’s own coding agent Another agent shell
    Pass rate 24.0% 17.3%
    Average completion 82.3% 71.2%
    Per-round benchmark total time 27.4 hours 12.0 hours
    Cost per benchmark run $317.4 $9.9

    Note on definitions: This article treats 27.4 hours and 12.0 hours as the “total elapsed time of a single benchmark round” — the time to run all 30 tasks in full. Under this reading, the average time per task is roughly:

    • Route A: 27.4 ÷ 30 ≈ 0.91 hours/task;
    • Route B: 12.0 ÷ 30 ≈ 0.40 hours/task.

    If the original leaderboard instead defines 27.4 hours and 12.0 hours as “average time per task,” you cannot divide the per-run cost directly by hours, and cost efficiency must be recomputed. This article uses the “single-round total time” definition throughout.

    Note on sources: The original input did not provide the benchmark name, release date, or source link. This article refers to it as a “mobile development long-horizon benchmark” and calls the two shells “vendor self-developed shell A” and “third-party shell B.” When citing formally, readers should return to the original release page to verify the benchmark name, publishing entity, data units, and shell names.

    Two quick conversions:

    • Cost ratio: 317.4 ÷ 9.9 ≈ 32.1, so Route A costs about 32 times as much per run as Route B.
    • Cost per task: If “single run” means one complete benchmark round, Route A is about 317.4 ÷ 30 ≈ $10.6 per task, and Route B is about 9.9 ÷ 30 ≈ $0.33 per task.

    In the public results, the best-performing route reached a 28.7% pass rate. Between the two routes compared here, Route A’s pass rate is higher than Route B’s; Route A trails the leader by 4.7 percentage points. With a small sample of 30 tasks, this gap may be no more than random variation.

    These numbers do not simply show “who is stronger,” but two typical strategies:

    • Route A: trades longer reasoning and finer exploration for higher completion;
    • Route B: converges quickly within a limited budget.

    What Each of the Four Metrics Measures

    1. Pass Rate: A Strict, Delivery-Oriented Metric

    Pass rate usually has no partial credit. Most implementations treat “all critical acceptance criteria pass” as a pass — for example, the app must build, install, and launch, and specified features must work as expected.

    Route A’s average completion already reaches 82.3%, yet its pass rate is only 24.0%. This shows that failures are mainly not about “can’t build it,” but about “just short”: a few errors in a small number of steps cause the entire task chain to be judged as failed.

    2. Completion: Measuring How Far a Task Got

    Completion describes “how much was achieved despite failure.” A typical definition is the proportion of acceptance cases passed; it can also be a weighted combination of dimensions such as successful compilation, working features, and code quality.

    Because of this, completion scores across different benchmarks are not naturally comparable — when citing them, first check the definition. Route B’s average completion of 71.2% means that even when a task does not fully pass, most of the work can still be retained.

    3. Time: Measuring How Long You Wait

    Under this article’s definition, time refers to the total elapsed time for the agent to complete one benchmark round, mainly including:

    • model inference time;
    • tool-call time for compilation, installation, and testing;
    • retry time after failures.

    Route A’s total time per round is about 27.4 hours, or about 0.91 hours per task; Route B’s is about 12.0 hours, or about 0.40 hours per task. For interactive development, if the per-task time is misread as “more than one working day,” it greatly overstates the actual wait; if misread as “total time per round,” it understates overall throughput cost. Time, therefore, must always be annotated alongside the cost definition.

    4. Cost: Measuring Resource Expense

    Cost usually includes model call fees plus environment overhead such as sandbox compute. It can be abstracted as:

    1. total cost = base environment overhead
    2. + number of calls × per-round token cost
    3. + execution time × sandbox resource unit price
    • When an execution trajectory is long and there are many back-and-forth rounds, repeatedly passed context is billed multiple times, and cost accumulates and amplifies.
    • Fixed environment costs affect short tasks more, while long tasks stretch the time-related overhead.

    Calculated by the total-time-per-round definition:

    • Route B: 9.9 ÷ 12.0 ≈ $0.83/hour;
    • Route A: 317.4 ÷ 27.4 ≈ $11.6/hour.

    The two differ by about 14 times. Roughly speaking, in the same sandbox environment, Route B’s inference spend may be only a small part of the total cost, with the bulk likely coming from environment runtime; switching only to a cheaper model may yield limited savings. Route A is different: its time is about 2.3 times Route B’s, and its cost about 32 times Route B’s, so its cost is more likely dominated by long-trajectory model calls, repeated context input across rounds, and failure retries. As a result, switching models, compressing rounds, introducing caching, or parallel scheduling may pay off more. The cost-reduction conclusions for the two routes cannot be applied to each other.

    Common Questions: How to Read These Metrics?

    1. What Is the Difference Between Pass Rate and Completion?

    Pass rate is a strict delivery metric — usually only “all critical acceptance criteria pass” counts as success. Completion is a process metric that describes how close a failed task got. An agent can have very high completion yet still fail because one critical step did not pass.

    2. How Do You Calculate Cost per Pass?

    Cost per pass = total cost per run ÷ number of passed tasks.

    Plugging in the data:

    • Route A: 317.4 ÷ (30 × 24.0%) ≈ $44.1;
    • Route B: 9.9 ÷ (30 × 17.3%) ≈ $1.9.

    If the business cannot tolerate Route A’s premium of roughly 23 times per additional passed task, Route B is more cost-effective; if a failure means days of rework, raising the pass rate may be worth it.

    3. Why Is the Pass Rate Under 30% for Long-Horizon Tasks?

    Long-horizon tasks require many constraints to hold at the same time: a self-consistent build environment, consistent interface contracts, correct resources and permissions, and all acceptance cases passing. The last few failure points are usually the most expensive to diagnose, so moving from high completion to a full pass requires disproportionate investment.

    The Engineering Tradeoffs Behind the Score Divergence

    The contrast between Route A and Route B shows there is no “have it all” solution among pass rate, time, and cost.

    1. The Capability Curve: Going from “Just Short” to “Fully Passing” Takes Disproportionate Investment

    Code generation has an “existence gap”: writing correct code is not the same difficulty as making many constraints hold simultaneously. Long-horizon tasks demand the latter more:

    • the build environment must be self-consistent;
    • interface contracts must be consistent;
    • every acceptance case must pass.

    Moving from 82.3% average completion to a full pass often means repeatedly investigating the 17.7% gap. That last sliver of gap is exactly the most expensive to diagnose. Route A’s use of several times the time and cost to gain 6.7 percentage points of pass rate is a clear case of diminishing marginal returns.

    2. Context, Retrieval, and Execution Latency: Where the Time Goes

    Route A’s time per round is about 2.3 times Route B’s, yet its cost is about 32 times Route B’s. This disproportion may come from three factors:

    • Route A repeatedly invokes command execution, runs tests, and reads errors; each round adds latency;
    • it processes more information and produces more reasoning output;
    • when conditions allow, the two routes may execute tasks serially and in parallel respectively, further inflating Route A’s absolute per-round time.

    3. Cost Structure: Unit Price, Round Count, and Scheduling Together Determine the Bill

    • Budget constraints: If each task has a step cap, Route B may be closer to that cap and converge earlier; Route A invests near the resource boundary to pursue higher completion.
    • Shifted to the execution environment: Long trajectories shift overhead to the execution-environment side.
    • Actual bill: depends on whether tasks are scheduled serially or in parallel. Under serial execution, environment resources are occupied for longer; under concurrent execution, resource consumption per unit time rises significantly.

    If your team budgets in a different currency, convert using the exchange rate on the budget date to avoid misleading results from an unspecified fixed rate.

    4. Shell Coupling: The Same Model with Different Shells Can Behave Very Differently

    Route A uses the vendor’s own coding agent, while Route B uses another agent shell. Differences in system prompts, tool sets, context-management strategies, and search strategies can all change the model’s behavior trajectory. So a cross-route comparison is fundamentally a comparison of the “model + shell” combination and cannot be reduced to the model’s strength alone. Only by fixing the shell and changing only the model can you compare models. The comparison in this article does not represent an evaluation of any specific product.

    5. Statistical Significance: Differences in Small Samples Need Careful Interpretation

    Route A’s pass rate works out to about 7.2 of 30 tasks, and Route B’s to about 5.2 — roughly 7 versus 5. Using the normal approximation to the binomial distribution (the following assumes all tasks are equally difficult and is only illustrative of order of magnitude):

    • Route A’s 95% confidence interval for pass rate is about [8.7%, 39.3%];
    • Route B’s is about [3.8%, 30.8%];
    • the standard error of the difference between the two is about 10.4 percentage points, so the 6.7-point difference corresponds to about 0.64 standard deviations.

    Even if the actual difference reaches 6.7 percentage points, that is not enough to declare Route A stronger at conventional significance levels. In practice, task difficulty varies, so a more appropriate approach is a paired test on the same task set — for example, observing how many tasks Route A beats Route B on, and using a sign test to assess significance.

    From Leaderboard to Decision: How to Use These Benchmarks Well

    1. Use “Cost per Effective Pass” Instead of Unit Price

    Pass rate, time, and cost must be compared in the same table. You can use two derived metrics:

    • Average cost per task = total cost per run ÷ number of tasks;
    • Cost per pass = total cost per run ÷ number of passed tasks.

    Plugging in the public data:

    • Route A: 317.4 ÷ 30 ≈ $10.6/task; $44.1 per passed task;
    • Route B: 9.9 ÷ 30 ≈ $0.33/task; $1.9 per passed task.

    If the task’s value cannot support Route A’s premium, Route B is more reasonable; if the cost of failure is high, Route A’s investment may be worth it.

    2. Build Tiered Routing: Let the Right Capability Handle the Right Task

    Task type Characteristics Recommended strategy
    Local edits Clear scope, quick to verify Prefer the low-cost, fast-converging route with a tighter budget
    Long-horizon builds and migrations Many constraints, high failure cost Use the high-completion route, or “low-cost first pass + escalate when needed”
    Unattended batch tasks High volume, partial output acceptable Optimize for throughput and unit cost, and set up automatic retries

    Combined with tiered budgets and result caching, you can further reduce repeated investment.

    3. Treat Budget Caps as First-Class Citizens

    Public benchmarks usually set time or resource caps, so an extreme route’s “completion” is really an achievement under a budget constraint. In engineering practice, apply three constraints — a step cap, a cost cap, and a wall-clock timeout — to keep the agent from falling into endless trial and error.

    The following is pseudocode, showing only control logic, not a directly runnable interface definition:

    1. # Sample pseudocode: dual limits on steps, cost, and wall-clock time
    2. MAX_STEPS = 200 # Prevent non-terminating exploration
    3. MAX_COST_USD = 5.0 # Per-task cost cap
    4. MAX_WALL_SECONDS = 3600 # Per-task wall-clock cap
    5. def solve(task, model, tools, budget):
    6. history = [task.prompt]
    7. for _ in range(MAX_STEPS):
    8. if budget.cost_usd >= MAX_COST_USD or budget.elapsed_s >= MAX_WALL_SECONDS:
    9. break
    10. action = model.next_action(history)
    11. budget.add(action.usage) # Record input, output, cache hits, and other tokens
    12. if action.is_final:
    13. return action.patch
    14. result = tools.run(action.tool, action.args)
    15. history.append(result)
    16. return tools.collect_partial(history)

    In a real system, these three caps can be set as task-level configuration items and connected to the team’s cost-budgeting system, enabling per-project, per-team cost-quota management.

    4. Quantify “Just Short” with Acceptance Scripts

    High completion with low pass rate means much of the output is only one step away from being usable. Teams can:

    • turn key acceptance cases into scripts that output the specific reason on failure;
    • give the agent a “targeted fix” entry point so it only handles failed cases rather than redoing everything from scratch;
    • require changes to include an explanation, so reviewers can focus on the key differences.

    Each run should record at least:

    • whether the task passed overall;
    • completion;
    • per-round or per-task elapsed time;
    • actual cost;
    • failure-category classification: build failure, case failure, timeout, budget exhausted, etc.

    5. Acknowledge the “Ceiling” of Long-Horizon Tasks

    Even the route with the highest pass rate in the public results stays below 30% on the long-horizon mobile development benchmark. This shows that for quite some time, such tasks are better suited to a collaboration model of “agent-led execution, engineer-led review and fallback” than to a fully unattended, fully automatic mode.

    For migration and conversion tasks, a reasonable approach is:

    • first have the agent generate a migration plan and a diff list;
    • have engineers confirm the key decisions;
    • then execute in batches and verify section by section.

    Practical Advice for Building Your Own Coding Agent Evaluation System

    If an enterprise wants to apply the public-benchmark approach to its own codebase, the following steps can help:

    1. Task set: sourced from real engineering. Select tasks from historical requirements, bug fixes, refactoring, and migration work, with clear inputs and acceptance criteria. To reduce difficulty variance, tasks should be layered by difficulty and sufficient in number.
    2. Environment: reproducible and strongly isolated. Run each task in an independent sandbox with fixed dependency versions and controlled external access, ensuring reproducible results and no interference between tasks.
    3. Shell: fix the variables. Within an evaluation, fix the system prompt, tool set, and budget caps, and change only the variable under comparison.
    4. Metrics: record multiple dimensions. Each record should include at least pass/fail, completion, elapsed time, cost, and failure-category classification.
    5. Analysis: paired comparison. Compare at the task level, focusing on the win-tie-loss distribution and significance rather than averages alone.
    6. Cadence: continuous regression. Wire the evaluation pipeline into the development process: whenever a model or shell is upgraded, automatically run the full task set and produce a report to prevent regressions.

    On mainstream public clouds, such a pipeline can combine common capabilities — container platforms, object storage, message queues, logging services, and monitoring and alerting — to build the evaluation into sustainable engineering infrastructure. This way, model selection is no longer a gut-feeling decision but an engineering decision backed by data.

    Conclusion

    A set of public scores reveals the basic structure of coding agent evaluation: pass rate measures the result, completion measures how far it got, time measures how long you wait, and cost measures the expense. The contrast between Route A and Route B shows there is no “have it all” solution among pass rate, time, and cost. For R&D teams, three more pragmatic approaches are:

    • Measure input versus output with “cost per pass,” matched to your own task structure;
    • Use budget constraints and tiered routing to match capability to task;
    • Replace one-off leaderboard rankings with continuous internal evaluation.

    When evaluation data becomes the basis for selection, coding agents can truly evolve from “benchmark contenders” into stable, reliable engineering forces in the development workflow.