Long-Horizon AI Coding Benchmarks: Engineering the Tradeoff Between Pass Rate and Cost
This article analyzes long-horizon AI coding benchmarks through the lens of Android development, a domain that represents full-stack, multi-step tasks requiring real-device verification. Drawing on a public benchmark that evaluated three agents across 30 long-horizon tasks, it breaks down four core metrics—pass rate, average completion, average time, and single-run cost—and shows how higher completion often comes at the expense of longer runtime and higher cost. It explains how to interpret each metric, how to convert them into business-relevant costs (per hour and per successful task), and why high completion and low pass rate can coexist. It also offers practical guidance on task tiering, budget guardrails, observability, cloud collaboration, and how to build a trustworthy long-horizon benchmark, helping teams turn long-horizon AI coding from a personal tool into a measurable, governable engineering capability.
This article starts with a coding benchmark aimed at Android development. In November 2026, an official developer account on a social platform published the benchmark results for three agents. Android development is a typical representative of full-stack, long-horizon tasks that require real-device verification, and the results confirmed a counterintuitive pattern: the more complete solutions tended to take longer and cost more. Behind this lies the inevitable “explore–execute–verify” tradeoff of long-horizon agent reasoning. Rather than focusing on any single product, this article lays out a reusable analytical method covering metric definitions, cost conversion, selection strategy, and benchmark setup.
1. Background: Why Android Development Is the Litmus Test for Long-Horizon Tasks
The public benchmark examined 30 long-horizon tasks across three typical engineering scenarios: building an application from scratch, adding features to an existing project, and code migration and cross-platform app conversion. In earlier instruction-generation benchmarks, a single question usually asked only for writing one function or fixing one bug. Android tasks, by contrast, often span multiple modules and files as well as the build and testing pipeline—exactly the hallmark of long-horizon work.
From an engineering-practice standpoint, a seemingly “simple” Android requirement may touch the following areas:
- Multi-module build system: A project is split into multiple modules by feature and by layer, so changing one shared dependency can ripple across the whole codebase.
- Language and ecosystem: Work involves mixed Java/Kotlin code, plus multiple subsystems such as layouts, resources, annotation processing, and dependency injection.
- Multi-version compatibility: The same codebase must support multiple OS versions and vendor customizations.
- Multi-device verification: Coverage across different screen sizes, resolutions, and chip architectures is required.
- Release pipeline: This includes signing, channel packaging, and staged (canary) rollout.
- Performance constraints: Metrics such as cold-start time, memory usage, power consumption, and frame rate.
These areas are both time-consuming and heavily dependent on toolchain feedback. As a result, whether a model can stay aligned with the goal and continuously self-check and self-correct throughout a long-horizon task reflects real engineering capability more than simply “knowing how to write code.”
2. Four Core Metrics: What the Benchmark Actually Measures
The public data contains four key groups of metrics: pass rate, average completion, average time, and single-run cost. To stay focused on methodology, the agents being compared are referred to below as A and B (A being a high-completion, long-duration type; B a lightweight, efficient type), with the term “leading group” used for the solution with the highest pass rate. The raw data is as follows:
| Dimension | A | B | Leading group |
|---|---|---|---|
| Agent framework | High-completion framework | Lightweight general framework | Chosen by each evaluator |
| Pass rate | 24.0% | 17.3% | 28.7% |
| Average completion | 82.3% | 71.2% | Not disclosed |
| Average time | 27.4 hours | 12.0 hours | Not disclosed |
| Single benchmark run cost | $317.4 | $9.9 | Not disclosed |
2.1 Pass Rate: “Right on the First Try” Under Strict Criteria
Typically, a task counts as passed only when all of its acceptance conditions are met. These conditions include: automated tests fully green, core scenarios verified in an emulator or on a real device, installable build artifacts, and no obvious regressions. It measures whether the result can go directly into review and merge.
2.2 Average Completion: Continuous Progress Rather Than “Pass/Fail”
Completion describes how far each task was advanced. It is closer to a continuous estimate of progress than a discrete pass/fail judgment. A common approach is to weight or interpolate between predefined checkpoints and combine signals such as the proportion of passing tests, the completion ratio of an acceptance checklist, and changes in code coverage.
Completion therefore indicates “how much was done,” not “how much was done correctly.” High completion but low pass rate often means the work reached a substantial level across many stages yet fell short at a few hard gates. This is precisely why strict acceptance criteria are indispensable.
2.3 Average Time: Time Is Also Cost
Long-horizon tasks are usually measured in hours or even days. Runtime is not merely a question of “how long to wait”; it directly affects:
- feedback-loop speed;
- code branch freeze time;
- environment occupancy duration;
- debugging and review cadence.
Average time is also a proxy for the “density of human involvement.” The more time-consuming the task, the more it requires humans to set boundaries and stage-level checks.
2.4 Single Benchmark Run Cost: The Budget Ceiling
This metric reflects the cost of one full evaluation round. It depends on task complexity, the design of the agent framework, the model-invocation strategy, and the execution environment, and it is not a unit price that can be applied directly across the board. When comparing across leaderboards, one must confirm whether the cost basis is consistent—for example, whether it includes the extra overhead from emulator compute and parallelism.
3. Three Engineering Tradeoffs Revealed by the Comparison
3.1 High Completion, Long Duration, High Cost: A as “Slow and Careful”
A’s completion is about 11.1 percentage points higher than B’s, and its pass rate is about 6.7 percentage points higher, but the cost is roughly 2.3x B’s runtime and about 32x B’s cost. At the same time, A’s pass rate is still about 4.7 percentage points below the leading group.
This shows that higher completion and a stronger framework can significantly improve progress on complex tasks, but do not always translate into a higher pass rate, nor are they necessarily the most cost-effective option.
3.2 Fast and Cheap: B as “Efficiency First”
B’s runtime is about half of A’s, at less than 3% of A’s cost, making it suitable for budget-sensitive scenarios that prioritize rapid iteration. But the gap between its 71.2% completion and 17.3% pass rate suggests it is more likely to leave “half-finished” work on complex tasks.
3.3 The Lesson from the Leading Group: Selection Is Not Either/Or
The existence of the leading group shows that a solution with a higher pass rate is possible on this leaderboard. However, its result was achieved after each evaluator chose its own framework, so its conditions may not match those of A and B.
Engineering selection, then, is not a binary choice between “fast” and “good.” It is about finding the balance point that matches business goals across pass rate, completion, runtime, cost, and stability.
4. From “Average Time” to “Wall-Clock Time”: The Underrated Dimension
It is also necessary to distinguish average time from wall-clock time (the actual calendar time that elapses). If all 30 tasks ran strictly serially, a linear estimate based on average time alone would give:
- A: 27.4 × 30 ≈ 822 hours, about 34 days;
- B: 12.0 × 30 = 360 hours, about 15 days.
(This is a rough estimate; actual time is affected by factors such as concurrency, number of retries, environment, and model load.) The real bottleneck for long-horizon tasks is often “how many tasks can be advanced at the same moment.” But raising concurrency comes at a cost:
- Resource contention: When multiple build and test tasks run in parallel, CPU, memory, and I/O compete, and build time can rise nonlinearly.
- Isolation needs: Each sandbox requires its own code copy, build cache, and device environment, so storage and startup overhead increase accordingly.
- Rising cost: The higher the concurrency, the larger the resource bill per unit of time.
Therefore, actual wall-clock time ≈ average task time ÷ effective concurrency, plus queuing and failed retries on top. Looking only at “average time” is not enough when choosing a solution.
5. Dissecting the Engineering Tradeoffs Behind High Completion
Solutions like A being “slow and expensive” is not inefficiency; it is a deliberate choice along the capability dimension. The following mechanisms are common sources of rising runtime and cost for long-horizon agents and apply to most agent frameworks in the industry.
5.1 More Explore-and-Verify Rounds
Long-horizon tasks typically follow an “observe–think–act” loop: retrieve code, read context, edit files, run builds and tests, analyze results, and iterate. The more rounds there are, the greater the number of tool calls and the length of context. The longer the context, the higher the cost of each inference, producing a compounding effect of “long task → long context → higher unit price.”
5.2 The Cost of Finding Errors Rises Over Time
An incorrect architectural assumption early on often surfaces only after many rounds of builds and tests, by which point the cost of rolling back is high. Strict planning, decomposition, and self-checking increase up-front overhead but lower the probability of a costly restart later.
5.3 Context Management and “Forgetting”
The context window is limited, so early constraints in a long task can be diluted. Mature frameworks typically use techniques such as summary compression, external memory retrieval, and task lists to maintain goal alignment—mechanisms that themselves carry token and time overhead.
5.4 More Conservative Termination Conditions
Higher-completion solutions usually do not “stop once the code is written”; instead, they verify repeatedly until the evidence is sufficient or the budget and time limits are exhausted. The strategy of “keep going until the standard is met or the limit is reached” pushes up runtime and cost while raising completion. The 82.3% completion and 27.4-hour average time in the case above are a typical manifestation of this strategy.
5.5 Toolchain and Execution-Environment Overhead
Steps such as building, installing, launching the emulator, taking screenshots, and reading logs all have fixed overhead. More thorough tasks run more end-to-end verifications, and this time adds up considerably.
A simplified model helps frame this: task cost ≈ per-round inference cost × number of rounds × average context multiplier + tool-execution cost. Gains in completion often come from the product of “rounds × context,” and when both grow at once, the cost increase approaches superlinear.
6. Cost Conversion: Three Perspectives Worth More Than “Unit Price”
6.1 Cost per Hour
Roughly converting as “single-run cost ÷ average time”:
- A: 317.4 ÷ 27.4 ≈ $11.6/hour;
- B: 9.9 ÷ 12.0 ≈ $0.83/hour.
(This assumes tasks run serially; if run in parallel, the same total spend corresponds to fewer wall-clock hours, so the cost per hour rises accordingly.) The two differ by about 14x, less than the 32x gap in total cost. This shows that part of A’s cost comes from “running longer,” while another part comes from being “more expensive per hour.” The latter may stem from:
- a higher per-token price for the model used;
- longer context;
- more frequent tool calls.
6.2 Cost per Successful Task
A metric closer to business value is: single-run cost ÷ number of tasks passed.
- A: 317.4 ÷ (24.0% × 30) ≈ $44.1;
- B: 9.9 ÷ (17.3% × 30) ≈ $1.91.
A’s cost efficiency is about 1/23 of B’s. If we let C denote the leading group’s cost, then this metric is C ÷ (28.7% × 30) ≈ C ÷ 8.61. That is, C only needs to be below about $379.8 for its cost efficiency to beat A’s; it only needs to be below about $16.5 to beat B’s.
This means that if a solution is “stronger” but its single-run cost exceeds these thresholds, its cost per success is actually higher. You cannot judge by pass rate alone when selecting.
6.3 Hidden Costs
Beyond API fees, the following often-overlooked expenses also apply:
- Labor: time spent monitoring long tasks, doing stage reviews, and reworking;
- Rework: a low pass rate means a large number of reruns and human fallback effort;
- Compute: the cost of builds, tests, emulators, and real devices;
- Opportunity cost: a branch held for a long time affects delivery cadence.
7. Why “High Completion, Low Pass Rate” Is Not a Contradiction
A has already reached 82.3% completion, yet its pass rate is only 24.0%, and B shows a similar gap. This is the most typical phenomenon in long-horizon tasks, with common causes including:
- Different aggregation methods: Completion may be a weighted summary of local progress across tasks, while pass rate requires all acceptance conditions to be met at once. For multi-stage tasks, a single failed link means all is lost.
- The “last mile” gate: Acceptance items such as build signing, multi-resolution adaptation, compatibility, and performance regression are often the most time-consuming and the hardest to fix automatically.
- Inconsistent acceptance criteria: An agent may consider a task “done,” but the human acceptance standard is stricter, or there is a hidden test set.
This yields an important practical inference: rather than pursuing a “flawless first-time pass,” it is better to design feedback mechanisms that let the agent surface problems early, such as fast failure, staged acceptance, and periodic human intervention. Converting 82.3% completion into passes requires not just a longer runtime, but more efficient verification and human-collaboration processes.
8. Business-Oriented Selection and Implementation: Task Tiering and Budget Guardrails
8.1 Task Tiering Strategy
Different tasks suit different modes, and they can be tiered by risk and complexity:
| Task type | Typical scenario | Recommended mode |
|---|---|---|
| Low-risk small change | Copy, comments, single-point style fixes | Lightweight efficient mode; merge after automated checks |
| Medium-complexity feature | New pages, routine business logic | Lightweight mode produces a first draft, followed by strict testing and human review |
| High-complexity refactor | Cross-module architectural changes, dependency upgrades | High-completion mode; merge only after thorough verification |
| Migration and conversion | Cross-platform conversion, language migration | High-completion mode; split into subtasks by module |
In addition, you can first let the lightweight mode make a quick attempt. If it passes acceptance, adopt it directly; if it fails, hand it to the high-completion mode for a retry. This “light first, then heavy” layered routing avoids running every task in the most expensive way.
8.2 Budget Guardrails and Observability
Set hard limits on cost, time, and number of retries at the scheduling layer, and output structured metrics including: per-task runtime, number of tool calls, token consumption, number of retries, and final status. Illustrative code:
# Illustrative code: describes generic scheduling logic only; not tied to any specific product or toolMAX_RETRY = 2BENCHMARK_BUDGET = 60.0 # Units must match the billing basis; this is for illustration onlydef run_benchmark(agent_factory, tasks, budget=BENCHMARK_BUDGET, max_retry=MAX_RETRY):results = []for task in tasks:usage = {"cost": 0.0, "duration_s": 0.0, "tool_calls": 0}outcome = "failed"# A per-task budget is allocated by task complexity; the remaining quota is reserved for high-complexity taskstask_budget = budget * task.weight / sum(t.weight for t in tasks)for _ in range(max_retry + 1):agent = agent_factory(task)try:outcome = agent.solve(task_budget=task_budget)except BudgetExceeded:outcome = "timeout"finally:# Regardless of success or failure, accumulate actual consumption each attempt to avoid "reruns hiding the cost"usage["cost"] += agent.consumed_costusage["duration_s"] += agent.wall_time_susage["tool_calls"] += agent.tool_callsif outcome == "passed":breaktask_budget *= 0.5 # Shrink the budget on retry to prevent a single failure from dragging down the whole runif sum(r.usage["cost"] for r in results) + usage["cost"] > budget:outcome = "aborted_budget_exhausted"breakresults.append(TaskResult(task=task.id, outcome=outcome, usage=usage))return summarize(results)
8.3 Output and Collaboration Standards
- Unified output standards: Require change descriptions, scope of impact, self-test records, and rollback plans, along with verifiable evidence such as screenshots and logs.
- Context engineering: Codify coding conventions, module boundaries, and build/test commands into project context files (such as an AGENTS.md or CLAUDE.md convention) to reduce repeated trial and error.
- Tiered review: Merge low-risk changes after automated checks; require human review for high-risk changes.
8.4 Cloud Collaboration
Long-horizon tasks are naturally suited to cloud collaboration and can combine the following general-purpose capabilities:
- Build environment: Use containerized build environments to provide each sandbox with an independent, reproducible runtime while ensuring cache isolation.
- Artifacts and logs: Archive build artifacts and runtime logs to object storage for traceability and auditing.
- Cost and runtime monitoring: Connect to logging and monitoring/alerting services, and set threshold alerts on cost, runtime, and failure rate.
- Large-scale evaluation scheduling: Use message queues for task distribution and a container platform for elastic compute scaling, to handle high-concurrency long-horizon tasks and evaluation loads.
For example, general-purpose capabilities offered by Baidu AI Cloud—such as containers, object storage, logging, and monitoring—can serve as the building blocks for this kind of “agent execution + evaluation scheduling” architecture. Teams can build on this to upgrade long-horizon AI development from a “personal tool” into a measurable, governable engineering capability.
9. The Evaluator’s Perspective: How to Build a Trustworthy Long-Horizon Benchmark
If a team wants to build its own benchmark for selection or regression, the following practices can serve as a reference.
9.1 Task Design Close to Reality
- Source: Draw from real issues and PRs, avoiding artificially constructed “toy tasks.”
- Type: Cover different difficulties such as feature development, bug fixing, code migration, and performance optimization.
- Distribution: Ensure a sufficient number of tasks for statistical significance.
9.2 Objective and Executable Acceptance Criteria
- Use automated tests and hidden acceptance tests.
- Define hard gates for compilation, execution, compatibility, and performance.
- Version the scoring scripts to ensure reproducibility.
9.3 Environment Isolation and Reproducibility
- Run each task in an independent sandbox, with no access to the internet or to test answers.
- Pin dependency versions and build-tool versions.
- Record the full execution trace, including every tool call, code diff, and test output at each step.
9.4 Budget and Time Limits
Set uniform budget and time limits for each task to ensure comparability across the board.
9.5 Data Structure and Statistical Interpretation
Record the result of each run in a structured way to facilitate later analysis:
# Illustrative structure; fields can be extended as neededrecords = [{"task": "T-012","model": "agent_A","outcome": "failed", # passed / failed / timeout"completion": 0.91, # average completion"duration_s": 91200, # elapsed time, about 25.3 hours"cost_usd": 8.40, # single-run cost"retries": 1,},# ... remaining tasks]
In particular, before publishing a leaderboard, verify whether the framework configuration of each compared item is consistent, and clearly disclose the cost basis. Otherwise, it is easy to reach the misleading conclusion that “a certain solution is more cost-effective.”
10. Conclusion: Turning “Tradeoffs” into Measurable Engineering Decisions
On the surface, the public Android benchmark results are just a set of scores. In essence, they present three paradigms of long-horizon AI coding:
- High completion, long duration, high cost: A represents “depth first”;
- Fast and cheap: B represents “efficiency first”;
- Leading group: a reminder that selection is not either/or.
Real engineering wisdom does not lie in maximizing a single metric, but in:
- Using pass rate to measure the reliability of results;
- Using completion to judge the depth of task progress;
- Using runtime and cost to assess the time and economic expense;
- Using task tiering to apply different modes in the most appropriate places;
- Using budget guardrails and observability to keep long-horizon tasks controllable, auditable, and optimizable.
For development scenarios like Android—full-stack, long-horizon, and verification-heavy—AI agents are evolving from “code-completion tools” into “task executors.” The competition in future development productivity will be determined not only by which model is stronger, but by whether a team can combine model capability, engineering standards, and an evaluation system into a sustainable, continuously improving engineering loop. Understanding and quantifying this tradeoff is the first step toward truly integrating AI into the R&D process.
