Android Development AI Benchmark: High Completion Rate vs. Low Cost
Android Bench evaluates AI models on long-horizon Android development tasks such as app creation, feature development, code migration, and app conversion. Public results show a clear tradeoff: a balanced model reaches a 24.0% pass rate and 82.3% average completion in 27.4 hours at about $317 per benchmark run, while a fast low-cost model reaches 17.3% and 71.2% in 12.0 hours at $9.90. This article builds a simple cost model that includes human rework, explains why completion distribution matters, discusses CI timeout constraints, and outlines a four-step selection process plus key risks of multi-agent designs.
What This Benchmark Measures
Android Bench is Google’s benchmark for Android development. It does not test isolated function calls or class snippets. Instead, it evaluates models on long-horizon tasks like app creation, feature development, code migration, and app conversion. The model must plan, execute, and iterate on its own, sometimes working continuously for hours, and finally deliver a runnable engineering module.
These tasks are difficult because they require multi-file coordination, resource management, lifecycle handling, and third-party SDK integration. A model rarely satisfies every acceptance criterion on the first attempt, so the benchmark records both pass rate and average completion. Pass rate is the share of tasks that meet all acceptance criteria. Completion is the share of a task that the model actually produced, which is useful even if the task is not fully passed.
The Two Models’ Results
The table below shows two models from public results. A third model, the flagship, had the highest pass rate at 28.7%, but the source material did not report its completion, runtime, or cost, so it is not listed.
| Model tier | Pass rate | Avg completion | Avg runtime | Cost per benchmark run |
|---|---|---|---|---|
| Balanced | 24.0% | 82.3% | 27.4 hours | ~$317.40 |
| Fast low-cost | 17.3% | 71.2% | 12.0 hours | ~$9.90 |
These figures come from a publicly available evaluation and are presented neutrally; the model names have been removed. The release date, task-set version, number of runs, and statistical methodology were not independently verified for this article. When citing these results, use the official benchmark page.
From the table, the balanced model’s pass rate is about 7 percentage points higher than the fast low-cost model’s, but its runtime is 2.3 times longer and its cost is about 32 times higher. This is a typical tradeoff.
Comparing Total Cost with a Worked Example
Looking only at the cash cost per run can be misleading. Here is a simple example: a team runs one batch of 30 similar tasks per week. Each failed task needs 4 hours of human rework, and rework labor costs about $30 per hour. These are illustrative assumptions, not real project values.
Expected passes per batch. The balanced model passes about 30 × 24.0% ≈ 7.2 tasks; the fast low-cost model passes about 30 × 17.3% ≈ 5.2 tasks. The difference is about 2 tasks.
Rework saved. Two extra passing tasks × 4 hours × $30/hour = $240 saved.
Extra cash cost. The balanced model costs about $317.40 − $9.90 = $307.50 more per batch.
Net difference. $240 − $307.50 ≈ −$67.50. Under these assumptions, the balanced model costs about $68 more per batch even though it saves roughly two rework tasks.
Break-even point. For the balanced model to break even, each additional pass must save $307.50 ÷ 2 ≈ $153.75. At $30/hour, that means rework time must exceed about 5.1 hours. Alternatively, with 4 hours of rework per failed task, the labor rate would need to exceed about $38.40/hour.
This calculation does not include waiting cost. The balanced model’s average runtime is 27.4 hours, versus 12.0 hours for the fast model, a difference of about 15 hours. If the team cannot do other work while waiting for results, the cost gap grows. Conversely, if failed tasks are complex or often need to be rewritten from scratch, the balanced model’s advantage grows. Teams should replace these assumptions with their own rework data.
Runtime and CI Timeouts
With an average runtime of 12 hours, even the fast low-cost model often exceeds a 1-hour CI gate. If the CI timeout is 1 hour, neither model can finish one full run inside the gate, so they cannot be used directly in a synchronous CI check.
Three workable approaches:
- Put the agent in an asynchronous pipeline. After the code is committed, the agent runs in the background and results return to the code review UI.
- Split the task into smaller modules so each sub-task’s runtime fits within the gate. Splitting increases integration cost, so it needs separate evaluation.
- Set a retry count and an overall timeout. If the run times out, automatically hand off to a human so the pipeline is not blocked for too long.
Completion Distribution Matters More Than the Average
Average completion can hide differences between tasks. Two models may have similar averages, but one may be near complete on half the tasks and produce almost nothing on the other half, while the other is mid-quality across all tasks. Their value to a team is not the same.
Before adopting a model, run a pilot on 10 to 20 internal real tasks. Record completion for each task, then look at the median, the minimum, and the share of tasks with completion above 60%. If most tasks reach 60% or more, the output can be used as a draft that developers complete and fix, even if the pass rate is low.
Four Steps for Choosing a Model
Identify the task type. New modules are greenfield work: there are fewer historical constraints, the model has more freedom, and lower-completion output can still help build a prototype quickly. Modifying old code or migrating an SDK is brownfield work: the model must understand the existing architecture, and one wrong change can affect compatibility, so pass rate matters more.
Decide how to run it. If the pipeline needs a synchronous result, choose a fast model or split the task. If asynchronous batch processing is acceptable, a slower higher-precision model can also work.
Estimate rework cost. Count how many human hours failed tasks took on average over the past month, then plug those numbers into the cost model above to see which model is more cost-effective.
Check compliance requirements. If code cannot be uploaded to a cloud service, only locally deployed models are available; compare performance after that constraint is fixed.
These four steps lead to a few common combinations. A team with a tight budget, acceptable manual rework, and no synchronous CI gate is a good fit for the fast low-cost model. A team with high rework cost, mostly brownfield tasks, and the ability to run overnight batches is a better fit for a higher-precision or balanced model. For teams with strict compliance requirements, deployment choice comes first, followed by performance comparison.
Multi-Agent Collaboration: Opportunities and Risks
A common design is to use a planner model to decompose the task, an executor model to write code, and a reviewer model to check errors. This can concentrate high-quality reasoning on key steps while using a low-cost model for repetitive coding, so it may control both runtime and cost in theory.
There are also clear risks. A planner error can propagate through the chain and become amplified in the final output. Context handoff between models can lose details. More model calls do not necessarily reduce total token consumption. Debugging becomes harder when something goes wrong. Before adopting a multi-agent design, run a small pilot and measure total cost and runtime per accepted task, not just the cost of one run.
Another direction is adaptive computation: the model adjusts its reasoning depth according to task complexity. Techniques such as self-consistency sampling and tree-of-thought are currently mostly seen in research, and their effectiveness on real Android development tasks needs separate validation.
Conclusion
The main value of this benchmark is that it provides a comparison framework. A high pass rate combined with long runtime and high cost is not necessarily the best deal; the right choice depends on rework cost, run mode, and compliance constraints. Teams should run their own small comparison on 10 to 20 real tasks, record pass rate, completion, runtime, and rework time, then calculate the net cost with the formula above. That approach is more reliable than copying a leaderboard ranking.
