Android AI Coding Benchmarks: Android Bench Results, Cost Analysis, and Model Selection

    Android Bench highlights the shift from short coding tasks to long-horizon Android engineering work. This article explains the latest pass rates, why strict evaluation remains challenging, how to calculate cost per successful delivery, and how teams can build an in-house evaluation system and select models by scenario.

    1. Background: Coding Evaluation Is Shifting to Long-Horizon Engineering Tasks

    A public coding benchmark for Android development, Android Bench, has released a new round of results. The previous top model had a pass rate of 28.7% across 30 long-horizon tasks. Two newly disclosed models were positioned around completing real engineering tasks: one used a code agent framework, while the other was positioned as lightweight and fast.

    According to the public description, the benchmark evaluates models across four task categories:

    • App creation: Building project structure from scratch, configuring build scripts, and implementing basic functionality.
    • Feature development: Adding new capabilities to an existing codebase while preserving existing behavior.
    • Code migration: Moving logic from one technology stack, UI approach, or architecture pattern to another.
    • App transformation: Making broad changes to the app’s form factor or underlying technical foundation.

    Unlike code completion or single-function generation, these tasks span multiple files and multiple steps. Models must continuously plan, read and write files, invoke commands, interpret errors, and self-correct within a time limit. These are commonly called long-horizon tasks.

    1.1 Why the Evaluation Paradigm Is Changing

    Previously, a coding model was often measured by its accuracy in completing code given a function signature. Real software development, however, usually involves ambiguous requirements, cross-file changes, dependency conflicts, and layered build errors.

    Several factors explain the shift:

    • Single-point benchmarks have limited discrimination: Short-task pass rates for mainstream models are approaching saturation, making differences in the high-score range less informative.
    • Real tasks form long chains: A typical requirement may involve requirement decomposition, code retrieval, solution implementation, compile verification, and defect fixing. An error in any stage can propagate to the final result.
    • The focus has changed: The question is no longer whether a model can write code, but whether it can deliver independently in a real engineering environment.

    As a result, the industry has begun evaluating coding ability through combinations of agents, tool calls, and real repositories. Android Bench is one example of this trend.

    1.2 Four Core Metrics

    The benchmark uses a combination of metrics to assess how well a model completes tasks, how quickly, and at what cost:

    Metric What It Measures What It Means for Teams
    Pass rate Share of tasks that fully pass acceptance Delivery independence and manual fallback frequency
    Average completion Average share of requirements implemented per task Closeness to a deliverable state and where the model gets stuck
    Average duration Average time from task start to finish Iteration pace, scheduling cost, and concurrency needs
    Cost per run Average cost to complete the benchmark Directly comparable budget consumption

    It is especially important to note that pass rate is a strict 0-or-1 metric. A task counts as passed only when the build succeeds, tests pass, and the functionality meets the requirements. Average completion is scored separately as completed requirement items divided by total requirement items. Together, these metrics reveal whether a model delivers reliably or can complete most of the work but falls short at the final step.

    2. Reading the Results: Three Key Facts Behind the Numbers

    2.1 Overall Public Results

    Under the strict acceptance standard, pass rates among the models in this batch ranged from 17% to 29%. In practical terms, only about 1 out of every 3 to 6 long-horizon tasks can be fully delivered in a single attempt. The two newly disclosed models are summarized below.

    Model Positioning Pass Rate on 30 Tasks Average Completion Average Duration Cost per Run
    High-capability model using a code agent framework 24.0% 82.3% 27.4 hours $317.4
    Lightweight fast model using another code agent framework 17.3% 71.2% 12.0 hours $9.9
    Previous top model 28.7% Not disclosed Not disclosed Not disclosed

    Several previously reported models had pass rates below 24.0%, but specific values were not disclosed.

    The high-capability model’s 24.0% pass rate is higher than the other disclosed values in this batch, but still 4.7 percentage points below the previous top model. Based on 1 / 24.0%, the high-capability model would still need roughly 4 to 5 attempts on average to fully deliver one long-horizon task.

    2.2 Pass Rate and Completion Reinforce Each Other

    The high-capability model’s pass rate is about 38.7% higher on a relative basis than the lightweight model’s, while its average completion is only 15.5 percentage points higher. Combined with the differences in duration and cost, the positioning of the two models becomes clearer:

    • High-capability model: Its average duration is about 2.3 times that of the lightweight model, and its cost is about 32 times higher. It represents a heavy-investment approach: more resources produce more successful deliveries.
    • Lightweight fast model: Its average completion is above 70%, showing that it can complete most of the work on most tasks. Its weakness lies in the final mile. From the perspective of average output per average time, its completion-efficiency may actually be slightly higher.

    2.3 The Evaluated Object Is a Combination, Not an Isolated Model

    The results are produced by a combination of a model and a code agent framework. The framework’s prompt design, tool orchestration, and error-retry strategy all influence the outcome. The public information does not provide detailed attribution, but this distinction is important: when comparing model capability, teams must also consider the agent solution around the model.

    3. Why Pass Rates Are Low: Errors Compound Across Long Chains

    A single long-horizon task can be broken down into a chain of steps:

    1. Read requirements → retrieve related files → create a plan → modify multiple files
    2. → build → read errors → identify root cause → modify again → rebuild → run tests

    Each additional step creates another opportunity for error or deviation. If each step has an independent probability of being correct of about 0.9, then the probability that all 10 steps are correct is only 0.9^10 ≈ 0.35. The more steps involved, the faster overall success declines.

    Errors also propagate through the chain:

    1. Misunderstanding: An early misinterpretation of requirements sends subsequent changes in the wrong direction.
    2. Context overload: When many project files and conversation turns are involved, key constraints can be buried.
    3. Difficult debugging: Agents may make local patches against error messages repeatedly without removing the root cause.
    4. Strict acceptance: A task must pass builds, tests, and other checks to be considered complete.

    For this reason, low pass rates on long-horizon tasks are natural. They result from task difficulty and evaluation strictness, not simply from model degradation. This is also where engineering practice intersects with evaluation:

    • Clear prompts and requirement specifications.
    • Sufficient context supply.
    • Stable and automated build and test environments.

    Improving any one of these factors can increase a model’s real-world success rate.

    4. Reexamining Cost: From Cost per Run to Cost per Successful Delivery

    4.1 Direct Cost

    Decision-makers should not look only at cost per run. A more useful metric is cost per successful delivery:

    1. Cost per run ÷ pass rate = average cost per successful delivery

    Using the public data:

    • High-capability model: $317.4 ÷ 24.0% ≈ $1,322 per successful delivery
    • Lightweight fast model: $9.9 ÷ 17.3% ≈ $57 per successful delivery

    The difference is about 23 times, close to the difference in unit cost itself. This suggests that the pass-rate difference is not enough to offset the unit-cost difference. Many teams face a clear gap between budget constraints and capability requirements, so the high-capability model is not the default answer for every scenario.

    4.2 Visible and Hidden Costs

    • Visible costs: Usage-based charges for model calls and tool calls. A 27.4-hour run also means prolonged resource occupation and may involve concurrency limits and rate-limit constraints.
    • Hidden costs:
      • Failed tasks require developers to review agent-produced code, troubleshoot issues, and resubmit tasks. This manual effort can be comparable to rewriting the code manually.
      • Outputs with more than 70% completion may look close to done. Without a disciplined review process, developers may underestimate the review burden or accept defects that are not easy to notice.

    When evaluating options, teams should include human review time in total cost of ownership rather than only looking at the API bill.

    4.3 Non-Linear Effects at Scale

    Engineering teams often do not use a model for isolated tasks only. They may invoke models in batches for continuous integration, large-scale refactoring, code migration, and similar scenarios. In these cases:

    • Total cost is approximately equal to per-task cost × task volume × average attempts.
    • Total duration is also constrained by concurrency limits, so it cannot be estimated simply as per-task duration × task volume.

    Before large-scale adoption, teams should run a small pilot to validate actual unit cost and task completion rates.

    5. Building an In-House Evaluation System: A Five-Step Practical Guide

    Leaderboards provide a horizontal reference point. The more reusable approach is to treat them as a methodology reference and build an evaluation system around your own repositories that is measurable, comparable, and reproducible.

    Step 1: Task Set Design

    • Use real sources: Select tasks from your own requirements, covering app creation, feature development, code migration, and app transformation.
    • Tier the difficulty: Balance simple, medium, and complex tasks to avoid an artificially high leaderboard score caused by mostly easy tasks.
    • Make acceptance explicit: Each task should define acceptance criteria that can be checked programmatically.

    A simplified task definition example:

    1. # Example task definition: sliced from a real repository and verified programmatically
    2. id: add-settings-toggle
    3. category: feature # app creation / feature development / migration / transformation
    4. difficulty: medium
    5. repo_snapshot: app-v1.0.3 # reproducible repository snapshot
    6. prompt: |
    7. Add a "Follow system dark mode" toggle on the settings screen.
    8. When the toggle is off, the app should keep the current light theme.
    9. acceptance:
    10. build: "./gradlew assembleDebug" # The build must pass.
    11. tests: [SettingsToggleTest] # Required tests must exist and pass.
    12. checks:
    13. - When the system theme is dark and the toggle is off, the UI stays light.
    14. - After enabling the toggle and restarting the app, the dark theme takes effect.
    15. max_attempts: 3 # Maximum number of correction rounds allowed per task.
    16. timeout: 90m # Fail on timeout, aligned with real iteration pace.

    Step 2: Environment Standardization

    Prepare an independent and reproducible isolated environment for each task:

    • Fix dependency versions and build tool versions.
    • Use containers or virtual machines to isolate runs and prevent cross-contamination.
    • Run smoke tests on an emulator or a real device.

    Only when the environment is consistent can horizontal comparisons be meaningful.

    Step 3: Automated Acceptance

    Use programmatic judgment instead of subjective manual evaluation:

    1. Whether the build succeeds.
    2. Whether static checks pass.
    3. Whether relevant unit tests and end-to-end smoke tests all pass.
    4. Whether key functionality behaves as expected.

    On this basis, calculate pass rate and completion. Manual effort should be limited to spot checks and edge-case judgments to keep the evaluation scalable.

    Step 4: Metrics Collection and Cost Control

    Each evaluation run should record at least the following data:

    1. {
    2. "task_id": "add-settings-toggle",
    3. "model_id": "model-a",
    4. "agent": "code-agent",
    5. "attempts": 2,
    6. "status": "passed",
    7. "completion_ratio": 1.0,
    8. "duration_seconds": 4380,
    9. "token_usage": { "input": 182340, "output": 26710 },
    10. "cost_usd": 42.7
    11. }

    Two decision-oriented metrics can be derived from this data:

    1. Task success rate = passed tasks / total tasks # delivery view: how much still requires manual rework
    2. Stability = 1 - standard deviation of attempts / average attempts # scheduling view: whether results are predictable

    Step 5: Repeated Experiments and Result Attribution

    A single run is not enough to support conclusions:

    • Repeat each task 3 to 5 times, take the average, and observe variance.
    • Keep the agent prompt and tool configuration fixed while changing only one variable. This helps determine whether differences come from the model or the framework.
    • Review call logs and failed samples to classify bottlenecks into categories such as requirement understanding, environment dependencies, build failures, and test failures, forming a continuous improvement loop.

    6. Selection Strategy: Scenario-Based Routing, Not a One-Size-Fits-All Choice

    Scenario Characteristic Recommended Direction Reason
    Budget-sensitive, small-batch tasks Lightweight fast model Lower unit cost and shorter per-task duration
    Latency-sensitive workloads Lightweight fast model About 12 hours on average, much faster than 27.4 hours for the high-capability model
    Complex migrations and long-horizon transformations High-capability model Higher completion and lower rework probability
    Parallel multi-task workflows Combination of both Route by task difficulty to balance cost and quality

    6.1 Tiered Routing

    Teams can first classify tasks using simple rules and then select the execution approach:

    • Tasks with few file changes and clear descriptions can be handled by the lightweight fast model.
    • Tasks involving multiple modules or architectural judgment should be handled by the high-capability model, with developer review.

    6.2 Safety First

    Any approach needs boundaries. Models and agents should receive sanitized, permission-limited copies of code and data. Sensitive credentials must not enter the context. AI-produced code still requires both human review and automated testing. High completion rates should not justify skipping review.

    6.3 Cloud Implementation

    If an organization chooses to use cloud-based intelligent coding or agent development capabilities, the selection should be based on results from its own task set rather than on leaderboard rankings. For resource organization, teams may consider:

    • Using a container platform to provide isolated and reproducible execution environments for evaluations.
    • Using object storage to store repository snapshots, evaluation artifacts, and call logs.
    • Using a message queue to schedule long-running evaluation tasks asynchronously.
    • Using logging services and monitoring alerts to track changes in success rate, duration, and cost.

    7. Conclusion

    The newly published benchmark results highlight two points:

    • Long-horizon engineering tasks remain the core difficulty in AI coding. Under strict acceptance standards, pass rates are generally low.
    • Capability and cost do not follow a linear relationship. Blindly pursuing the highest pass rate is not necessarily cost-effective.

    For enterprises, a more valuable approach is to treat public leaderboards as a reference for evaluation methodology, build an evaluation foundation around real business scenarios, and use data to guide model selection. By continuously optimizing task success rate, stability, and cost per successful delivery, AI coding assistants can move beyond “looking like they can write code” toward measurable, deliverable, and sustainable engineering productivity.