Android Bench Benchmarks: Performance and Cost of Long-Horizon Coding Agents

    A new Android development benchmark tests models on long-horizon tasks such as app creation, module implementation, codebase migration, and cross-platform conversion. This article explains how to read the leaderboard numbers, why pass rates are far below completion rates, where the long runtimes and cost gaps come from, how to choose models for different team scenarios, and the key limitations of such leaderboards.

    1. What This Benchmark Tests

    Traditional code completion evaluations only check whether a single function or file is generated correctly, which says little about how a model performs in real development. Recently, major mobile platforms introduced Android Bench, a dedicated benchmark for native Android app development. It covers tasks such as app creation, feature module implementation, codebase migration, and cross-platform engineering conversion.

    These are all long-horizon tasks. A model must understand the project structure, identify dependencies, keep interfaces consistent across multiple files, and debug compilation errors on its own. The benchmark therefore measures combined capabilities—context understanding, tool calling, error recovery, and task planning—rather than correctness on isolated pieces of code.

    Two caveats apply. First, the task descriptions are in English, so results cannot be directly generalized to scenarios using Chinese prompts. Second, the “long-horizon coding agent” in the title refers to these multi-step engineering capabilities, not multimodal inputs such as images or speech.

    2. How to Read the Leaderboard Numbers

    This round of evaluation included three models. The first two are programming agent stacks from their respective vendors; the third is a general-purpose flagship model included as a reference, with pass rate data only.

    Model / Agent Task Pass Rate Avg. Completion Avg. Runtime Cost per Run
    High-density specialized model 24.0% 82.3% 27.4 hours $317.4
    In-house long-horizon reasoning model 17.3% 71.2% 12.0 hours $9.9
    Top international general model (reference) 28.7% — — —

    The figures above are compiled from public leaderboard examples. This article cannot verify the leaderboard version, task count, hardware environment, agent configuration, cost pricing, or measurement methodology, so these numbers cannot be directly reproduced and should serve only as relative references. When reading, go back to the original leaderboard page to verify these details.

    In terms of distribution, the benchmark is very difficult. Even the reference model’s pass rate did not exceed 30%. The high-density specialized model achieved the highest completion rate, but also the longest runtime and highest cost, making it better suited for offline deep analysis. The in-house long-horizon reasoning model is far cheaper, with an average runtime of about 44% of the former, making it suitable for cost-sensitive batch tasks.

    3. Why Pass Rates Are Far Below Completion Rates

    The pass rate means the entire task is judged as fully successful by the acceptance system, requiring the output to satisfy all implicit test points. Completion rate is a looser metric, calculated by the number of functional steps implemented before the task ends. This round’s average completion rate was 82.3%, while the pass rate was only 24.0%. Models often complete most steps but fail on edge cases, type handling, or interface signatures across files.

    Common causes include: failing to handle boundary inputs correctly, type errors when encountering extreme values, or inconsistent function signatures across files.

    One common explanation is that most models are pretrained to predict the next token, so their attention is concentrated on local syntactic structure. When a task spans hundreds of lines and requires tracking dependencies across multiple files, earlier information tends to fade—a phenomenon known as long-horizon forgetting. This explanation comes from model theory; the leaderboard itself does not provide causal analysis.

    For development teams, high-completion code is only a first draft. Code review and integration testing cannot be skipped.

    4. Why Runtimes Are So Long

    Autoregressive models generate tokens one at a time. Each new token requires computing attention over all previous tokens. The cost of a single decoding step grows with context length, and the attention computation is typically O(n²), where n is the context length. Total computation for a task therefore grows roughly quadratically with sequence length—not exponentially.

    Agent tool calls further extend the context. After each command execution or file read, the tool output is appended to the context and processed again as input. The longer the context, the higher the cost of each step, and total runtime accumulates accordingly. The 27.4-hour average runtime in the leaderboard reflects exactly this cumulative effect.

    Two common acceleration approaches exist, with different benefits and trade-offs.

    Speculative decoding uses a small draft model to propose candidate tokens, which the large model then verifies in batches. When the verification rule is implemented correctly, the output distribution is identical to the original model; the main benefit is reduced latency.

    Quantization reduces numerical precision of weights or activations, cutting memory usage and computation. It may affect output quality, so it should be tested on your own tasks rather than assumed lossless.

    5. Where the Cost Gap Comes From

    The per-run cost gap in this round was roughly 32x ($317.4 vs. $9.9). The gap mainly comes from model size and deployment approach.

    Low-cost models may use sparse-activation Mixture-of-Experts (MoE) architectures, activating only a subset of expert parameters at inference time and reducing per-unit computation. This is a hypothesis that should be confirmed against the vendor’s technical documentation.

    Batching improves GPU utilization, and prefix caching reduces repeated computation over shared context. Both optimizations lower inference costs.

    Dynamic Voltage and Frequency Scaling (DVFS) mainly adjusts chip power consumption and operating frequency. There is no public data supporting its impact on per-token cost, so it should not be treated as a primary driver of cost reduction.

    Beyond list prices, there are hidden costs. Model-generated code requires human review and rework, and teams spend time consulting documentation. This time is often more expensive than API fees. You can estimate it as follows:

    • Review cost = generated lines of code ÷ team’s average review speed (lines/hour) × reviewer hourly rate
    • Rework cost = number of rework tasks × average hours per rework × hourly rate
    • Documentation cost = hours spent consulting docs × hourly rate
    • Total hidden cost = review cost + rework cost + documentation cost

    Compare hidden costs together with API fees. If a model has lower API fees but a significantly higher rework rate, its total cost may actually be higher.

    6. How to Choose a Model for Your Team

    Choosing models based on Android development benchmarks

    First, route tasks by complexity, then match models. Each of the following three scenarios comes with a checklist.

    Running high-complexity offline refactoring

    Applies to tasks involving dependency analysis across dozens of files, package renaming, or API replacement. Use the high-density specialized model with higher completion rates, and schedule asynchronously.

    • Complexity threshold: Set based on team experience—for example, tasks touching more than 10 files or spanning multiple modules go into this channel.
    • Timeout limit: Set based on the average runtime of your own evaluation set, with headroom.
    • Checkpoints: Commit intermediate results after each completed module so runs can resume after abnormal termination.
    • Fallback policy: On timeout or two consecutive compilation failures, hand off to a human.
    • Human review point: Must pass the full test suite before merging, with human review of interface changes.

    Handling low-latency incremental generation

    Applies to conversational tasks such as code completion, unit test generation, and single-file edits. These scenarios care about first-token latency; the leaderboard’s overall runtime data cannot be used as a proxy for it.

    • Traffic routing: Send completion and unit test generation to low-latency endpoints; route cross-file changes to the specialized model.
    • Latency threshold: Use your team’s measured first-token latency; switch endpoints when the threshold is exceeded.
    • Fallback policy: If the low-latency model’s output fails to compile once, automatically escalate to a more capable model.
    • Human review: Generated unit tests must run successfully, with human confirmation that assertions are meaningful.

    Building your own evaluation

    No leaderboard can replace testing against your own business. Follow these steps:

    1. Fix the model version, decoding temperature, timeout, and agent configuration.
    2. Extract 3–5 representative tasks from historical commits for small-scale validation. Strip credentials and comments, keeping only plain text. Once results are stable, expand into a formal evaluation set.
    3. Write a scoring script that records compilation success, unit test success, and interface checks, then computes pass rate and completion rate.
    4. Rerun monthly, logging model version, scores, runtime, and cost in the same table to track regressions.
    5. When scores drop below a team-defined threshold, pause model upgrades and investigate the cause first.

    7. Limitations of the Leaderboard

    Leaderboard results only indicate an upper bound on model capability; they cannot directly predict how a team will actually perform. There are three main limitations.

    First, task descriptions are in English. The leaderboard does not cover model performance on Chinese prompts or mixed Chinese-English comments.

    Second, the evaluation sandbox has dependencies pre-installed; it does not involve enterprise network prerequisites such as proxy configuration, private repository access, or internal certificate trust. In real environments, this preparation adds time.

    Third, the pass rate only counts final acceptance results and does not record the number of human interventions. It therefore should not be equated with end-to-end automation rate.

    8. From Code Completion to Task Autonomy

    Including long-running, high-cost models in this round shows the industry’s focus is shifting. Evaluation emphasis is moving from “how much code can a model write” to “how many things can a model complete independently.”

    Agents can plan, call tools, execute commands, and read feedback in a loop. Models are no longer just generating one output and stopping—they can collaborate across multiple steps. Context caching, linear attention (e.g., Mamba-style architectures), and KV cache compression are all reducing the cost of long sequences, but the magnitude of these savings still needs to be measured on your own tasks.

    Whether unattended long-running development becomes routine depends on whether review costs and rework rates can drop to acceptable levels. For technical decision-makers, the current priority is to use reproducible evaluations to judge which capabilities are worth budgeting for—rather than being led by marketing numbers.