Understanding Mobile AI Coding Benchmarks: Long-Horizon Evaluation, Pass Rates, and Cost Tradeoffs

    A practical breakdown of the new Android Bench results and what they mean for engineering teams. This article explains how coding-agent evaluation has shifted from single-function generation to full application delivery, decodes the four core metrics (pass rate, average completion, average duration, and cost per run), analyzes the capability-cost-latency tradeoff revealed by two competing agent runs, and details four common failure modes in long-horizon tasks. It also provides a concrete methodology for building your own internal evaluation set, routing tasks by tier, measuring effective time savings, and constructing a controllable execution environment with guardrails, sandboxing, and observability.

    1. The Evaluation Shift: From “Writing Code” to “Delivering Apps”

    The way we evaluate code LLMs is changing. Early assessments relied heavily on short-horizon tasks such as single-function generation and single-file completion, where a model only had to produce a logically correct snippet given the prompt and surrounding context. In real projects, that is only a small slice of the work. Developers search for code across a repository, modify several related files, adjust build configuration, run tests, diagnose compile errors, and iterate until the project runs — a process that can span hours or even days.

    A major mobile OS platform’s official developer account recently updated the results for its AI coding benchmark (Android Bench), covering real engineering tasks such as app creation, feature development, code migration, and application conversion. The newly published scores signal that evaluation centered on long-horizon tasks has become a meaningful way to measure the capability of coding agents. This article does not focus on any specific product’s ranking. Instead, it examines the methodology and metrics behind long-horizon evaluation and analyzes the questions engineering teams should care about.

    Long-horizon evaluation can be defined as follows: completing a single task requires an agent to keep working across multiple sense-plan-execute-verify cycles, spanning hours, with each task touching multiple files and tool calls. It has four notable characteristics:

    • Complex state: The agent must maintain consistency in project structure, interface contracts, and prior decisions across turns.
    • Tool-chain dependence: Tools for compilation, testing, static analysis, and version control form the execution and verification loop.
    • High verification cost: A single “build + test” cycle often takes minutes, and one round of trial and error can consume a large amount of time.
    • Accumulating errors: Early deviations get amplified in later steps, ultimately surfacing as an overall failure.

    Long-horizon scoring typically uses a “partial credit” mechanism: when a task is not fully passed, the score reflects the proportion of work completed — for example, the app launches successfully, the core UI is interactive, key functionality works, or static checks report no errors. As a result, a single evaluation produces two sets of outcomes: the pass rate, which is the share of tasks that pass all acceptance checks, and the average completion, which reflects how far the agent still is from finishing. Together with duration and cost, these make up the four-dimension framework analyzed below.

    2. Benchmark Composition: What the Four Metrics Tell Us

    The published Android Bench task set contains 30 long-horizon tasks spanning four capability categories:

    • App creation: Building a runnable app skeleton from scratch, including project structure, UI entry points, and basic interactions.
    • Feature development: Adding or modifying business functionality in an existing project, which requires understanding the current architecture and avoiding breakage of existing logic.
    • Code migration: Migrating an existing implementation to a new language, framework, or version, which requires understanding both the source and target technology stacks.
    • Application conversion: Changing the form factor or hosting model of an app while preserving functional equivalence, which places higher demands on equivalence.

    The engineering meaning of each metric is as follows:

    • Pass rate: The share of tasks that complete all acceptance checks end to end. It is the metric closest to “can this go straight into production?” and typically applies a veto for compile errors, test failures, or unusable critical paths.
    • Average completion: The degree of completion for partially finished tasks, useful for judging whether a failure is a “wrong direction” problem or a “final stretch” problem.
    • Average duration: The wall-clock time for a single task from start to finish, including model inference, tool calls, and the wait for builds and tests. It determines the actual wait time a team faces after integrating the agent into the workflow.
    • Cost per benchmark run: The total expense of completing all tasks, and the direct basis for budgeting at scale.

    It is worth stressing that a long-horizon benchmark measures not the model in isolation but the combined performance of the “model + agent.” In the original results, each approach was evaluated through an independent coding agent. The agent handles prompt orchestration, tool calls, context management, error recovery, and termination decisions, and the same model can score very differently under different agent shells. So three cautions apply when reading the leaderboard:

    1. Cross-comparison conclusions hold only for the “model + agent” combination.
    2. With a sample of 30 tasks, differences of a few percentage points should be judged alongside confidence intervals.
    3. Cost fluctuates with the number of retries, so a single result is best treated as a dimensional reference rather than a budget commitment.

    3. Results Comparison: Capability Ceiling vs. Engineering Cost

    The newly published results involve agent runs of two models, abstracted here as “Plan A” and “Plan B”:

    Metric Plan A Plan B Comparison
    Pass rate 24.0% 17.3% 6.7 pp higher, ~39% relative improvement
    Average completion 82.3% 71.2% 11.1 pp higher, close to most of the work done
    Average duration 27.4 hours 12.0 hours ~2.3x
    Cost per benchmark run $317.4 $9.9 ~32x

    Under the same benchmark and different agents, this data shows the classic “capability-cost-latency” tradeoff:

    • Pass rate is not a linear gain: The pass rate improves by less than 40% relatively, yet cost rises by roughly an order and a half, and duration more than doubles. The most capable approach is not necessarily the best choice for every task.
    • Duration reflects the efficiency of the verification loop: The two plans differ by about 11 percentage points in completion but by about 15 hours in average duration. The gap may come from single-turn inference speed, or from the number of build-and-test iterations — that is, trial-and-error efficiency.
    • Nearly complete is not the same as near delivery: Tasks at 70–80% completion are often missing critical finishing work — fixing the last round of test failures, handling edge states, or adapting to different screen sizes. This is precisely the part that determines whether acceptance passes, and it is also key to assessing how much manual effort human-AI collaboration actually saves.
    • Cost structure differs markedly: A cost gap of more than an order of magnitude means Plan B suits high-frequency, batch, low-risk scenarios, while Plan A is better suited to a small number of high-complexity, high-value tasks.
    • Horizontal position: Plan A’s pass rate is higher than several other previously published approaches, but below the current leader.

    Overall, long-horizon capability remains a weak spot: even the leading plan’s pass rate is under 30%. This matches the difficulty profile of long-horizon tasks — failures are rarely “a line of code written wrong,” but rather the chain reaction of deviations accumulated across multiple stages.

    4. Why Long-Horizon Tasks Are Hard: Four Failure Modes

    4.1 Context Bloat and Constraint Forgetting

    A mobile app project typically contains UI descriptions, business logic, a data layer, configuration files, and resource files — many files with complex reference relationships. As execution turns accumulate, the conversation history keeps growing, and constraints established early on — naming conventions, interface contracts, architectural decisions — easily get pushed out of the attention focus. The model then edits code according to local optima and introduces inconsistencies. Common remedies include layered summarization, persisting key constraints into project spec files, and retrieving relevant code snippets on demand, rather than dumping the entire context into the model at once.

    4.2 Error Recovery Capability

    Any step in a long-horizon task can fail — a compile error, a failing test, a dependency conflict. Whether the agent can accurately read error messages, locate the root cause, and apply a targeted fix — rather than blindly retrying or reverting to the previous version — directly determines the number of iterations. A well-designed agent should set a maximum number of turns per task and pause to request human intervention after consecutive failures, avoiding an unproductive “edit-fail-re-edit” loop.

    4.3 Missing Verification Loop

    The gap between “looks right” and “actually runs” is the most common source of mistakes for coding agents. Whether a task passes acceptance depends on the presence of an automated verification loop: does the build succeed, do unit tests cover the critical logic, can UI automation scripts traverse the core paths? Without any one of these links, an agent may believe the task is complete when the actual result cannot be delivered.

    4.4 Semantic Equivalence in Migration and Conversion

    Code migration and application conversion demand not only correct syntax but also behavioral equivalence. Mobile development involves many implicit conventions — UI lifecycles, permission models, background tasks, data storage, and device adaptation. If an agent only performs superficial syntax replacement and ignores differences in platform behavior, it is likely to expose crashes, state loss, or UI misalignment on a real device. Such tasks require explicit equivalence acceptance criteria, such as comparing the behavior of old and new versions on the same set of cases.

    5. From Benchmark to Decision: A Selection Methodology

    5.1 Build Your Team’s Own Long-Horizon Evaluation Set

    Public benchmarks measure general capability and cannot cover a team’s own engineering constraints. The recommendation is to draw 20 to 50 tasks from real backlog items, classify them into app creation, feature development, code migration, and application conversion, and write an automatically executable acceptance script for each task.

    An illustrative task description can use the following structured format:

    1. task_id: feat-0231
    2. category: feature # create / feature / migration / conversion
    3. description: Add price-range filtering to the product list page, combined with existing search criteria
    4. constraints:
    5. - Keep the existing list component's data flow and state management unchanged
    6. - Do not introduce any new third-party dependencies
    7. acceptance:
    8. build: ./gradlew assembleDebug
    9. unit_tests: ./gradlew testDebugUnitTest
    10. ui_smoke: ./scripts/ui_smoke.sh --case filter_by_price
    11. diff_scope: [app/src/main/java/.../product, app/src/test/...]

    The value of an internal evaluation set lies in repeatability: after a model or agent version is updated, running regression tests on the same task batch is the only way to tell whether the upgrade is progress or a regression.

    5.2 Route Solutions by Task Tier

    Different tasks vary widely in complexity and value, so using a single solution for everything is not economical. Borrowing the continuous-integration idea of “quick checks first, full build later,” tasks can be tiered:

    Task tier Typical tasks Solution recommendation Budget and time limit
    L1 Lightweight Code completion, formatting, comment and doc generation, small-scale refactoring Low-cost, fast solution Sub-second response, no separate budget
    L2 Standard New features, writing unit tests, fixing defects, minor adaptation Mid-tier solution, may fall back on failure Minute-level, set call-count and cost caps
    L3 Complex Code migration, application conversion, cross-module architectural changes High-capability solution with strict acceptance Hour-level, set a total budget and termination conditions

    The following illustrative code shows routing logic based on task type and risk:

    1. # Tiered routing: select the solution and guardrails by task type, risk, and change scope
    2. HIGH_CAPABILITY = "high_capability"
    3. BALANCED = "balanced"
    4. FAST = "fast"
    5. COMPLEX_KINDS = {"migration", "conversion", "cross_module_refactor"}
    6. MAX_ATTEMPTS = {"high_capability": 40, "balanced": 15, "fast": 3}
    7. COST_LIMIT_USD = {"high_capability": 60, "balanced": 5, "fast": 0.2}
    8. def route(task):
    9. if task.kind in COMPLEX_KINDS or task.risk == "high" or task.changed_files_estimate > 20:
    10. tier = HIGH_CAPABILITY
    11. elif task.kind in {"feature", "bugfix"} or task.test_coverage_delta > 0:
    12. tier = BALANCED
    13. else:
    14. tier = FAST
    15. return {"tier": tier, "max_attempts": MAX_ATTEMPTS[tier], "cost_limit_usd": COST_LIMIT_USD[tier]}
    16. def run_with_guardrail(task, agent):
    17. plan = route(task)
    18. spent = 0.0
    19. for attempt in range(plan["max_attempts"]):
    20. if spent >= plan["cost_limit_usd"]:
    21. return escalate_to_human(task, reason="budget_exceeded", spent=spent)
    22. result, cost = agent.step(task)
    23. spent += cost
    24. if result.verification_passed:
    25. return result
    26. if result.stalled: # consecutive failures, no new effective progress, etc.
    27. return escalate_to_human(task, reason="stalled", spent=spent)
    28. return escalate_to_human(task, reason="max_attempts_reached", spent=spent)

    The core idea of this tiered strategy is “save where you can, invest where you must.” A large volume of low-risk tasks is handled by the fast solution, while a small number of complex tasks are tackled by the high-capability solution, with guardrails keeping everything under control.

    5.3 Measure ROI Around “Effective Time Saved”

    Gross gain = manual effort time − AI solution duration − human review and rework time.

    Take a complex task as an example. If a skilled engineer needs 16 hours to complete it manually, using Plan A takes 27.4 hours and still requires human review — a rough calculation yields a negative net gain, making it better suited as a “first-draft assistant” than a “fully automated delivery” tool. The same task on Plan B, though with a lower pass rate, takes only 12 hours per run; if it fails, a human takes over, and the overall cost may still be significantly lower than doing it manually. So selection cannot rest on pass rate alone — the fallback cost of failed tasks matters too.

    5.4 Focus on the Failure Distribution, Not a Single Average

    Average completion masks differences across tasks. Go further and analyze: which task categories failures cluster in, which stage they occur at (planning, coding, building, or testing), and how far from passing they were after failing. Distribution characteristics guide optimization direction better than an average: if most failures happen during the build stage, prioritize strengthening feedback from build logs; if they cluster in the test stage, prioritize improving acceptance cases.

    6. Engineering Implementation: Building a Controllable Long-Horizon Execution Environment

    6.1 Sandboxing and Reproducible Environments

    • Isolated execution: Assign each task an independent containerized execution environment, preloaded with the required runtimes, build tools, and test frameworks, with dependencies warmed via images or caches to avoid re-downloading on every task.
    • Version control: The agent commits code on a dedicated branch, so every change is traceable and revertible.
    • Environment consistency: Builds and tests run in an environment close to production, reducing “runs locally but fails in the environment” discrepancies.

    6.2 Task Descriptions and Acceptance Scripts

    A task description should contain four elements — goal, constraints, input/output, and acceptance criteria — avoiding vague instructions like “just optimize this page.” The acceptance script is the sole basis for judging the pass rate and should cover as much as possible:

    • Build artifact generation;
    • Unit tests;
    • UI automation smoke tests for critical paths;
    • Static analysis and style checks.

    6.3 Budgets and Guardrails

    • Cap control: Set cost limits and maximum execution turns per task, per day, and per project.
    • Circuit breaker: When consecutive failures reach a threshold, terminate the task and hand it to a human to avoid wasted consumption.
    • Human gate: Outputs from L3 tasks must go through code review and human acceptance; AI should not merge directly into the main branch.

    6.4 Observability

    It is recommended to feed the structured logs of each run into a unified logging service, recording fields such as task ID, solution, duration, cost, number of retries, and failure reason, then use monitoring and alerting to flag abnormal costs and sudden spikes in the failure rate. An illustration:

    1. # Task-level observability record (illustrative)
    2. record = {
    3. "task_id": "feat-0231",
    4. "tier": "balanced",
    5. "status": "failed", # passed / failed / escalated
    6. "attempts": 9,
    7. "duration_sec": 5420,
    8. "cost_usd": 3.21,
    9. "failure_stage": "ui_smoke", # planning / coding / build / test / ui
    10. "completion_ratio": 0.78,
    11. "escalate_reason": None,
    12. }

    With this data, you can answer three questions: what the return on investment is, which task categories are best suited to AI, and where the bottlenecks in the current workflow lie.

    6.5 Knowledge Accumulation and Continuous Iteration

    Organize the team’s specific coding conventions, architectural agreements, and common error patterns into a project knowledge base, and inject them on demand at task start via retrieval augmentation to ease the long-context forgetting problem. Turn failed tasks into new evaluation cases, forming a “evaluate-optimize-re-evaluate” loop.

    7. Platform Support: Making Long-Horizon Tasks Stable and Manageable

    Long-horizon tasks place higher demands on infrastructure. Using general cloud capabilities, you can build a stable, controllable AI-assisted R&D platform:

    • Container platform: Provides isolated, elastic execution environments to support many concurrent long-running tasks.
    • Object storage: Stores task descriptions, source snapshots, build artifacts, and evaluation results for traceability and reuse.
    • Message queue: Enables asynchronous scheduling and peak shaving for long tasks, avoiding blocking of the main flow.
    • Logging service: Collects runtime logs uniformly to support failure attribution and auditing.
    • Monitoring and alerting: Provides real-time monitoring and anomaly alerts for duration, cost, and failure rates.
    • Model service: Accesses model services of different capability and cost tiers through a unified interface, making it easy to route by task tier and to roll out changes gradually.

    For example, the general product capabilities of Baidu AI Cloud in areas such as model services, containers, object storage, logging, and monitoring can serve as a foundation for building such a platform. Enterprises can also combine capabilities as needed to progressively build AI R&D infrastructure that fits their own workflows.

    8. Conclusion

    Mobile AI coding benchmarks are moving from “writing a correct snippet” to “delivering an app,” marking a shift in evaluation focus from a model’s knowledge base to its engineering execution ability. Pass rate and average completion measure the capability ceiling; duration and cost determine feasibility in practice. Judging by these published results, every plan’s pass rate stays under 30%, long-horizon capability still has substantial room to improve, and there is a clear tradeoff between cost and capability.

    For R&D teams, finding the right combination for their own scenarios matters more than asking “which plan is strongest”: use an internal evaluation set to calibrate selection, use task tiering to control cost, use sandboxes and acceptance scripts to guarantee quality, and use observability to keep improving. Turning benchmark data into engineering decisions is the real value of technology selection.