Evaluating Mobile AI Coding Agents: A Long-Horizon Benchmark Guide for Selection Decisions

    AI coding agents can now handle the full chain of mobile development work, from scaffolding an app to migrating existing code. This guide explains how to read the latest long-horizon evaluation results for these agents, what the four core metrics (pass rate, average completion, average runtime, and cost per run) actually measure, and why high completion scores often fail to translate into working deliverables. It also covers the engineering tradeoffs behind "high completion, long runtime" behavior, a stage-by-stage selection strategy, and a practical method for building your own evaluation loop with layered scheduling, cost accounting, and cloud infrastructure.

    1. Introduction: Mobile Development Evaluation Has Entered the Long-Horizon Engineering Stage

    Public evaluation data in the mobile app development space has been updated recently, and results for several AI coding agents that took part are now being released. These evaluations measure the complete engineering chain from requirement to runnable application, covering real-world tasks such as app creation, feature development, code migration, and application conversion. Pass rates for long-horizon tasks land roughly in the 15%–30% range, average runtimes span from several hours to tens of hours, and the cost of a single evaluation run varies by tens of times across candidates.

    Rankings are only the output. What matters more is understanding the engineering logic behind them:

    • Why do pass rate and average completion diverge so noticeably?
    • What technical tradeoffs does “high completion, long runtime” reflect?
    • How should teams turn these metrics into an executable selection decision?

    This article addresses those three questions and provides a practical method for building your own evaluation loop.

    2. Why Mobile Development Needs Dedicated Long-Horizon Evaluation

    Traditional code generation benchmarks mostly consist of short tasks such as single-function completion or unit test generation, and they measure local correctness. Mobile app development is a classic long-horizon engineering activity. From requirement decomposition, project scaffolding, and build configuration to UI implementation, data persistence, multi-device adaptation, and producing build artifacts, a mistake at any stage can zero out the entire delivery.

    For this reason, evaluations in the mobile development context typically divide tasks into four categories:

    Task type What it mainly tests Typical engineering risk
    App creation Building a complete, compilable, runnable project from scratch, including directory structure, build configuration, and manifest declarations Missing configuration entries that prevent the project from building
    Feature development Incrementally implementing screens, interactions, and business logic on top of existing code without breaking current functionality Regressions and inconsistency with the existing architectural style
    Code migration Equivalent porting across languages, across UI paradigms, or across build systems Behavioral drift and hidden loss of semantics
    Application conversion Platform-level stack replacement or cross-platform rework Incompatible dependencies and platform-specific behavioral differences

    Such evaluations usually require a coding agent to autonomously run the loop of “write code, execute build, run verification, read feedback, fix code” inside a controlled sandbox across multiple rounds, and a task counts as passed only when every verification script succeeds. The results therefore test far more than raw code generation quality: they also cover task planning, tool invocation, error diagnosis, and maintaining state over long workflows. That is the premise for the metrics discussed below.

    3. Four Core Metrics: How to Read Evaluation Results

    3.1 Pass rate: How much can actually be delivered

    Pass rate is a task-level binary metric: a task whose verification fails is recorded as a failure no matter how high its completion score is. It directly describes the deliverable proportion and is usually the primary indicator of a model’s upper-bound capability. However, it is highly sensitive to the difficulty distribution of the task set. If the test set includes tasks that depend on external environments or involve subjective review, a single outlier can noticeably move the result.

    3.2 Average completion: Where does it fall short

    Average completion describes how far tasks were advanced, and it is useful for locating common bottlenecks. It is important to note that completion does not scale with pass rate, and some stages score “cheaply” by nature:

    • App creation: Creating code files and laying out the directory structure already yields a fairly high completion score, yet a single configuration detail or manifest declaration error makes the project unbuildable.
    • Migration and conversion: Once the main code blocks are replaced, completion looks substantial, but behavioral consistency can only be confirmed through a full regression pass.

    Completion is therefore suitable for diagnosing where tasks typically get stuck, and it cannot substitute for pass rate as evidence of deliverability.

    3.3 Average runtime: How long do you wait

    Average runtime is the actual wall-clock time from task start to task end. Runtime for long-horizon tasks is largely dominated by the verification loop: each build, each deployment to an emulator or physical device, each test run, and each analysis of failure logs carries a fixed overhead of minutes or more. Such overhead is hard to eliminate with faster model inference — the more loop iterations, the longer the total runtime.

    In addition, long-running tasks generally cannot proceed unattended end to end. People commonly insert human confirmation at key points, for example when changes touch a production branch, when permission declarations are modified, or when dependencies are introduced. Once a task runs for tens of hours, it has to be coordinated with the developer’s working rhythm and holiday schedule.

    3.4 Cost per run: Is it worth it

    Cost per run is the resource overhead required to complete one full evaluation run. It consists of three main parts:

    • Model invocation cost: determined by input and output token consumption. Context accumulates continuously in long-horizon tasks, and this part of the cost is often underestimated.
    • Environment cost: compute consumption such as build resources and device resources.
    • Human cost: requirement confirmation, task supervision, and acceptance review.

    The tens-of-times cost gaps seen in public data are a reminder that teams must establish a consistent cost accounting basis.

    3.5 The inherent tension between metrics

    Real development work frequently calls for “fast validation, small iterative steps.” When the cost of a single run approaches several days of one developer’s working hours, the speed advantage is easily offset by cost, and can even invert:

    Assumptions: one task runs about 8 hours, with 4 tasks running concurrently each night.

    Option Pass rate Cost per run New passing tasks per night (expected) Cost per pass
    Option A ~25% ~$320 4 × 25% ≈ 1.0 ~$320
    Option B ~17% ~$10 4 × 17% ≈ 0.7 ~$14

    Option A produces slightly more passing tasks per night, but the cost of each individual passing task is more than twenty times that of Option B. Two conclusions follow:

    1. Iteration frequency matters as much as unit cost: splitting a night into 4 short task runs yields more output than one 8-hour run, depending on task size and startup overhead.
    2. Capabilities are complementary: Option A’s strength is pass rate on complex tasks, which suits hard problems; Option B is better suited to fast exploration and batch trial and error.

    Note: the figures in this table are derived from the public data cited in this article and are used only to illustrate the conversion relationship between metrics. They do not represent any specific product.

    4. Deep Dive: The Engineering Tradeoffs Behind “High Completion, Long Runtime”

    4.1 The verification loop sets the runtime baseline

    A simplified model captures the runtime of a single task:

    1. T ≈ Σ(inference time + tool call time) + N × (build time + deploy time + test time) + T_wait

    Here, N is the number of verification loop iterations and T_wait is waiting caused by queuing, rate limiting, and human confirmation. Inference speed is usually not the main bottleneck; the fixed overhead of build, deploy, and test accounts for a large share of total runtime. The effective way to shorten runtime is therefore to optimize the verification stage — for example incremental builds, build and device caching, and test tiering (static checks first, then build, then end-to-end verification) — rather than simply switching to a faster model.

    4.2 High completion comes from a “steady and slow” technical route

    Agents that pursue high completion generally adopt a more conservative strategy:

    1. Search the codebase thoroughly before acting, to understand the existing architecture and conventions;
    2. Prefer small incremental changes, running verification after each step and retrying across multiple rounds on failure;
    3. Keep a large context throughout the task and run fairly comprehensive regression tests.

    This strategy significantly reduces the risk of introducing regressions, at the cost of more verification rounds, longer context, and higher input and output token consumption — showing up as high runtime and high cost at the same time. It embodies a classic reliability–efficiency tradeoff: closer to an “auditable automation engineer,” well suited to structurally complex tasks with high change costs. It also means fast trial and error is difficult, and it should be stressed that regardless of how high the completion score is, output still has to pass code review and testing.

    4.3 High completion but limited pass rate: The gap is in the last mile

    The combination “average completion above 80%, pass rate below 30%” is very common, for four reasons:

    1. Verification acts as a veto: the test suite contains environment-dependent or flaky cases, and the task is marked as failed because of them.
    2. Long-tail defects cluster at the end: boundary conditions, concurrency behavior, and adaptation to low-end devices often surface only in the closing stage.
    3. Non-determinism: small differences along the planning path accumulate continuously, producing high result variance.
    4. Evaluation design effects: if completion is weighted by number of subtasks while a single critical step carries a very small share of the total weight, it gets amplified in the score.

    When interpreting the data, it is worth tracking these metrics alongside:

    • Standard deviation across repeated runs: to judge whether results are stable;
    • Distribution of completion scores among failed tasks: to judge whether the weak point is “a few hard problems” or a “systemic gap”;
    • Correlation between completion and pass rate: if the correlation is weak, the definition of completion needs revision, for example by assigning higher weight to critical checkpoints such as “build succeeded” and “tests passed.”

    4.4 Cost advantage as a trial-and-error budget

    The other meaning of low cost is the number of attempts it makes affordable. Facing a difficult task, Option A buys a slightly higher pass probability with one expensive run, while Option B can spend the same budget on multiple low-cost attempts — for example changing task decomposition granularity, adding context, or revising the prompt. The cost advantage also has limits: when a task demands extremely high one-shot accuracy and the cost of failure is large, the total cost of the fast option may turn out to be higher.

    4.5 Selection strategy by stage

    Scenario Key constraint Metric focus Recommended strategy
    Requirement exploration and prototype validation Feedback speed, low cost per attempt Runtime, cost per run Prefer low-cost, fast-responding options
    Feature iteration in an existing project Delivery rate of runnable code, regression risk Pass rate, stability Prefer high-pass-rate options with full regression testing
    Large migration or conversion Long cycle, complex environment Pass rate, reviewability Use high-completion options for the hard part, low-cost options for upfront analysis
    Tight budget, large-scale batch processing Unit cost, concurrent throughput Cost per pass, pass rate Layered scheduling with human fallback where necessary

    5. From Leaderboard to Decision: Building Your Own Evaluation Loop

    5.1 Building the evaluation set

    • Real sources: sample from requirement tickets, defect tickets, and migration plans, stratified by task type and difficulty to ensure representativeness.
    • Suppress non-determinism: pin dependency versions, pre-warm caches, fix the emulator image, and eliminate network jitter.
    • Machine-decidable: write scripted acceptance criteria for every task to avoid subjective scoring.

    An example task definition (illustrative):

    1. {
    2. "task_id": "feature-0007",
    3. "category": "Feature Development",
    4. "repo_snapshot": "repo://mobile-demo@a1b2c3d4",
    5. "goal": "Add draft auto-save to the sample app and show a hint when a draft is restored",
    6. "acceptance": [
    7. {"stage": "static_check", "command": "./build.sh lint"},
    8. {"stage": "build", "command": "./build.sh assemble"},
    9. {"stage": "unit_test", "command": "./test.sh unit"},
    10. {"stage": "e2e_smoke", "command": "./test.sh smoke"}
    11. ],
    12. "budgets": {
    13. "max_wall_clock_minutes": 90,
    14. "max_tool_calls": 200,
    15. "max_tokens": 2000000
    16. }
    17. }

    5.2 Layered scheduling strategy

    1. def layered_execution(tasks, fast_engine, strong_engine):
    2. results = []
    3. for task in tasks:
    4. # Layer 1: cheap engine attempts quickly, raising the resource budget on demand
    5. r = run_with_retry(fast_engine, task, max_attempts=2)
    6. # Layer 2: tasks still failing go to the high-pass-rate engine
    7. if not r.passed and task.difficulty in ("hard", "migration"):
    8. r = run(strong_engine, task, human_review=True)
    9. # Layer 3: human fallback
    10. if not r.passed:
    11. r = manual_takeover(task)
    12. results.append(r)
    13. return summarize(results)

    5.3 Cost accounting and reporting

    Write every run into a structured table, for example:

    1. runs(task_id, engine_id, attempts, passed, completion,
    2. wall_clock_minutes, tokens_in, tokens_out, compute_minutes, human_minutes)

    On top of this, define two core metrics:

    • Cost per effective pass: total cost ÷ number of passing tasks;
    • Daily task pass rate: number of passing tasks ÷ total tasks ÷ average runtime (in days).

    Reports should present the mean, median, and P90 together, so that long-running tasks do not inflate the average and hide the actual experience of most tasks. Also note that completion and pass rate must be counted separately. If samples that “failed the task but had high completion” are also counted as passes, the evaluation results become meaningless.

    6. Running Long-Horizon Evaluation in the Cloud

    Long-horizon evaluation places high demands on infrastructure. The following general cloud capabilities can be combined:

    • Container platform: provides isolated, reproducible build sandboxes for each task and supports large-scale concurrency;
    • Object storage: centrally stores task artifacts, logs, test reports, and screen recordings for traceability;
    • Log service: collects agent runtime logs and build logs in one place to support failure attribution;
    • Monitoring and alerting: continuously tracks pass rate, runtime, and cost trends to catch version regressions early;
    • Model services and code intelligence capabilities: provide model invocation plus assistance such as code completion and code review;
    • Careful permission design: archive artifacts with write-only permissions, and block outbound internet access and production resources from the sandbox by default.

    For example, the relevant Baidu AI Cloud products can be combined along these general categories, and enterprises can also implement the same capabilities on their own clusters. Whichever approach is chosen, the evaluation process should follow the principle of least privilege and keep “the agent autonomously modifying code” strictly isolated from “merging to the main branch and publishing artifacts.”

    7. Conclusion: Turning Evaluation Data into Engineering Decisions

    AI coding agents now cover the complete chain of mobile development work, from app creation to code migration, but public data shows that one-shot pass rates on long-horizon tasks remain limited. The steady route represented by “high completion, long runtime” and the efficiency route represented by “low cost, fast response” are not mutually exclusive; they are complementary choices for different stages.

    More valuable than a leaderboard is a continuously running, self-built evaluation system: use real tasks to test capability, use a consistent accounting basis to measure cost, use layered scheduling to balance quality and efficiency, and use a rollback-capable release process to hold the line. Only then does evaluation data genuinely translate into engineering certainty.