A New AI Coding Benchmark for Mobile Development: First Results, Pass Rates, and Cost Trade-offs

    Across leading general-purpose large models and agent products, a new AI coding benchmark for mobile development reports pass rates all below 30%, showing that most long-horizon engineering tasks still cannot be completed end-to-end by automation. This article breaks down the benchmark's four published metrics—pass rate, completion rate, average runtime, and per-run cost—compares capability against cost, and explains how teams can apply the findings through tiered model routing, cost guardrails, and a self-built evaluation harness. It also highlights the benchmark's limitations so readers can treat the leaderboard as a signal of magnitude rather than exact rankings.

    Recently, a mobile OS vendor’s official developer community account published the first results of an AI coding benchmark built for mobile development scenarios (referred to here as “the benchmark”). Across the leading general-purpose large models and agent products, pass rates all fall below 30%, meaning that under current technical conditions the vast majority of long-horizon engineering tasks still cannot be completed end-to-end by automation. China-based agent products also appear on the leaderboard: one product built on an agent framework ranks second, ahead of several top model products; another lightweight fast version has a lower pass rate but costs only about 1/32 of the flagship product.

    For development teams, the value of these numbers lies not just in “who ranks where,” but in the engineering perspective they offer on AI coding capability. The analysis below covers five levels—evaluation design, metric interpretation, data comparison, selection methodology, and building your own evaluation—and explains how to reproduce this evaluation approach in your own projects.

    Why Mobile Development Deserves Its Own Benchmark

    Current AI coding evaluations tend to focus on classic algorithm problems, general-purpose unit tests, or small-scale function completion. These tasks have clear inputs, immediate feedback, and can be finished in a single turn; they measure a model’s “code generation” ability. But real application development is far more than writing a few lines of code:

    • Large codebases: projects are often made up of dozens of modules and thousands of files, with complex dependencies and long build chains;
    • Long task spans: adding features, refactoring modules, or migrating interfaces often requires repeated navigation across many files;
    • Long feedback cycles: after the code changes, it still has to be compiled, packaged, installed on an emulator or a real device, and manually accepted—a single verification pass can take several minutes;
    • Strict acceptance criteria: functionality must not only “run” but also meet the platform’s strict requirements on UI layout, permission declarations, compatibility, and resource compliance.

    If general coding ability is like a “single-subject exam,” mobile engineering tasks are a “comprehensive practical project.” A model must not only write code but also understand the project structure, read existing implementations, manage long contexts, and stay on target across multiple rounds of operations. Based on public information, the benchmark therefore covers four task categories—app creation, feature development, code migration, and app conversion—which map to the full lifecycle of starting from scratch, iterating, upgrading technology, and migrating across platforms.

    From the reported average runtime (from about ten-plus hours to more than a day) and the completion metric (usually around 80%), this evaluation is designed to answer at least three questions:

    1. Can an agent keep working for a long time without human intervention, without drifting from the goal early on?
    2. Can it maintain engineering consistency across multi-file collaboration?
    3. Can it produce a usable deliverable, rather than just outputting code snippets?

    This is also why the benchmark emphasizes “long-horizon tasks.” In this context, agents become the central vehicle: “model + chat” alone can hardly complete the full loop of compilation, testing, and installation verification on its own. Through tool calls, file reads/writes, command execution, and result self-checks, agents extend model capabilities into an executable engineering workflow. The object of evaluation shifts from “the model” to the combination of “model + agent + toolchain,” which is closer to how enterprises actually deploy these systems.

    Breaking Down the Metrics: Four Dimensions of Real Capability

    The benchmark publishes four metrics—pass rate, completion rate, average runtime, and per-run cost—corresponding to accuracy, effectiveness, and economy.

    Metric Dimension Meaning How to use it
    Pass rate Accuracy Share of tasks that fully meet all acceptance requirements The core metric for delivery quality
    Completion rate Effectiveness Mean proportion of each task actually completed Supplements the information lost in pass rate
    Average runtime Efficiency Average time to complete one benchmark run Affected by model speed and toolchain
    Per-run cost Economy Total cost of one complete benchmark run Watch the billing and accounting scope

    1. Pass Rate: The Strictest and Closest to Real Delivery

    Pass rate uses an “all-or-nothing” rule: if any acceptance case fails, the task is marked as not passed. This matches a developer’s judgment of “can this requirement be delivered as-is,” and it is also the metric most easily misread. The more complex the task and the longer the chain, the more likely failures are to accumulate.

    A simple probability model helps. Suppose each critical step has a probability p of being done right the first time, and n independent steps must all succeed, then the probability of completing the task is roughly pⁿ:

    • When p = 0.95 and n = 20, the task pass rate is about 36%;
    • When p rises to 0.99, the pass rate climbs to about 82%.

    In other words, a pass rate below 30% does not mean “the model can only write simple code”; it more reflects the inevitability of error accumulation in long-horizon tasks. Improving single-attempt success often improves the overall pass rate more than improving the quality of any one code segment. Approaches include: breaking down tasks, narrowing the scope of each single change, introducing test-driven verification, and having agents self-check and roll back regularly.

    2. Completion Rate: A Ruler for “Almost There”

    From the published data, the gap between the highest and lowest completion rates is about 11 percentage points, while the gap in pass rates is far wider. This shows the two metrics are not equivalent:

    • Some products complete most feature points but repeatedly get stuck on a boundary case or an acceptance detail, ultimately forfeiting everything;
    • Some products show a seemingly high completion rate, yet never get the critical path working.

    Completion rate is best used as a diagnostic tool: combined with the distribution of failure reasons across failed tasks (compile errors, runtime crashes, UI mismatches, logic defects, and so on), you can tell whether the weakness lies in “writing a lot” or “writing correctly.” For non-delivery scenarios—such as one-off script generation or internal tool prototypes—completion rate is more informative; once formal release is involved, pass rate should still be the standard.

    3. Average Runtime: A Hidden Cost Amplified by the Toolchain

    Runtime depends not only on model inference speed but also on the execution environment:

    • The thinking time between rounds of code generation;
    • The wait for building, installing, and running tests;
    • The retries and rollbacks after failed tool calls.

    Runtime therefore reflects the overall efficiency of “model + agent + toolchain.” In long-horizon tasks, whether build caches hit, whether tests run in parallel, and whether the emulator stays resident can all significantly affect the final runtime.

    4. Per-Run Cost: What Determines Whether You Can Scale

    Per-run cost is the metric most easily misread. By the leaderboard’s general definition, it refers to the total cost of one complete benchmark run, usually covering the input/output fees of model calls, tool-call fees, and the consumption of the execution environment (such as build machines, emulators, and containers), but excluding the labor investment of the R&D team. When interpreting it, note that:

    • Different accounting scope: cost that counts only model calls cannot be directly compared with figures that include infrastructure costs;
    • Same task difficulty: costs labeled with the same “per-run” figure are not comparable across different task sets;
    • Exchange-rate and time dependence: amounts converted to a given currency fluctuate with exchange rates and should only serve as an order-of-magnitude reference.

    Data Comparison: A Two-Dimensional Map of Capability and Cost

    Placing the public data in the same coordinate system yields clearer conclusions (the table below is only an order-of-magnitude reference and does not constitute a selection recommendation):

    Product (anonymized) Category Score level Per-run runtime Per-run cost
    Agent A Flagship general-purpose LLM 1st 27.4 hours $317.4 (~¥2,123)
    Agent B China-based LLM 2nd — —
    Agent C Leading general-purpose LLM Last (this group) — —
    Agent D China-based lightweight fast version Lowest (this group) 12.0 hours $9.9 (~¥66.2)

    Note: figures not shown were not disclosed in the original leaderboard and are not guessed here.

    1. Being First Does Not Mean Monopoly

    The top product has a pass rate of 28.7%, and the second-place Agent B has 24.0%—a gap of only about 4.7 percentage points. With a limited sample size and randomness in single-run results, this gap is not enough to indicate a clear generational difference between products. By comparison, the overall fact that “the best result is still under 30%” is more instructive: long-horizon automatic development remains a frontier problem, and expecting “one requirement description to generate a complete app” is unrealistic.

    2. The Lightweight Version Is Not a “Cheap Version”

    Agent D has a pass rate of 17.3%, about 6.7 percentage points below Agent B, with an average completion rate of 71.2%. On its own, this number looks like it lags behind, but once cost is factored in, the picture changes completely:

    • Cost gap: 317.4 ÷ 9.9 ≈ 32x;
    • Runtime gap: 27.4 ÷ 12.0 ≈ 2.3x;
    • Cost-to-runtime ratio: about 32 ÷ 2.3 ≈ 14, meaning the cost efficiency obtainable per unit of time is about 14x that of the flagship.

    This means that on the same budget, the lightweight version can complete about 32 full runs, while the flagship can complete only about 1. For scenarios that require repeated trial and error and batch exploration, the lightweight version’s “number of iterations” is itself a form of productivity. For team-internal tools, prototype validation, and routine feature development, the lightweight version is often the better fit; only for tasks with extremely high success-rate demands—such as complex migrations and core architecture overhauls—do you need to bring in the flagship.

    3. The Methodology Behind Runtime

    The flagship agent averages 27.4 hours, close to a full day. Although the evaluation ran unattended and automatically, in a project with team collaboration and joint-debugging needs, a delivery rhythm on this time scale is hard to replicate directly. By contrast, the lightweight version’s average of 12.0 hours, combined with a shorter feedback cycle, fits more easily into a daily continuous-integration workflow.

    One notable inference: rather than waiting for a single 27-hour long task, it is better to break the task into several small batches that can be accepted independently, iterate quickly with the lightweight agent, and hand off to the flagship only for the final push when necessary. This is the basic logic of a “tiered invocation” strategy.

    4. The Marginal Relationship Between Cost and Success Rate

    Let C be the per-run cost and p the pass rate; the average cost per successful task is about C/p. Estimated from the published data:

    • Agent B: 317.4 / 0.240 ≈ $1,322.5;
    • Agent D: 9.9 / 0.173 ≈ $57.2.

    The two differ by about 23x. Notably, while the pass rate improves by only about 6.7 percentage points, the cost per successful task differs by more than twentyfold. This shows that “spending money to buy success rate” has clear diminishing returns, so selection should not focus only on “who is strongest” but also on “how much certainty each dollar buys.”

    Still, cost is only half the decision; the other half is the cost of failure. A simple model quantifies this:

    Total cost ≈ runtime cost + (1 − pass rate) × cost of failure + human review and correction cost

    If failure only means “do it again,” the lightweight version is usually more economical; if failure triggers chain reactions—for example, a wrong change gets merged into the main branch and causes an online incident—the flagship’s higher pass rate is worth the premium. A pragmatic approach is to route tasks with high failure cost and wide blast radius to the high-pass-rate product, and tasks with low failure cost and easy rollback to the low-cost product.

    Practical Approach: Designing Tiered Invocation and Cost Guardrails

    Combining the above analysis, teams can turn “tiered invocation” into a concrete strategy:

    1. Task tiering: classify tasks into three tiers by blast radius and complexity:
      • L1 Exploration and prototyping: one-off validation, scripts, internal tools;
      • L2 Routine iteration: feature development, bug fixes, local changes within code migration;
      • L3 High-risk changes: core module refactors, cross-platform conversion, key pre-release changes.
    2. Routing decisions: L1/L2 go to the lightweight agent by default, escalating automatically on failure or timeout; L3 uses the flagship directly, with mandatory human review.
    3. Budget guardrails: set cost caps, maximum execution steps, and timeout limits for each run and each iteration, pausing and escalating once a threshold is reached.
    4. Evidence retention: keep full logs, code diffs, build output, and test results for later review of failure causes.
    5. Regression verification: after a model or agent version update, re-run the same task set to observe score drift, avoiding switching “by feel.”

    The following pseudocode illustrates the core logic of this strategy. The tool names, task sets, and budget settings in the pseudocode are examples:

    1. # Tiered invocation and cost guardrails (pseudocode, not tied to any specific product)
    2. TASK_LEVELS = {"L1": "light", "L2": "light", "L3": "flagship"}
    3. def run_task(task, budget, tools, verify):
    4. tier = TASK_LEVELS[task.level]
    5. for attempt in range(2):
    6. # Budget guardrail: stop once the limit is reached to avoid endless retries
    7. if budget.spent >= budget.limit or task.deadline.expired():
    8. return TaskResult("aborted", reason="budget_exceeded")
    9. # Choose the agent tier based on the task level
    10. agent = AGENTS[tier]
    11. try:
    12. # Execute in an isolated environment: read code, edit files, run commands, check docs
    13. agent.solve(task, tools=tools, max_steps=agent.max_steps)
    14. # Automatic acceptance: build, install, run cases
    15. report = verify(task)
    16. if report.passed:
    17. return TaskResult("passed", report=report)
    18. except Exception as exc:
    19. task.log.error(str(exc))
    20. # Upgrade from the light tier to the flagship after a failure (only for rollbackable tasks)
    21. if tier == "light" and task.rollbackable:
    22. tier = "flagship"
    23. else:
    24. break
    25. # Return to a human when not passed, preventing wrong changes from entering the main branch
    26. task.escalate_to_human(reason="needs_review")
    27. return TaskResult("escalated")

    This pseudocode reflects three points:

    • The task determines the tier, rather than “more expensive is always better”;
    • Three guardrails—budget, steps, and timeout—prevent runaway costs;
    • An upgrade-and-human-fallback mechanism keeps wrong changes out of the main branch.

    Building Your Own Evaluation: Turning “Leaderboard Numbers” into “Team Capability”

    The reference value of external leaderboards is ultimately limited. Different vendors’ task sets, execution environments, and accounting methods differ, so scores cannot be compared directly across the board. Rather than staring at the rankings, it is better to build your own evaluation system. A suggested starting point covers five areas:

    1. Build a task set: select 20–30 typical tasks from historical requirements, covering the four scenarios of app creation, feature development, code migration, and app conversion, and define each task’s input (requirement description, baseline project), acceptance method (build, install, UI assertions, functional cases), and allowed tool permissions.
    2. Set up an isolated environment: create an independent sandbox for each run, providing a version-controlled build toolchain, an emulator or container, a cached build directory, and a consistent network-access policy to avoid environmental interference with results.
    3. Unify cost accounting: convert model calls, tool calls, and the consumption of build machines and emulator resources into a consistent accounting basis, and also record human-intervention time to form a combined view of “machine cost + labor cost.”
    4. Automate runs and scoring: use a scheduling system to trigger tasks in batches, recording pass/fail, completion rate, runtime, and cost item by item, and generate visual reports.
    5. Regress and calibrate regularly: after each model or agent version change, re-run the task set and observe score changes; periodically add new tasks to prevent “evaluation-set aging.”

    Going further, teams can use “task pass rate” to work backward to process-optimization opportunities:

    • If failures concentrate on compilation and builds, prioritize optimizing toolchain configuration and dependency caching;
    • If failures concentrate on UI and interaction, add UI assertions and screenshot comparison;
    • If failures concentrate on cross-module changes, refine task descriptions and provide a module index.

    This way, evaluation results are no longer just “scoring” but a feedback loop that continuously drives engineering-capability improvement.

    Limitations and Risks: Reading the Leaderboard Rationally

    • Limited sample size: from the “30-task” scope, the sample is small, and whether a single task passes can noticeably change the final ranking, so the leaderboard is better for gauging magnitude than exact positions;
    • Results depend on the agent framework: the same model paired with different agents can score very differently, so the leaderboard measures “combined capability”;
    • Cost and runtime are environment-dependent: execution-machine performance, network conditions, and build-cache hit rates all affect results;
    • Capability ≠ production readiness: passing the benchmark does not mean an agent can directly replace engineers. Code review, security scanning, permission compliance, and release processes still require human oversight.

    Conclusion

    The first results of this benchmark convey three core messages:

    1. The capability ceiling is still limited: even the most advanced models have a pass rate below 30% on long-horizon mobile engineering tasks, and end-to-end automatic delivery is still immature;
    2. China-based products perform well: China-based agents rank second on the leaderboard, and the lightweight version delivers strong cost-efficiency at very low cost;
    3. Cost-efficiency differences are huge: per-run costs between products differ by about 32x, so selection should be based on task value and the cost of failure rather than a simple “capability ranking.”

    For development teams, the pragmatic approach is to treat AI coding agents as “measurable productivity tools”: use tiered routing to assign tasks to the right tier, use cost guardrails to control budgets, and use a self-built benchmark to continuously calibrate results. Only by closing the loop of “evaluate—select—verify—improve” can AI coding capability truly translate into predictable development efficiency.