Mobile AI Coding Agent Benchmarks: Completion, Cost, and Engineering Trade-offs
Mobile app development tasks are long-horizon engineering jobs that require building, testing, UI validation, and repeated repair. This article explains why such benchmarks are difficult, defines four core metrics—pass rate, average completion, average elapsed time, and cost per benchmark run—and interprets anonymized results comparing a high-completion, high-cost approach with a faster, lower-cost approach. It also outlines how teams can build reproducible evaluation pipelines, read benchmark results without being misled by noise, and integrate mobile AI coding agents into real development workflows.
1. Introduction: From Writing Code to Completing Engineering Tasks
Over the past two years, progress in large language models for code generation has mostly shown up in isolated capabilities: completing functions, explaining errors, and suggesting algorithmic ideas. Real software development, however, cares about a different question: given a complete requirement, can an AI coding agent independently carry out the full workflow from design to verification and reach a deliverable quality bar?
Tasks such as creating a new application skeleton, adding a feature to an existing project, or migrating code from an older framework to a newer one share the same profile:
- The task chain is long.
- Many files need to be changed.
- Success depends on builds and test verification.
- Intermediate errors require backtracking and correction.
The industry commonly groups these as long-horizon tasks. Simple subjective human scoring is no longer sufficient. Evaluating an agent’s ability on long-horizon tasks requires engineering benchmarks that can be executed automatically and verified repeatedly.
A mobile operating system platform’s official developer account recently published a set of coding benchmark results for mobile application development. The benchmark focuses on real engineering tasks, covering application creation, feature development, code migration, and application conversion. Two types of approaches in the public results contrast sharply:
- One type has a relatively high pass rate and high completion, but takes a long time and costs a lot.
- The other type is fast and cheap, but has relatively lower completion.
Specific product or model names are omitted in this article, and the key figures are cited anonymously, because what matters to developers is not “who ranks first,” but the metric design, cost structure, and engineering trade-offs behind the numbers.
2. Why Mobile Engineering Tasks Are Heavy
Before discussing the metrics, it is useful to understand why mobile coding benchmarks are hard to run and why they are slow.
1. Long dependency chains amplify errors
A feature requirement often spans multiple layers: data models and persistence, business logic and APIs, and UI/interaction. An agent must perform requirement understanding, code retrieval, multi-file editing, build repair, and test verification in sequence. In an “observe-decide-act” loop, one early mistake can trigger a series of corrective actions and force a rollback when the build fails.
2. Verification itself is expensive
For a mobile app, “correct” does not only mean that the code compiles. It also means that the UI renders properly, interactions behave as expected, and the app runs on the target OS version. This requires:
- Installing dependencies and producing build artifacts.
- Launching the app in an emulator or on a real device.
- Running UI-level automated tests.
A single build-and-test cycle can take from a few minutes to tens of minutes. Evaluation cost accumulates linearly as the number of tasks grows.
3. Engineering context is large
A medium-sized mobile project may contain thousands of source files, resource files, build scripts, and dependency declarations. The agent must locate the relevant files precisely while retaining key information within a limited context window. The more complex the project, the harder retrieval and context management become.
4. Environments are heterogeneous, and reproduction is costly
Different projects may use different versions of system SDKs, build tools, or native libraries. The same benchmark result is comparable only under fully consistent images, dependency versions, and device environments.
For these reasons, mobile coding benchmarks naturally exhibit the properties of high completion requirements, long execution time, and high cost. The four metrics below are quantified expressions of those properties.
3. Four Core Metrics: What Each Metric Answers
1. Pass rate: the final binary judgment
Definition: The share of evaluation tasks that fully satisfy all acceptance criteria. Common acceptance criteria include: the project builds successfully, all existing test cases pass without regression, and all test cases for the new feature pass.
Value: It is a binary metric with no vague “partially correct” state. It best reflects whether an agent can independently complete a full engineering objective.
Limitations:
- Cliff effect: Code that is almost correct but fails one boundary condition receives the same score as code that made no progress at all.
- Sensitive to the task set: With 30 tasks, one additional pass changes the pass rate by about 3.3 percentage points. Task difficulty composition can strongly affect the result.
- Small samples are noisy: A 24.0% pass rate means about 7 of 30 tasks passed. Because actual performance can vary across runs, a rough binomial estimate gives a 95% confidence interval of about 12.4% to 41.6%. For 17.3%, the interval is about 7.7% to 34.3%. These intervals overlap substantially.
Therefore, a single-run pass-rate ranking is not enough to support a conclusion about which approach is stronger. It is better to repeat the evaluation 3 to 5 times and report both the mean and the variability.
2. Average completion: how far the task progressed
Definition: Each task’s acceptance criteria are broken into automatically judgeable subitems—such as “build passes,” “existing tests pass,” “new feature tests pass,” and “UI demonstration script runs successfully.” Each subitem is scored by weight, and the results are averaged across all tasks.
Value: Completion captures gradual progress and avoids the information loss caused by all-or-nothing judgment. It is especially useful for diagnosing:
- Which task types repeatedly fail at the same stage.
- Whether improvements between iterations come from “0 to 1” progress or from “passing to excellent” refinement.
Design considerations:
- Subitem definitions and weights: Weights should be public and reproducible. If “build passes” is weighted too low, results can be inflated by a few simple edits.
- Regression penalty: Regressions introduced by modifying existing code should be reflected as negative impacts, not ignored.
3. Average elapsed time: a key constraint on human-AI collaboration
Definition: The end-to-end wall-clock time from the start of a task to the final output, including inference, tool calls, and verification, averaged across all tasks.
Value: Elapsed time directly affects whether an agent can fit into daily development rhythm.
- If a feature task takes 27.4 hours on average, the agent behaves more like a background batch job than a pair-programming assistant that developers can call at any time.
- At 12.0 hours, the agent is still slow, but it becomes feasible in an overnight automated mode.
Influencing factors: Beyond the model itself, elapsed time is affected by build-cache hit rates, emulator startup overhead, dependency download speed, number of failed retries, and task-scheduling concurrency. Public evaluations often use single-machine serial execution, so elapsed-time figures reflect “processing time within the evaluation environment” and may not exactly match performance after integration into a real development pipeline.
4. Cost per benchmark run: determines whether an approach can scale
Definition: The total cost of running the full benchmark once, commonly measured by input and output token charges for model calls.
Note: Some evaluations may exclude infrastructure costs such as build machine time, device resources, and storage. Teams building their own evaluations should use total cost of ownership as the measurement basis, not only model-call fees.
5. The four metrics must be read together
The four metrics answer four different questions: Can it be done? How much is completed? How long does it take? How much does it cost?
Looking only at pass rate ignores the sunk investment in failed tasks. Looking only at cost ignores the extra consumption caused by failed retries. Looking only at completion may overestimate the share of work that is actually deliverable.
4. Case Study: Engineering Trade-offs in Anonymized Benchmark Data
For readability, the two types of approaches in the public results are referred to below as Approach A and Approach B. Both are anonymous labels used only to explain the structural relationship among metrics. They are not recommendations for any specific product or model.
| Metric | Approach A | Approach B |
|---|---|---|
| Pass rate (30 long-horizon tasks) | 24.0% | 17.3% |
| Average completion | 82.3% | 71.2% |
| Average elapsed time | 27.4 hours | 12.0 hours |
| Cost per benchmark run | USD 317.4 | USD 9.9 |
| Approximate cost per task* | USD 10.6 | USD 0.33 |
| Approximate cost per 1% pass rate* | USD 13.2 | USD 0.57 |
| Average completion of failed tasks** | 76.7% | 65.2% |
* Calculated directly from the original data: cost per run divided by the number of tasks, or divided by the pass-rate value. Infrastructure costs in the evaluation environment are not included.
** Estimation assumption: passed tasks are counted as full completion. Let x be the average completion of failed tasks. Then pass rate × 100% + (1 − pass rate) × x = average completion. Solving gives the value.
1. The completion gap is concentrated in unfinished tasks
Approach A and Approach B differ by 6.7 percentage points in pass rate and 11.1 percentage points in average completion. More importantly, the average completion of failed tasks is about 76.7% for Approach A and 65.2% for Approach B.
This means that most failures for Approach A are “nearly done but missing the last step,” such as:
- One test case was not fixed.
- The UI rendered incorrectly on a target OS version.
- The app crashed under a boundary condition.
For engineering teams, these “unfinished but substantially progressed” artifacts can still be valuable. If handed over to humans or resumed in the next agent run, they may require only a small amount of final cleanup. A high-completion approach does not necessarily produce more deliverable results, but it can reduce the marginal cost of human takeover.
2. High cost is payment for determinism
From the derived cost metrics:
- For every 1% of pass rate, Approach A costs about USD 13.2, while Approach B costs about USD 0.57.
- At the observed run cost, a 30-task benchmark run costs about USD 317.4 for Approach A and USD 9.9 for Approach B.
Approach A’s run cost is about 32 times that of Approach B, while its pass rate is only about 1.39 times higher. This suggests that a large part of Approach A’s cost is spent on more extensive exploration, stricter self-verification, or more retries.
In scenarios where errors are expensive—core paths, non-rollback releases—this premium may be justified. In scenarios where trial-and-error is cheap and fault tolerance is high, the same premium may not pay off.
3. Elapsed-time structure: verification loops and iteration count
Approach A’s average elapsed time is about 2.28 times that of Approach B. The difference mainly comes from two factors:
- More iteration rounds: Long-horizon approaches often repeatedly try in the “edit-build-test” loop, and every round bears a full verification cost.
- Heavier single-round tool calls: These may include emulator startup, multiple test-case executions, and related steps.
If the task flow is simplified into three stages—edit, build, and test—then:
Single-task elapsed time ≈ number of iterations × (inference time per round + build time per round + test time per round) + emulator/device preparation time
This shows that the dominant factor in elapsed time is often not model inference itself, but the number of repeated verification loops. It also suggests an optimization direction: instead of only compressing single-pass inference time, reduce ineffective iterations—for example, by having the agent read relevant tests before making changes and generate an impact analysis in advance.
4. How to interpret this data rationally
- Match the scenario first: Scenarios that need overnight batch migration with human review as the final check may suit long-horizon approaches like Approach A. Scenarios that need rapid prototypes and fast feasibility checks may suit Approach B.
- Watch out for statistical noise: With only 30 samples, single-run fluctuations are obvious. The overlap between A and B’s pass-rate intervals reminds us that ranking differences need to be confirmed with multiple runs.
- Pay attention to cost inflection points: When budget is the main constraint, a combination of “low-cost approach as the baseline + human correction” can trade a small amount of human effort for an order-of-magnitude cost reduction.
5. Building Your Own Evaluation: Reproducible and Comparable Benchmarks
Regardless of which approach is chosen, teams need their own benchmark for model selection, iteration acceptance, and regression monitoring. A reproducible evaluation pipeline should include the following elements:
- Fixed execution environment: Use container images with locked versions, pre-cache dependencies and build artifacts, and fix target OS versions and device configurations. For tasks requiring GUI verification, screenshots can be compared or UI structure assertions can be used.
- Deterministic acceptance scripts: Every task should be judged automatically, without subjective human scoring, so that the same code produces the same conclusion at any time and on any machine.
- Full-chain traceability: Record every tool call, key file diff, build log, and token consumption so that failure causes can be traced and categorized.
- Cost accounting and statistics: Measure model calls, tool calls, and infrastructure costs separately. Run the same task multiple times and report means and intervals.
The following Python snippet is illustrative. Function names and weights are for explanation only and should be adjusted to the actual project.
# Example: skeleton of a single-task evaluation flowdef run_case(case, agent, cfg):# 1. Prepare the task environment in an isolated sandboxenv = sandbox.create(image=cfg.image, task=case.id)# 2. Run the agent and record code changes, tool calls, and token usagetrace = agent.solve(problem=case.statement, env=env, budget=cfg.budget)# 3. Deterministic verification: build -> regression tests -> feature testsresults = {"build": run_build(env), # whether the project builds"regression": run_tests(env, case.regressions), # whether regressions are introduced"feature": run_tests(env, case.feature_tests), # whether feature cases pass}resolved = all(results.values()) # pass rate: solved only when all subchecks pass# 4. Completion: weighted score of subchecks; failed items count as 0completion = sum(w * int(ok) for w, ok in zip(cfg.weights, results.values()))# 5. Record metrics for aggregationreturn {"task_id": case.id,"resolved": resolved,"completion": completion,"hours": trace.wall_clock_hours, # elapsed time for this task"cost": trace.total_cost, # cost for this run"fail_stage": trace.last_failed_stage # failed stage, useful for attribution}
On this basis, teams can further classify failure causes:
- Environment issues: dependency download failures, device startup timeouts.
- Reasoning issues: misunderstanding requirements, locating the wrong files.
- Verification issues: build errors not fixed, regression cases failing.
The share of each category determines optimization priorities. For example, if environment failures dominate, the first priority is to fix the evaluation infrastructure, not to replace the approach.
6. Cost Structure and Performance Variability: Engineering Reality of Long-Horizon Evaluation
1. Long-horizon task costs grow nonlinearly
In long-horizon tasks, agents often send the full conversation history with each request. Suppose a task has n interaction rounds and each round adds context size c. Input-side token consumption is approximately:
c × (1 + 2 + … + n) = c × n(n+1)/2
That is, it grows roughly quadratically with the number of interaction rounds. This means a task that takes 40 rounds to converge may have more than ten times the input-side cost of a task that converges in 10 rounds. If the evaluation method does not set limits on rounds or budget for failed tasks, a small number of “stubborn tasks” can consume most of the overall budget. This is one important structural reason why long-horizon approaches can be expensive.
2. Performance is not a single value; it is a distribution
Averages can hide large differences between tasks. In the same batch, a few simple tasks may pass quickly, while a few complex tasks may take tens of hours. In practice, percentile metrics are more informative:
- P50 elapsed time: the time within which half of the tasks complete.
- P90 elapsed time: the time required by the slowest 10% of tasks.
- Confidence intervals for pass rate.
For engineering teams, P90 often reflects the “worst experience” better than the mean. One task stuck for tens of hours can be enough to make developers lose trust in the whole approach.
3. A single run is not stable performance
Token-based billing, load fluctuation, retry policies, and even network conditions during evaluation can cause variation in a single run. Recommended practices include:
- Use the median of multiple runs as the baseline.
- Focus on variance rather than a single value.
- After an approach upgrade, run an A/B comparison to confirm that the improvement is statistically meaningful rather than random fluctuation.
7. Team Selection and Integration: Turning Benchmark Numbers into Productivity
1. Configure approaches by scenario
| Task type | Recommended strategy | Reason |
|---|---|---|
| Prototype validation, boilerplate code | Fast, low-cost approach | High fault tolerance, frequent trial-and-error, cost-sensitive |
| Routine feature development | Fast approach as primary, with human review | Balances efficiency and cost |
| Core modules, large-scale migration | Long-horizon approach, or human-led with tool assistance | High cost of errors, requires higher determinism |
2. Design “handoff points” for human-AI collaboration
The value of a high-completion approach is that it advances tasks to a “ready-to-take-over” state. Teams can set checkpoints after agent output, for example:
- Pause after the build passes, letting developers review the code diff.
- When test results are below target, automatically summarize failure logs for human judgment.
This preserves the progress-making ability of long-horizon approaches while avoiding continued investment in the wrong direction.
3. Embed evaluation into the development workflow
The ultimate goal is to make agent performance part of the daily engineering rhythm. An evaluation workflow can be designed as follows:
- Use scheduled jobs to run batch evaluations overnight and produce reports the next morning.
- Include average elapsed time, P90 elapsed time, cost per task, and pass rate in monitoring and alerting.
- Notify owners promptly when metrics deviate from the baseline.
In practice, an evaluation pipeline often requires containerized execution environments, object storage for build artifacts, queue services for scheduling evaluation jobs, and logging/metrics services for monitoring and alerting. Teams can use general-purpose cloud services to support these components; in the original context, Baidu AI Cloud is mentioned as one example where it is the chosen provider. This allows teams to focus on evaluation design and task-set construction.
4. Build internal task sets for continuous regression
Public benchmark task sets are fixed and may become over-optimized as approaches iterate. Teams should build internal task sets based on their own business, such as real requirements from historical tickets, and update them regularly. The advantages of internal task sets include:
- Closer to real business distribution: difficulty composition matches actual work.
- Controllable difficulty gradient: makes it easier to observe performance across difficulty ranges.
- Long-term regression capability: after each approach upgrade or parameter change, the same task set can quickly verify whether performance has degraded.
8. Conclusion
Public mobile coding benchmarks reveal a simple but important engineering principle through four groups of numbers: there is no universally optimal AI coding approach; there is only an approach matched to the task, budget, and risk constraints.
- Long-duration, high-completion approaches trade higher single-run cost for higher determinism and are suitable for tasks where errors are expensive.
- Fast, low-cost approaches sacrifice some pass rate and are suitable for scenarios with frequent trial-and-error and fast feedback requirements.
For developers, understanding the design logic behind the metrics is more valuable than the leaderboard ranking:
- Pass rate should be interpreted together with confidence intervals, to avoid being misled by single-run noise.
- Completion reveals how close failed tasks are to success, which directly affects the cost of human takeover.
- Elapsed time is dominated by the verification chain and iteration count, and can be optimized by reducing ineffective iterations.
- Cost grows nonlinearly with interaction rounds, so budget and round limits must be set.
On this basis, building reproducible internal evaluations, clarifying human-AI handoff points, and embedding evaluation into the development workflow can turn AI coding agents from “numbers on a leaderboard” into real productivity on the engineering pipeline.
