Mobile App Engineering AI Coding Benchmark: Pass Rate, Completion, Time, and Cost Tradeoffs
This article examines a mobile app engineering benchmark for AI coding agents. It compares two published result sets: one with a 24.0% pass rate, 82.3% average completion, 27.4 hours average time, and about $317 per benchmark run; another with a 17.3% pass rate, 71.2% average completion, 12.0 hours average time, and about $9.9 per run. It explains why mobile app engineering is a difficult benchmark domain, interprets key leaderboard metrics, analyzes the engineering tradeoffs behind deep iterative and lightweight approaches, and provides a practical framework for building internal evaluation sets, using confidence intervals, routing tasks by complexity, and controlling cost and risk.
Recently, a cloud vendor published a new round of AI coding benchmark results for mobile application engineering. A coding tool with a configured model achieved the highest published pass rate at 24.0%, with an average completion score of 82.3%, an average time of 27.4 hours, and a single benchmark run cost of about $317. Another tool achieved a 17.3% pass rate and 71.2% average completion at about $9.9 and 12.0 hours.
Putting these two result sets together reveals a core issue in real software engineering: in mobile app development, which is a “verification-heavy, long-chain” scenario, a model’s task completion capability is only one dimension of decision-making. Teams also need to make explicit engineering tradeoffs among pass rate, completion score, wall-clock time, and cost.
This article first outlines the task characteristics and metrics behind such benchmarks, then analyzes the tradeoff logic behind the two types of results, and finally presents a practical internal evaluation and selection scheme that can be adopted within engineering teams. The goal is to help developers make reasonable choices under budget and schedule constraints.
1. Why mobile app engineering is a hard scenario for AI coding
Mobile application engineering is treated as a separate benchmark category because of its engineering characteristics.
- Complex build chain: A project is usually composed of multiple modules and languages, with a large build system. Compilation, packaging, signing, and installation are all required.
- Verification depends on runtime environment: Whether a feature works correctly ultimately depends on actual interaction on an emulator or physical device. Static code reading alone often cannot reveal runtime behavior, such as lifecycle callback timing, UI rendering effects, or system permission policies.
- Large context span: A single requirement may involve the UI layer, business logic layer, data layer, and other modules. The model must maintain architectural consistency and code-style uniformity across a long sequence of calls.
- Rapid ecosystem evolution: OS versions, UI components, permission models, and dependency libraries are continuously updated. Both legacy migration and new feature development require the model to keep up with changes.
For this reason, such benchmarks usually include four task types:
- App creation: Starting from an empty project and building an app with basic architecture and core UI.
- Feature development: Adding new feature modules to an existing codebase, or modifying existing logic.
- Code migration: Replacing old APIs, old components, or legacy architectural approaches with newer alternatives while preserving existing behavior.
- App conversion: Converting an application from one platform or technology form into another.
The common characteristic of these four task types is that they are end-to-end: the agent needs to understand requirements, plan an approach, read and write multiple files, run builds, execute the app, iterate and fix issues based on logs and UI feedback, and continue until acceptance is passed. This is fundamentally different from algorithm-style benchmarks that only evaluate single code snippets. It is closer to the daily workflow of a junior to mid-level engineer.
2. How to read the leaderboard: what each metric means
Such benchmarks usually report the following metrics:
| Metric | Meaning | Why it matters for engineering decisions |
|---|---|---|
| Pass rate | Share of tasks meeting all acceptance criteria | Indicates the possibility of delivery without manual intervention |
| Average completion score | Average task progress, reflecting partial completion | Indicates the starting value of agent output and the cost of human takeover |
| Average time | Wall-clock time required for one benchmark run | Determines whether the agent can be embedded into CI/CD workflows |
| Cost per run | Model and tool-call cost consumed by a full benchmark run | Indicates marginal cost when scaling usage |
It is especially important to note that in these two result sets, the denominator of pass rate is the same 30 long-horizon tasks. This means that each additional passed task increases the pass rate by about 3.3 percentage points, while each missed task lowers it by the same amount.
Also, the two evaluations used different agent frameworks. This means performance differences may come not only from the model itself, but also from scaffolding design, tool-calling strategies, and context-management approaches. When interpreting the data, it is not appropriate to attribute all differences solely to model capability.
3. Engineering profiles behind the two result sets
| Metric | Coding Tool A (Agent A) | Another Tool (Agent B) |
|---|---|---|
| Pass rate | 24.0% | 17.3% |
| Average completion score | 82.3% | 71.2% |
| Average time | 27.4 hours | 12.0 hours |
| Cost per benchmark run | About $317 | About $9.9 |
| Agent framework | One agent framework | Another agent framework |
From a horizontal comparison, Agent A’s result ranks above other published model results, behind only the best published result at 28.7%. From this single data set, two distinct engineering profiles can be drawn:
- Deep optimization type: More willing to invest time and compute, repeatedly building, running, and self-correcting, pushing tasks from “almost complete” to “fully passing.”
- Lightweight iteration type: Completing the main work at lower cost and higher speed, leaving the remaining work to humans or other processes.
4. High completion with long runtime: what engineering tradeoff does it reflect?
4.1 Trading deep iteration for one-shot correctness
Long-horizon mobile development tasks are rarely correct on the first attempt. An agent usually needs to go through a loop of plan → implement → build → run → observe logs or UI → fix. Each round may produce new compile errors or runtime exceptions.
Time is mainly concentrated in three places: full builds and packaging, emulator or device startup, and app installation plus interaction verification. An average time of 27 hours suggests that the agent performs far more iterations per task than the lightweight approach.
The essence of this strategy is to use more trial-and-fix cycles to achieve a higher strict pass rate. The additional benefit is that the resulting code has undergone more thorough self-validation, so the rework cost for humans taking over is relatively lower.
4.2 Where is the gap between 82.3% completion and 24.0% pass rate?
The difference between completion score and pass rate deserves close attention. An 82.3% average completion score means that the main functionality of most tasks has been implemented and the project skeleton is largely formed. However, a 24.0% pass rate means that about three-quarters of tasks fail full acceptance.
These “not passed” tasks are likely blocked by a small number of critical issues, such as:
- Behavior differences in specific runtime environments;
- Incorrect handling of interaction details and boundary conditions;
- Compatibility scenarios missed during migration;
- UI and experience issues that are difficult to fully quantify in automated acceptance.
For teams, this means that high-completion output is not “unusable”; rather, it “needs finishing.” This type of output has value when the time required for humans to complete the remaining part is significantly less than the time required to build from scratch.
4.3 Cost rises nonlinearly with “reasoning depth”
From the published data, Agent A’s cost per run is about 32 times that of Agent B, while its time is about 2.3 times longer. The two do not scale proportionally. Cost is roughly determined by input context, output content, and the number of tool-call rounds:
- Return values from tool calls such as build, run, and log analysis continuously accumulate into the context, causing input consumption to grow rapidly with iteration rounds;
- Repeated file reads and reviews of historical decisions in long-horizon tasks further enlarge the context;
- Output length is usually positively correlated with iteration rounds, and reasoning-oriented models also have higher unit consumption for long-chain reasoning.
A simplified formula is:
Total cost ≈ Σ( input context × input unit price + output length × output unit price )+ environment resource cost (compute instances, storage, network, etc.)
where the sum is taken over tool calls and iterations.
When average completion improves from 71.2% to 82.3%, the pass rate only improves from 17.3% to 24.0%, while cost increases by an order of magnitude. This suggests that in the later stages of a project, the marginal cost of improving completion may be much higher than the marginal benefit of improving pass rate.
4.4 When is “slow and careful” worth it?
A high-completion, high-cost approach is more suitable for the following scenarios:
- Architecture-level migration: Rework is extremely expensive, so spending more time upfront to improve correctness is more worthwhile;
- Complex long-chain features: Multiple modules are involved and fast manual verification is difficult;
- Legacy projects without test protection: The agent needs to take on more self-validation responsibility;
- One-off high-difficulty tasks: Such as locating difficult defects or validating technical feasibility.
For high-frequency, batch-oriented, short-chain requirements, its time and monetary costs are difficult to sustain.
5. The value and limits of the low-cost, fast approach
Agent B achieved a 17.3% pass rate and 71.2% average completion with about $9.9 and 12.0 hours. From the perspective of output per unit cost, its advantage is clear. Based on public data:
- Cost per completion point: Each one-percentage-point increase in average completion costs about $0.14 for Agent B, while Agent A costs about $3.86, nearly 28 times higher.
- Rough labor-cost comparison: When roughly converted into human labor cost, the per-run cost difference between Agent A and Agent B is often no more than a small fraction of an engineer’s time.
Therefore, to judge whether a low-cost approach is worthwhile, the key is to calculate explicit cost plus human finishing cost:
Total task cost = tool running cost + human finishing hours × labor unit cost + rework risk cost
- If the lightweight output only needs minor human correction before delivery, its overall efficiency may be better than the expensive approach;
- If most tasks require large-scale human refactoring, the saved cost can be quickly offset.
The lightweight approach is more suitable for:
- Rapid prototyping and feature validation: Low-cost trial and error to quickly validate product ideas;
- Templated or repetitive development: Such as list pages and form pages with similar structures;
- Budget-sensitive teams: Teams that need to cover more tasks within limited budgets;
- Assistive steps embedded in CI/CD: Such as code completion, unit-test generation, and early defect screening, especially in latency-sensitive scenarios.
Its weakness is that a 71.2% completion score means nearly 30% of the work still requires humans to lead. For high-complexity tasks, teams should reserve enough time for review and rework.
6. How to build your own evaluation and selection system
Public leaderboards provide useful references, but each team’s tech stack, coding standards, and quality red lines are different. A more reliable approach is to build a reproducible internal evaluation system.
6.1 Build a private evaluation set from historical requirements
- Use real tasks: Extract tasks from historical requirements, defect tickets, and refactoring plans, covering creation, feature development, migration, and conversion.
- Make acceptance criteria verifiable: Write acceptance scripts for each task, including build, automated tests, static checks, and critical-path checks.
- Control difficulty distribution: Classify tasks into easy, medium, and hard levels based on estimated effort, avoiding a benchmark dominated by a few hard tasks.
- Keep the environment fully controlled: Fix dependency versions and runtime environments, and use one-time sandboxes to ensure reproducibility.
6.2 Metric definitions and aggregation (illustrative pseudocode)
from dataclasses import dataclass@dataclassclass RunResult:build_ok: bool # whether the build passestest_pass_rate: float # automated test pass ratelint_score: float # static-check score normalized to 0~1human_score: float # human review score normalized to 0~1wall_time: float # wall-clock time in hourscost: float # cost in the relevant currency unitattempts: int # number of submissionsdef completion_of(r: RunResult) -> float:"""Completion: measures how far the task has progressed."""if not r.build_ok:return 0.0 # if it cannot build, completion is set to zeroreturn 0.4 * r.test_pass_rate + 0.2 * r.lint_score + 0.4 * r.human_scoredef is_passed(r: RunResult, threshold: float = 0.8) -> bool:"""Passed: the build succeeds and completion reaches the threshold."""return r.build_ok and completion_of(r) >= thresholddef summarize(results: list[RunResult]) -> dict:"""Summarize leaderboard-style metrics."""n = len(results)passed = [r for r in results if is_passed(r)]return {"pass_rate": len(passed) / n,"completion": sum(completion_of(r) for r in results) / n,"avg_wall_time": sum(r.wall_time for r in results) / n,"avg_cost": sum(r.cost for r in results) / n,"cost_per_pass": sum(r.cost for r in results) / max(len(passed), 1),"rework_ratio": sum(r.attempts > 1 for r in results) / n,}
Among these, cost per passed task, obtained by dividing total cost by the number of passed tasks, reflects real cost-effectiveness better than total cost alone. The rework ratio reflects how close the approach is to one-shot correctness.
6.3 Use confidence intervals and multiple runs to resist sample fluctuation
With only 30 tasks, the sample size is small, so ranking differences need to be interpreted alongside statistical fluctuation. Assuming tasks are independent and have the same pass probability, a binomial distribution can be used:
- Standard error:
sqrt(p × (1 - p) / n), wherepis the pass rate andnis the total number of tasks; - 95% confidence interval: approximately
p ± 1.96 × standard error.
import mathdef confidence_interval(p: float, n: int, z: float = 1.96) -> tuple:se = math.sqrt(p * (1 - p) / n)return (p - z * se, p + z * se)
Using 30 tasks as the input:
- A 24.0% pass rate corresponds to approximately 12%–36%;
- A 17.3% pass rate corresponds to approximately 7%–27%;
- A 28.7% pass rate corresponds to approximately 15%–42%.
These intervals overlap with each other, indicating that at a scale of 30 tasks, the differences have not reached statistical significance. It is recommended to run each configuration at least 3 to 5 times, report the mean and dispersion, and compare results stratified by task difficulty.
6.4 Route tasks by complexity
Instead of choosing only one approach among multiple options, a more practical strategy is to establish a difficulty grading plus dynamic routing mechanism:
| Task characteristic | Recommended strategy | Reason |
|---|---|---|
| Small single-file change with clear rules | Lightweight approach, no deep reasoning | Fast and cheap, with low human-verification cost |
| Cross-module feature development | Try lightweight first, then upgrade after failure | Balances cost and success rate |
| Architecture migration or platform conversion | Deep approach with time and budget limits | Rework is expensive and thorough validation is needed |
| Core-path or security-related changes | Deep approach plus mandatory human review | Quality red lines cannot be left to automation |
Example routing logic:
def choose_mode(task) -> str:if task.category in ("migration", "conversion"):return "deep"if task.category == "feature" and task.affected_files > 5:return "light_then_deep" # try lightweight first; upgrade if it failsif task.category == "create":return "deep" if task.has_risky_dependency else "light"return "light"
Example routing outcomes:
Task: Develop a user login page (feature, 3 files changed)Route: light → passed → cost 0.6 in the example cost unit, time 4 minutesTask: Migrate an old list component to the new architecture (migration)Route: deep → not passed → manual completion took 25 minutes
6.5 Engineering guardrails for controlling cost and risk
- Budget and timeout circuit breakers: Set a cost cap and wall-clock timeout for each task. Terminate immediately when thresholds are reached, and preserve intermediate artifacts for human takeover.
- Sandbox isolation: Run agents in independent, disposable environments, and never use production credentials.
- Human-agent collaboration loop: Establish a workflow of agent output → automated validation → human review → feedback accumulation, and turn typical issues found during human review into new acceptance cases.
- Observability: Record time, cost, iteration count, and failure reasons for each run, forming a continuously comparable evaluation baseline.
- Keep the evaluation set updated: Since OS versions and dependency libraries continue to evolve, the evaluation set should evolve as well to avoid disconnecting benchmark tasks from real engineering work.
7. Conclusion
Returning to the two result sets at the beginning: the higher-pass-rate approach spent tens of times the cost and more than twice the time to achieve higher completion and pass rates, while the low-cost approach covered a larger task range with faster feedback.
In long-chain, verification-heavy engineering scenarios such as mobile app development, there is no universally optimal approach. There is only a combination that matches team needs. Teams are advised to:
- Build a private evaluation set that matches their own tech stack;
- Track pass rate, completion score, time, and unit cost together;
- Use confidence intervals and multiple runs to avoid being misled by sample fluctuation;
- Implement task routing based on difficulty;
- Use total task cost, rather than single-call price, as the decision basis.
At present, cloud vendors such as Baidu are also continuing to advance code generation and intelligent programming capabilities. For developers, building their own evaluation and selection system is the fundamental way to keep up with rapid technological iteration and continuously gain productivity improvements.
