AI Coding Agent Benchmarks for Mobile Apps: High Completion Rates and Long Runtimes Remain the Key Tradeoff
This article explains how AI-assisted coding has evolved from code completion to long-horizon software delivery, why native mobile app development is a stringent benchmark for coding agents, and how pass rate, completion score, runtime, and cost should be interpreted. It also shows how engineering teams can build internal evaluations, integrate agents into delivery pipelines, and apply security controls.
Introduction: Coding tool competition is shifting from writing correct code to delivering working software
Over the past few years, AI-assisted programming has gone through two major positioning shifts:
- Stage one was code completion: intelligent coding tools predicted the next line of code, and their value mainly came from reducing typing effort.
- Stage two was conversational coding: tools became capable of understanding requirements, generating functions or entire modules, and explaining or modifying existing code.
- The current stage is long-horizon task execution: given a natural-language requirement and an existing repository, a tool must explore the project structure, determine the scope of changes, edit code file by file, and go through repeated compile-run-test-fix loops before delivering a feature that can be built and run.
Official developer channels for major mobile operating systems have recently published AI coding benchmark results for native application development, including results from agent-based systems. The leaderboard shows that top performers can complete nearly 30% of long-horizon tasks end to end. At the same time, more than 70% of tasks are not fully completed. Higher completion and longer runtime do not mean that human review is unnecessary.
As a result, developers are asking more practical questions: How should these benchmark scores be interpreted? What engineering tradeoffs do different tool routes reveal? And how should teams build their own evaluation systems?
Why mobile app development is a litmus test for coding agents
Compared with pure backend services, native mobile application development has several engineering characteristics that directly target some of the hardest problems for AI coding tools:
Strong environment dependency. Correct code does not automatically mean a working application. A project must pass dependency resolution, compilation, resource packaging, and signing checks before it can be installed on an emulator or physical device and run. An agent must complete the full loop of modifying code, building, installing, and launching. If any step fails, earlier work may become unusable.
Complex state and lifecycle management. Switching between foreground and background, rotating between portrait and landscape, granting or denying permissions, and recovering after process termination can all change UI state. If an agent only performs static code reasoning, it is easy to miss edge cases.
System version and device fragmentation. Differences across OS versions, screen sizes, pixel densities, and vendor-customized system behavior require agents to consider compatibility and adaptation, rather than merely making something run on one local machine.
UI acceptance is inherently subjective. Whether a feature is correct often depends on visible interface behavior and interaction flow, which is much more complex than a unit test assertion.
Therefore, a strong mobile development benchmark must automate build systems, device runtime environments, and interaction validation. This places high demands on an agent’s ability to close the full engineering loop.
How benchmarks define success: task categories and scoring methods
Four task categories reflect real development workflows
These benchmarks are designed around four common types of native app development work:
- App creation: start from an empty project, build pages, navigation, and a basic data layer, and produce an installable and runnable application.
- Feature development: add new features or screens to an existing project without breaking existing behavior.
- Code migration: move code across different system versions, UI component systems, or language conventions.
- App conversion: transform one application form into another, such as rewriting a traditional view system into a declarative UI, which often involves deep cross-file refactoring.
These four categories increase in difficulty. Conversion tasks usually require the largest code changes and have the most cross-file dependencies, so they often determine the overall pass rate.
Three scoring methods: binary pass, completion score, and cost/runtime
Binary pass determination: a task passes only when the full automation pipeline succeeds. This usually requires three conditions: the project builds successfully, the application installs and launches normally, and all validation cases corresponding to the task goal pass.
Pass rate = passed tasks ÷ total tasks.Completion score: a 0 to 100 score is used to describe how much of a task has been completed, often by converting achieved outcomes into a percentage. It captures the value of partially usable results and helps identify tasks that are only missing the final step.
Cost and runtime: these metrics separately measure the model invocation cost for a single run, often converted to local currency at the current exchange rate, and the average time from start to finish.
These three types of metrics are complementary. Looking only at pass rate can hide code that is almost usable. Looking only at completion score can overestimate the proportion of results that are truly production-ready. For this reason, mature teams usually interpret four dimensions together: pass rate, completion score, runtime, and cost.
Why pass rate is the core metric
Mobile app development emphasizes installable, runnable, and releasable software. Any intermediate state has little value to users. Engineering acceptance is inherently binary: the build must pass, behavior must be correct, and defects must be fixed.
In long-horizon tasks, pass rate best answers the question that managers care about most: “If I assign these tasks to the agent, how many will I not need to redo?”
Key variables that affect benchmark results
Scores on the same task set can be affected by the following factors. When comparing results over time, these conditions must remain consistent:
- Agent orchestration method: whether there are explicit planning, retrieval, and self-validation stages can significantly affect success rates on long-horizon tasks.
- Execution environment and tool permissions: whether the agent can run emulators, perform full builds, and how many steps or how long a single run can take can all change outcomes.
- Model inference configuration and randomness: repeated runs with the same configuration may produce different results, so a single run should not be treated as a stable level of performance.
- Task difficulty distribution and version: changing the model or agent version may change scores, but the difference cannot always be attributed directly to one side.
Engineering tradeoffs behind benchmark results: long runtime for completion or high throughput at low cost?
Profiles of two typical approaches
Without tying the analysis to a specific product, the two typical approaches on the leaderboard can be summarized as follows. The table values are approximate and intended only for illustration.
| Metric | Scheme A: high-completion approach | Scheme B: high-throughput, low-cost approach |
|---|---|---|
| Agent form | Directly driven by a coding agent | Integrated through a general-purpose agent framework |
| Task scale | 30 long-horizon tasks | Same task set |
| Pass rate | About 24% | About 17% |
| Average completion | About 82.3% | About 71.2% |
| Average runtime | About 27.4 hours | About 12.0 hours |
| Cost per run | About US$317.4, about CNY 2,123 | About US$9.9, about CNY 66.2 |
From a cross-comparison perspective, Scheme A’s pass rate is higher than several earlier leaderboard entries and close to the top score of 28.7%. Scheme B’s pass rate is lower than Scheme A, but its runtime is about half and its cost is about 1/32.
Seven key conclusions
There is a gradient between pass rate and completion score. An average completion score of 82.3% means many tasks that do not pass are still near completion. On a linear scale, this is equivalent to about 18 fully completed tasks out of 22. This is where the value of long runtime appears: the agent has enough time to complete the build-run-fix loop and push tasks toward the finish line.
High completion is the result of both capability and strategy. A longer execution budget allows an agent to explore the project more thoroughly and validate repeatedly. Coding-focused agents often have workflows designed to match this style of work. Runtime is the cost; completion is the output.
Runtime is a hidden cost. If five long-horizon tasks are run in parallel, the per-run cost of US$317 may be amortized. However, an average runtime of 27.4 hours means that a single task can only complete one round within a workweek, and long-running execution depends on reliable sandbox environments and monitoring alerts.
The leverage of low-cost approaches is throughput. A unit price of US$9.9 and an average task runtime of 12 hours make this approach suitable for batch execution in continuous integration pipelines. A pass rate of 17.3% means roughly 80% of tasks still require human finishing, but it can push many tasks to 71.2% completion at very low cost, acting as a draft generator for human engineers.
On the same task set, the two approaches are not substitutes. They are constrained by the same benchmark and face the same tasks, but one goes deeper while the other goes broader. The high-completion approach is better suited to one-shot transformation work, while the high-throughput approach is better suited to frequent, quickly validated daily iterations.
Cost should be viewed in terms of human labor. US$317, about CNY 2,123, may be dozens of times the cost of small manual tasks measured in tens or hundreds of yuan. If it can replace several days or more of engineering work and the output is truly deliverable, the cost per task may be lower than the equivalent human labor cost.
Models and agents should not be treated as the same thing. Benchmark results are a combined outcome of model + agent + environment + task set. Attributing Scheme A’s results entirely to the model itself is not rigorous. Teams should treat the agent’s orchestration strategy as a first-class factor during evaluation.
Quantifying the tradeoff: a decision inequality and scenario comparison
A simple decision inequality can summarize the selection logic:
When the labor hours required to complete a task manually × team labor cost per hour > solution runtime cost + human finishing hours × labor cost per hour + sunk cost of failed tasks, introducing the solution is economically justified.
Based on this, teams can compare scenarios as follows:
| Scenario | Recommended approach | Reason |
|---|---|---|
| Low-frequency, high-difficulty tasks such as framework upgrades or cross-module refactoring | High-completion, long-runtime approach | Tasks can be queued offline, human finishing costs are high, and failure losses are large |
| Daily feature iteration and boilerplate code generation | High-throughput, low-cost approach | Can be embedded in pipelines for batch execution, and failure costs are low |
| Code review and defect localization | High-throughput, low-cost approach | High concurrency and low unit price enable broad coverage |
| Teams sensitive to budget and response time | Low-cost approach first, with human fallback | Trades machine cost for human efficiency |
| Critical path modules with zero tolerance for delivery quality issues | Human-led, AI-assisted | Pass rates are still around 10% to 30%, so human review is essential |
From leaderboards to teams: building internal evaluations and implementation practices
Five steps to build an internal evaluation
Public benchmark task sets are often based mainly on English-language community projects and may differ from a team’s own engineering practices. Teams should use public leaderboards as references rather than copying them directly. A practical internal evaluation can follow these five steps:
- Select tasks: choose 10 to 30 already merged tasks from real repository history, covering the four task categories and different levels of complexity.
- Establish a baseline: have senior engineers complete the tasks independently and record the labor hours as a human baseline.
- Fix the evaluation process: create automated build and behavior validation flows for each task to ensure that scoring criteria are objective and repeatable.
- Run tests: fix the model version, agent configuration, and environment specifications. Run each task at least three times, and record pass rate, completion score, runtime, cost, and the distribution of failure reasons.
- Review results: analyze whether failures are concentrated in build, runtime, or logic layers, then adjust task prompts, environment configurations, or agent strategies accordingly.
Integrating AI into the development process
- Code hosting: the agent submits changes on independent branches, preserving complete change records for review and rollback.
- Continuous integration pipelines: connect build, test, packaging, and distribution into a closed loop. Add stage timeouts and failure circuit breakers to prevent long-running tasks from occupying all execution resources.
- Containerized execution environments: provide each task with an isolated runtime and emulator environment so that tasks do not interfere with each other and can be reset quickly.
- Artifact and log archiving: store build artifacts, runtime logs, and failure screenshots in object storage so that every result is traceable and reproducible.
- Monitoring, alerting, and asynchronous scheduling: use message queues to dispatch long-running tasks, and use monitoring and alerts to track execution status and cost consumption. If anomalies occur, notify human engineers to intervene promptly.
For example, Baidu AI Cloud provides general engineering support capabilities such as code hosting, pipelines, containers, object storage, logging services, monitoring alerts, and message queues. These capabilities can cover many of the steps above and help teams integrate AI coding tools into existing development systems more smoothly.
Security controls: allowing AI to write code while keeping it controllable
- Credential isolation: grant the agent’s runtime environment only the minimum temporary credentials required, and prohibit access to production resources or sensitive data.
- Trusted dependencies: require locked dependency versions and perform source verification and license scanning for third-party libraries introduced by the agent.
- Auditable artifacts: all AI-generated code must pass human code review before merging into the main branch, especially in modules related to permissions, payments, or privacy.
- Traceable behavior: keep complete operation logs and cost details for the agent so they can be used for post-event audits and optimization.
Conclusion: turning benchmark scores into judgment
Returning to the leaderboard: 24% versus 17.3% pass rates, 82.3% versus 71.2% completion scores, 27.4 hours versus 12.0 hours runtime, and US$317.4 versus US$9.9 cost. These numbers are not conclusions about which option is stronger. They are snapshots of engineering tradeoffs under specific constraints.
For developers, the more important takeaway is how to extract judgment from the leaderboard:
- Look at pass rate, not only completion score. Pass rate determines the lower bound of human effort and is the first gate for practical adoption.
- Calculate total cost, not only unit price. Add runtime cost, execution cost, and human finishing hours, then compare them with the human baseline.
- Select tools by scenario, not with a one-size-fits-all rule. Long-runtime, high-completion approaches and low-cost, high-throughput approaches can coexist in the same development pipeline.
- Build your own evaluation instead of copying the leaderboard. Only benchmarks aligned with your own codebase and team habits have real decision value.
As mobile development benchmarks continue to include more realistic and complex engineering tasks, AI coding tools are moving from assisting with code completion to taking on actual work. Teams that first establish scientific evaluation systems and safe, controlled usage rules will gain an advantage in the next round of development efficiency competition.
