Interpreting Benchmark Results for AI Coding Agents in Mobile App Development
This article explains four benchmark metrics for AI coding agents in mobile system development: full pass rate, average completion, execution time, and cost per run. It compares a high-capability approach with a lightweight one, and describes how to design internal evaluations and layered workflows for real engineering teams.
As coding agents move into real development workflows, teams care less about whether an agent can generate code and more about whether it can finish complex tasks consistently. In mobile system development, a single task often involves understanding the project, adjusting the UI, changing business logic, handling dependencies, verifying the build, fixing tests, and iterating over several rounds. Accuracy on a single code generation step no longer reflects an agent’s real engineering ability.
A recent benchmark for mobile system development published new results. It compared two different coding agent approaches on long-running tasks. Across 30 tasks, one approach reached a 24.0% full pass rate and 82.3% average completion. It averaged 27.4 hours per task and cost about US$317.4 per benchmark run. The lightweight approach reached a 17.3% full pass rate and 71.2% average completion. It averaged 12.0 hours per task and cost about US$9.9 per run.
The main value of these results is not in declaring a winner. They show the typical trade-offs among capability, execution time, and cost.
1. What the Four Metrics Measure
1. Full pass rate: measuring delivery reliability
The pass rate usually indicates whether a task meets all acceptance criteria set by the benchmark. It does not measure whether code was generated. It measures whether the final result passes compilation, tests, functional checks, and the preset constraints.
In mobile development, a full pass typically means:
- the project builds successfully;
- key features run as the requirements specify;
- UI interactions behave as expected;
- automated tests or acceptance scripts pass;
- no obvious regressions are introduced;
- the scope of changes matches the task requirements.
The pass rate is therefore the metric closest to “ready to ship.” Used alone, however, it has limits. A sample of 30 tasks is small, a few complex tasks can strongly affect the overall result, and results depend on task type, initial code quality, and test coverage.
2. Average completion: measuring partial success
Average completion describes how much useful work an agent finished, even when a task did not fully pass. An agent might complete the page layout and data processing but leave edge cases unhandled. It might implement the core logic while the build still has dependency or configuration problems.
For real development, completion is a useful reference. Enterprise tasks are not always all-or-nothing. An agent can often finish most of the routine work, and developers then handle a small number of difficult problems.
Completion scores require clear scoring rules, though. If tasks carry different weights, a simple average can hide critical defects. A 90% completion on a minor UI tweak carries very different risk from 90% completion on a payment flow. Internal evaluations should therefore weight security, data consistency, and core business flows more heavily.
3. Average time: measuring end-to-end efficiency
Time on long-running tasks covers more than code generation. It includes reading the project, running builds, executing tests, analyzing errors, and repeating fixes.
A longer execution time can reflect stronger iterative capability. It can also point to:
- inefficient error localization;
- too many repeated edits;
- poor use of test feedback;
- an incomplete understanding of the project structure;
- excessive time spent on low-value steps.
Shorter time is therefore not automatically better. If an agent completes little in 12 hours and still needs heavy manual rework, its real productivity may not exceed that of a slower agent that delivers higher-quality work.
4. Cost per run: measuring the barrier to scale
Cost per benchmark run reflects the compute, model calls, execution environments, and other resources needed to complete a set of tasks. High-cost setups tend to use more reasoning steps, longer context, or more elaborate verification. Low-cost setups usually prioritize fast responses and resource efficiency.
Cost matters especially for enterprises. For occasional high-value tasks, a higher cost may be acceptable. For daily runs, bulk migrations, automated fixes, or continuous integration, costs grow quickly with task volume.
2. Core Differences Between the Two Approaches
The high-capability approach reached a 24.0% pass rate and 82.3% average completion. These figures suggest strong performance in breaking down complex tasks, making cross-file changes, and running multiple verification cycles. Its average of 27.4 hours per task and roughly US$317.4 per benchmark run, however, indicate that it fits high-difficulty, low-frequency, high-value work best.
The lightweight approach reached a 17.3% pass rate and 71.2% average completion, which trails the first approach. Its average time was 12.0 hours per task, and it cost about US$9.9 per run. Where budget and turnaround time are tight, this approach is more attractive. For code search, simple feature adjustments, test completion, documentation sync, and low-risk refactoring, it may offer a better return on investment.
These results show that coding agents do not have an absolute ranking that holds across all scenarios. A more practical approach is to tier work by risk and complexity:
- High-risk core features: prioritize pass rate and verification capability.
- Medium-complexity iteration: focus on completion and the cost of manual takeover.
- Large-scale repetitive tasks: focus on cost per task.
- Urgent fixes: weight end-to-end time more heavily.
- Exploratory development: accept lower pass rates, but enforce isolation and rollback.
3. Why Long-Running Mobile Tasks Are Harder
Mobile engineering usually involves multiple layers of technical dependency. A request that looks simple may affect the UI layer, state management, network communication, local data, permissions, and build scripts at once.
First, project context is complex. An agent must understand the directory structure, module boundaries, resource files, configuration files, and existing coding conventions. A change that is correct within a single file can still break the system as a whole.
Second, the validation chain is long. After code changes, the project must be compiled, installed, run, and tested. Some issues appear only under specific device states, lifecycle transitions, or abnormal network conditions.
Third, tasks often carry implicit constraints. New features must not break existing behavior, interfaces must adapt to different screen sizes, data reads and writes must stay compatible, and error messages should follow established interaction patterns. These constraints may not all appear in the written requirements, yet they directly affect acceptance results.
Fourth, failed repairs compound. One incorrect change can trigger new build errors. If the agent does not identify the root cause accurately, it may keep patching and widen the change scope, which raises maintenance costs.
4. How Enterprises Can Design More Reliable Evaluations
Before adopting coding agents, teams should not rely only on public leaderboards. Enterprises should build internal evaluation sets that reflect their own business. These sets should cover at least the following task types:
Basic modification tasks
Include page field changes, resource replacement, simple logic updates, and test additions. These tasks measure basic code-editing ability.Cross-module development tasks
Require changes across the UI, business logic, and data access layers. These tasks test project understanding and coordination.Bug-fixing tasks
Provide real or anonymized bug cases. Observe whether the agent can reproduce the issue, locate the root cause, and verify the fix.Compatibility tasks
Cover different system versions, screen sizes, network states, and lifecycle scenarios to assess how the agent handles edge cases.Refactoring and migration tasks
Focus on controlling the size of changes, preserving interface compatibility, and limiting regression risk.
Evaluations should record more than whether a task finally passed. They should also log process data:
- time to first successful compilation;
- time to first passing test run;
- number of failures and their types;
- number of modified files;
- number of manual interventions;
- time spent on final manual fixes;
- issues found during code review;
- actual resource consumption per task.
Agent time plus manual repair time often reflects real efficiency better than agent time alone.
5. Recommended Layered Collaboration Workflow
In practice, teams can use a three-stage workflow: a lightweight agent screens tasks first, a higher-capability agent reviews the remaining work, and a developer makes the final acceptance decision.
In the first stage, a lightweight agent handles low-risk tasks with clear boundaries, such as adding tests, generating boilerplate code, consolidating duplicate logic, or modifying static configuration. This work should happen on a separate branch or in an isolated workspace to keep the main codebase unaffected.
In the second stage, tasks that failed, affect a wide range of code, or involve critical modules go to a stronger agent. Provide complete build logs, test results, and records of earlier changes so the agent does not repeat earlier exploration.
In the third stage, developers review and accept the code. They should focus on data security, exception handling, permission boundaries, performance impact, and compatibility. For changes to core business logic, passing automated tests does not replace human review.
This layered approach controls cost without sacrificing quality, and it avoids running every task in a high-cost mode.
6. Risks When Using Benchmark Results
Public benchmarks offer directional guidance, but they do not directly predict production performance. Reasons include:
- the task set may differ substantially from an enterprise’s real projects;
- test coverage is limited;
- averages can hide extreme failure cases;
- costs depend on context length, execution environment, and call strategy;
- agents may have different tool permissions or run configurations;
- pass
