Mobile Development AI Coding Benchmarks Update: The Engineering Tradeoffs Behind the Scores
A new round of results on a mobile app development benchmark shows leading agents clustered within a few percentage points on pass rate while differing by an order of magnitude in cost and runtime. This article breaks down what each metric actually measures (pass rate, average completion, wall-clock time, per-run cost), explains why mobile evaluation is unusually expensive, and contrasts a high-completion, long-running profile (A) with a low-cost, fast profile (B). It then covers the practical side: building an internal task set, describing tasks as structured configuration, comparing options with a weighted scoring function, a cloud execution architecture, and delivery metrics that matter more than benchmark rankings.
1. Background: Why Mobile Development Has Become a New Benchmark Focus
Recently, the official developer account of a major mobile OS vendor published a batch of AI model results on the same mobile app development benchmark. Benchmarks of this kind, typically released by industry bodies or platform owners, contain dozens of long-horizon tasks that cover realistic engineering scenarios: app creation, feature development, code migration, and app conversion. Unlike code-completion or single-function evaluations, they require an agent to complete the full loop:
- Plan autonomously
- Modify code across multiple files
- Build and verify repeatedly
- Deliver a runnable result within a fixed time and budget limit
Three signals stand out from this round.
First, the gap between top performers is narrowing. The highest pass rate is about 28.7%, with multiple options clustered in the 17%-24% range — a spread of only a few percentage points. Baseline coding ability is converging quickly, which means “who can write code” alone is no longer a differentiator.
Second, the gap in time and cost is far larger than the gap in capability. Some options cost close to $320 per run and take more than a day; others cost under $10 and finish in about half a day. Their pass rates differ by only a few percentage points, yet their resource consumption differs by an order of magnitude.
Third, different options show different engineering “personalities.” Some pursue high completion and are willing to run autonomously for long stretches; others optimize for low cost and fast feedback, which makes timely human intervention easier.
For the discussion below, we label two typical profiles A and B. Figures are single-run values (converted to USD at the exchange rate in effect when the results were published):
| Profile | Pass rate | Avg. completion | Avg. time | Cost per run (approx.) |
|---|---|---|---|---|
| A: high completion, long runtime | 24.0% | 82.3% | 27.4 hours | $317.4 |
| B: low cost, fast pace | 17.3% | 71.2% | 12.0 hours | $9.9 |
| Reference: current highest pass rate | 28.7% | Not published | Not published | Not published |
Comparing side by side:
- A’s pass rate is about 6.7 percentage points higher than B’s, roughly 1.4x B;
- A’s average completion is about 11 percentage points higher;
- but A costs about 32x more per run and takes about 2.3x as long.
A also trails the highest-pass-rate option by about 4.7 percentage points, and its gap to B is smaller than the cost and time premium A itself pays. This raises a key question: for long-horizon engineering work like mobile development, how should teams trade “finishing more” against “spending less”?
One caveat up front: benchmark scores are strongly tied to the specific task set, acceptance scripts, and runtime environment, so they hold only under the stated conditions. What engineering teams actually need is not to copy a ranking but to build their own evaluation criteria and selection method.
2. Reading the Evaluation: What Each of the Four Metrics Means
2.1 Pass Rate: Did It Actually Ship End to End?
Mobile development benchmarks typically use a binary verdict: does the build succeed, do all tests pass, do the critical scenarios run correctly. If any step fails, the task counts as failed.
Pass rate therefore answers a strict question: with no human intervention, can the agent independently complete this class of task? It depends on three factors:
- Model capability;
- The difficulty distribution of the task set;
- How strict the acceptance scripts are.
Pass rate is the core measure of autonomous delivery, but it should never be used alone. Two evaluations with different acceptance criteria produce pass rates that cannot be compared directly.
2.2 Average Completion: The Residual Value of Failed Tasks
Failed tasks are not necessarily worthless. Average completion measures the progress ratio from task start to task end, describing how far an agent got on the tasks it failed.
Take A: its average completion is 82.3%. If you read completion as “how far is the task from done,” then on average its failed tasks are less than 20% away from closure. In real development, such near-finished work can substantially reduce the effort of human handoff: an engineer reviews the changes, fills in the edge cases, and runs the tests, instead of rewriting from scratch.
Completion has two limitations, though:
- It depends on how the evaluation defines “progress,” and definitions may differ across benchmarks;
- High completion can come with redundant changes. Code bloat, duplicated implementations, and unnecessary dependencies all raise maintenance cost later.
Completion scores should therefore be read together with code review findings and regression test results.
2.3 Average Time: Wall-Clock Time in Long-Horizon Tasks
Time here is the real elapsed duration from task start to finish, including model inference, tool invocation, compilation and builds, test execution, and waiting for feedback.
In single-step code completion, second-level differences barely matter. In long-horizon tasks that routinely run for a dozen hours or more, time directly determines how the task can be scheduled:
- Whether it fits into unattended overnight batch processing;
- Whether multiple iteration rounds can finish before a submission deadline;
- Whether it will compete with the rest of the team for shared build resources.
Long tasks also demand careful context management. As the task progresses, context keeps accumulating and inference rounds increase, so the cost of each additional step tends to be higher in the later stages than in the earlier ones.
2.4 Per-Run Cost: The Sum of Prompts, Tool Calls, and Retries
Per-run cost is the total resource spend for one complete evaluation round. It has three main components:
- Model inference consumption;
- Tool invocation and verification overhead;
- Retry overhead after failures.
Once a long-horizon task enters repeated trial and error, cost can rise in steps. So when comparing costs, always confirm that tasks ran under the same budget and time ceiling — otherwise a “low cost” may just mean the task was terminated early.
2.5 Reading Rule: Interpret All Four Metrics Together
The four metrics form a system of mutual constraints:
- Raising completion usually costs more time and money;
- Cutting cost usually means converging earlier or trying less;
- Shortening time may sacrifice handling of edge cases.
The right order is: first check whether the task type and acceptance criteria are comparable, then check whether the overall profile matches your needs, and finally measure value with “cost per unit of effective delivery” rather than “cost per run.”
3. Why Mobile AI Evaluation Is Unusually “Heavy”
3.1 Long Build-and-Verify Chain
A single mobile feature typically goes through:
- Writing code and resources;
- Compiling and packaging;
- Signing;
- Installing to an emulator or physical device;
- Launching and executing the interaction flow.
A failure at any step can fail the task. On top of that, the waiting time for build, install, and launch is itself significant. This partly explains why slow-profile options average more than a day: much of the time goes into verification, not generation.
3.2 Environment Fragmentation and Combinatorial Explosion
System versions, screen sizes, permission policies, and vendor-specific customizations all differ. The same code that behaves correctly in one environment may crash in another.
A serious mobile evaluation usually verifies across multiple environments, so verification cost grows with the number of environment combinations. The “verifiability” a benchmark score depends on is itself an expensive engineering investment.
3.3 Hard End-to-End Verification, Sparse Feedback Signals
Unlike backend services, where unit tests and API tests can cover behavior, much of a mobile app’s behavior happens in the UI:
- Screen navigation;
- Gesture input;
- Permission dialogs;
- Background/foreground switching.
These are hard to cover fully with a handful of assertions. The sparser the feedback signal, the harder it is for an agent to tell whether it is heading in the right direction — and the easier it is to fall into an unproductive loop of “it looks changed, but nothing actually took effect.”
3.4 State Drift in Long-Horizon Tasks
The longer the task chain, the more state the agent has to maintain:
- The original requirement;
- Files already modified;
- Approaches already tried;
- Problems still unresolved.
As context grows, early constraints get diluted, leading to drift from the requirement, repeated modifications, or regression of work already completed. Pass rates on long-horizon tasks are clearly lower than on single-step tasks, and state drift is one of the main reasons.
3.5 The Distinct Challenge of Code Migration and App Conversion
Code migration and app conversion (for example, moving across platforms or frameworks) are hard because they require semantic mapping, not just syntax translation. Different tech stacks often differ structurally in:
- UI description: declarative vs. imperative;
- Lifecycle management;
- State management;
- Async and concurrency models.
These tasks put an agent’s long-horizon planning ability on full display: it must understand the original implementation, rebuild equivalent behavior in the new stack, and pass verification. An agent’s performance on this class of task usually says more about its real capability than performance on “add a new feature” tasks.
4. The Core Tradeoff: High Completion, Long Runtime vs. Low Cost, Fast Pace
4.1 The Real Difference Between the Two Profiles
The difference is not just speed; it is a different optimization target:
- Profile A is quality-first. It tends to spend more rounds verifying, fixing, and polishing until completion is high enough. The price is a long time span and a high per-run cost.
- Profile B is efficiency-first. It produces usable results quickly and cheaply, which suits fast trial and error, parallel exploration, and timely human course correction.
4.2 When to Prefer Quality-First
Favor Profile A in these situations:
- High-risk, hard-to-roll-back changes: features on core paths such as data persistence, permissions, payments, and accounts.
- Highly automated verification: bulk code migration or dependency upgrades where test coverage is strong — this is where long autonomous execution yields the most marginal benefit.
- Asynchronous, unattended work: overnight builds and batch refactors, where elapsed time is not the bottleneck and final quality is what matters.
- Teams where human cost is much higher than compute cost: reducing human rework matters more than saving on invocation fees.
4.3 When to Prefer Efficiency-First
Profile B fits better here:
- Exploratory requirements: the approach is not settled yet and several prototypes need to be produced quickly for comparison.
- Small, well-bounded changes: copy changes, style fixes, simple logic completion.
- Budget-sensitive or large-scale parallel work: many low-complexity tasks at the same time.
- Interactive development with a human in the loop throughout: the developer is right there and can correct course immediately, so fast feedback is worth more than a single high-completion output.
4.4 The Key Variable: the Value of Human Time
The most overlooked aspect of long-running options is whose time they occupy.
Consider B’s 12 hours: that spans more than half a working day, so developers can hardly walk away — they end up checking progress frequently and preparing to take over. A takes about 2.3x longer, but as long as the task can be scheduled overnight, the actual human effort may be lower.
So evaluate options beyond machine cost, and account for the hidden cost of human waiting and intervention. In one line: letting the machine do more work only pays off if the human isn’t chained to the screen.
4.5 Recommended Strategy: Layer Tasks and Combine Both
In practice, the better answer is not either/or but layering by task:
- Exploration layer: use the low-cost, fast option to generate multiple candidate implementations.
- Implementation layer: use the high-completion option for deep implementation and fixing on the chosen candidate.
- Closure layer: have a human or an automated pipeline handle final review, testing, and release.
Set budget caps and timeout policies per task stage as well, so a single long-running task cannot spiral out of control.
5. Putting It into Practice: From an Internal Baseline to a Cloud Execution Architecture
5.1 Build Your Team’s Own Task Set
Public benchmarks only offer directional guidance. Teams should sample tasks from their real work and build an internal evaluation set. Recommended coverage:
- New page or feature development;
- Bug fixes;
- Dependency and version upgrades;
- Code migration and cross-platform adaptation;
- Refactoring and code-health work.
Every task needs clear acceptance criteria that can run automatically. For example:
- Does the build succeed;
- Do unit tests pass;
- Can key UI flows be completed;
- Does the scope of changes match expectations.
5.2 Describe Tasks as Structured Configuration
To keep evaluation reproducible and auditable, manage tasks as configuration files. The example below shows how a migration task can be described:
task_id: migrate_settings_formcategory: code_migrationdescription: >Migrate the settings page from a list-based implementation to a groupedform implementation, preserve all configuration items and default values,and add the corresponding tests.constraints:timeout_hours: 6 # hard timeout so long-running tasks cannot run awaybudget_usd: 15 # per-run cost ceilingprotected_paths: # directories that must not be touched, prevents "drive-by" refactors- "src/core/auth/**"acceptance:build: ./scripts/build.sh # does the build succeedtest: ./scripts/test.sh # do the unit tests passui_smoke: ./scripts/ui_smoke.sh # can key UI flows be completedmax_changed_files: 20 # cap on changed files to keep the change scope under controlgrading: # weighted scoring separates "full pass" from "partial completion"build: 0.2test: 0.4ui_smoke: 0.3scope: 0.1
This configuration delivers three benefits:
- Explicit constraints: timeouts and budget caps prevent unbounded consumption.
- Core code protection: protected directories stop agents from changing what they should not.
- Pass rate and completion measured separately: tiered scoring supports a strict pass rate while still quantifying the value of partial completion.
5.3 Quantify the Decision with a Weighted Function
Putting pass rate, completion, time, and cost into one model turns option selection from guesswork into a calculation. Illustrative code:
def utility(m, weights, cost_budget, time_budget):"""Compute an overall score for a task execution profile.m: {"pass": pass rate, "progress": avg completion, "hours": runtime, "cost": cost}Once runtime or cost exceeds its budget, no further credit is given,so a large overspend cannot be traded for a marginal gain."""cost_ratio = min(m["cost"] / cost_budget, 1.0)time_ratio = min(m["hours"] / time_budget, 1.0)return (weights["pass"] * m["pass"]+ weights["progress"] * m["progress"]+ weights["speed"] * (1.0 - time_ratio)+ weights["cost"] * (1.0 - cost_ratio))# Scenario 1: overnight unattended batch migration, quality and completion firstw_batch = {"pass": 0.40, "progress": 0.25, "speed": 0.15, "cost": 0.20}# Scenario 2: interactive day-to-day development, speed and cost firstw_daily = {"pass": 0.20, "progress": 0.20, "speed": 0.30, "cost": 0.30}
Weights are not a fixed formula; they are a quantified expression of the team’s priorities:
- Teams with ample budget and tight delivery windows can raise the weight on quality and completion;
- Cost-sensitive teams handling small, fine-grained tasks should raise the weight on speed and cost.
The point is to use different weights for different scenarios within the same team, rather than judging every task by a single standard.
5.4 Cloud Execution Architecture: Isolation, Asynchrony, Observability
Long-horizon AI coding tasks share three traits: a single run takes a long time, resource consumption fluctuates a lot, and retries are frequent. Running them on developers’ local machines creates a set of problems:
- Tasks block each other;
- Environments drift out of sync;
- There is no record of the process.
A more robust approach is a unified execution environment in the cloud:
- Containerized sandboxes: an isolated runtime per task, with pinned dependencies and reproducible results, so tasks cannot contaminate one another.
- Object storage: keeps code snapshots, build artifacts, and complete execution traces for replay and audit.
- Message queues and task orchestration: queue large numbers of tasks asynchronously, dispatch them to execution nodes, and support timeouts and failure retries.
- Logging and monitoring with alerting: continuously collect per-task runtime, spend, and failure reasons, feed a cost dashboard, and surface abnormal consumption quickly.
- Model serving and elastic compute: route to different capability tiers through a unified model access and inference layer, and use elastic containers and function compute to absorb traffic peaks and troughs instead of permanently reserving capacity for long-tail tasks.
On Baidu AI Cloud, these capabilities are typically offered through general-purpose products such as container platforms, object storage, message queues, logging, and monitoring with alerting. Teams can use them to build their own AI coding evaluation and execution platform, closing the loop of evaluate, execute, measure, optimize.
5.5 Metrics: From Benchmark Scores to Effective Delivery
Beyond public benchmarks, teams should add these engineering metrics:
- Human takeover rate: the share of tasks that require substantial manual edits before they can be merged;
- Rework rate: the share of generated results that fail on first review and must be regenerated;
- Merge cycle time: average time from task dispatch to code landing on the main branch;
- Cost per effective delivery: total cost / number of requirements (or tasks) successfully merged;
- Regression defects: issues introduced by AI-generated code that surface in later testing or in production.
These metrics sit closer to business value than a single benchmark score, and they better reflect how a profile performs inside a team’s real workflow.
6. Common Misreadings and a Rational View
Watch out for these four traps when interpreting benchmarks like this one:
- Pass rate only: focusing on pass rate while ignoring the rework cost that completion represents. The gap between 17% and 24% may be far smaller than the cost of a human taking over near-finished work.
- Cost only: looking at the per-call price while ignoring the human waiting and coordination cost of long-running tasks.
- Leaderboard only: public benchmark tasks are relatively fixed and differ from a team’s real task distribution, so the ranking should not be extrapolated into “this one suits us best.”
- Full automation only: treating agents as a fully automatic code factory and overlooking the gatekeeping role of review, testing, and release.
A more rational stance: treat the leaderboard as a health checkup, not a ranking. It shows where the gaps are and which engineering posture fits which class of task. What truly determines team effectiveness is whether you can build an evaluation baseline, execution architecture, and quality process that match your own business.
7. Conclusion
This update to a mobile app development AI coding benchmark looks like the release of a few score sets. In substance, it exposes the core tensions that AI coding must confront as it moves into the long-horizon engineering stage:
- Quality vs. cost;
- Speed vs. reliability;
- Autonomy vs. controllability.
The high-completion, long-runtime profile and the low-cost, fast profile are not better and worse; each is the optimal answer for a different task shape. For engineering teams, the right approach comes down to three points:
- Read the engineering meaning behind each metric;
- Layer tasks and combine the profiles accordingly;
- Turn selection into a sustainable engineering capability with your own baseline, a quantified model, and a cloud execution architecture.
