Mobile App Development Agents: Benchmark Evaluation and Selection Guide
Mobile app development is rarely a single-turn code-completion task—it spans app creation, feature development, code migration, UI adaptation, build and test, and version compatibility—so choosing an AI agent by benchmark score alone is misleading. This guide shows how to read mobile-development agent benchmarks, explains why a high completion rate does not guarantee deliverable results, and introduces a cost-per-successful-delivery model that combines compute with human review. Using two example configurations, it breaks down pass rate, average completion, wall-clock time, and per-task cost, adds a sensitivity analysis, and shows how to turn benchmark results into repeatable team capability through version locking, sandboxing, context engineering, and tiered routing. It closes with the risks and limitations of trusting leaderboard numbers.
1. Conclusion First: How to Choose a Mobile App Agent
Mobile app development tasks are usually not single-turn code completion. They form a full loop that spans app creation, feature development, code migration, UI adaptation, build and test, and version compatibility. So agent selection cannot hinge on “which model scores highest.” It has to answer three engineering questions: is the task suitable to hand to an agent, can failures be caught by automated gates, and can the cost be mapped to a successfully delivered result?
A quick way to do a rough first split:
- Template migration, scaffolding, bulk rewrites: clear task boundaries and easy-to-verify results. Prefer a lightweight, fast agent that emphasizes low cost and quick response.
- Large module refactoring, cross-platform migration, complex dependency changes: high human-replacement cost. A long-horizon agent is an option, but you must set a budget cap, timeout control, and human review checkpoints.
- Real-time completion during coding: response latency directly affects the experience, so prefer a low-latency option and escalate to a higher-capability agent for complex tasks.
2. Background: From Leaderboard Scores to Deliverable Verification
A recent benchmark focused on mobile app development published a new round of model scores. Because the original materials did not fully publish verifiable links, this article treats the data as anonymized second-hand samples. Before any formal procurement or project kickoff, you should verify the original leaderboard, task set, scorer, scaffold configuration, and cost basis.
Two of the sample groups form a sharp contrast:
- Plan A (long-horizon agent): 24.0% pass rate across 30 long-horizon tasks, 82.3% average completion, 27.4 hours average time, example cost of about ¥2,123.
- Plan B (lightweight fast agent): 17.3% pass rate, 71.2% average completion, 12.0 hours average time, example cost of about ¥66.2.
The value of these numbers is not a simple ranking. It is that they put three hidden costs—machine time, monetary cost, and rework risk—on the table once an agent enters the development pipeline. When an agent can complete only about 80% of tasks while possibly consuming a full day and a high amount of compute, what the team really has to judge is whether it can fit into a real delivery process.
3. Metric Breakdown: The Engineering Meaning Behind Each Number
3.1 Pass Rate: The Hard Gate for Deliverability
Pass rate usually uses a binary judgment: a task either passes all acceptance conditions or is marked as failed. It is the metric closest to “deliverable,” but it is also the easiest to mask the structure of failures.
When using pass rate, focus on two things:
- Failure composition: is it a functional logic error, or a missing dependency, a build failure, or a UI adaptation that does not pass? Different failure types consume the development process very differently.
- Statistical confidence: 30 tasks is a limited sample. At a sample size of 30, pass rate swings widely, so you should describe it with a binomial distribution or Wilson interval combined with the number of repetitions; if you rely only on a normal approximation, 24% at n=30 corresponds to roughly 8.7%–39.3%. Whether the 6.7-percentage-point pass-rate gap between Plan A and Plan B is stable therefore needs more tasks or multiple repeated evaluations to confirm.
3.2 Average Completion: Measuring “How Far From the Finish Line”
Completion is a continuous progress value. It is good for observing how far a task has progressed, but it cannot replace final acceptance. If half the tasks are stuck at 80% progress and another 30% fail completely, the average can still exceed 80%. Completion is therefore better suited to longitudinal comparison between versions.
When using completion, you must confirm three questions:
- Is the score divided by module, or by acceptance test case?
- Is it judged automatically by a third party, or self-reported by the model?
- Is the scorer adjusted in sync with model versions?
An opaque completion metric is easy to “game,” so it should not be used alone as a selection basis.
3.3 Average Time: An Undervalued System-Level Cost
The time here is wall-clock time, not just inference time. It usually also includes context retrieval, tool calls, build and test execution, and failure retries.
Looking at the two plans together:
- Plan A’s completion is about 11.1 percentage points higher and its pass rate is relatively improved by about 39%, at the cost of time rising to about 2.3x.
- If the 30 tasks run serially, a single evaluation round alone consumes about 822 hours and 360 hours of machine runtime, respectively.
This shows that a long-horizon agent’s demands on the execution environment are systemic: high-concurrency sandboxes, task queues, runtime isolation, and checkpoint recovery are all indispensable. Simply “switching to a stronger model” does not solve scheduling, queuing, and resource-management problems.
3.4 Per-Run Cost: You Must Define the Basis
This article defines “per-run cost” clearly as per-task cost. If the original leaderboard gives the total cost of a full round of 30 tasks, you cannot simply divide the total cost by the pass rate; instead, you should use:
Total benchmark run cost ÷ (number of tasks × pass rate)
To make it easier to combine with human review cost, the example below is shown on a per-task-cost basis. If ¥2,123 is actually the total cost of 30 tasks, then the cost per successful delivery should be:
- Plan A: 2,123 ÷ (30 × 24.0%) ≈ ¥295.0
- Plan B: 66.2 ÷ (30 × 17.3%) ≈ ¥12.8
The multiplier is still about 23x. In actual selection, you must first unify the cost basis, then compare absolute values and the premium.
3.5 Raw Data at a Glance
| Metric | Plan A | Plan B | A/B ratio | Engineering interpretation |
|---|---|---|---|---|
| Pass rate | 24.0% | 17.3% | ~1.39x | Directly affects the deliverable share; judge with confidence intervals |
| Average completion | 82.3% | 71.2% | ~1.16x | Good for longitudinal comparison; requires a clear scoring rubric |
| Average time | 27.4 hours | 12.0 hours | ~2.3x | About 34 or 15 days for one serial round; depends on parallel sandboxes |
| Per-task run cost (example basis) | ~¥2,123 | ~¥66.2 | ~32x | Large nominal gap; clarify whether it includes retries and caching |
| Per successful delivery (compute only) | ~¥8,845.8 | ~¥382.7 | ~23.1x | Closer to real return; still add human review cost |
4. The Core Contradiction: Why High Completion Does Not Mean Deliverable
Plan A’s average completion exceeds 80%, yet its pass rate is under a quarter; Plan B shows a similar trend. This does not necessarily mean completion is faked. It is determined by the structural nature of mobile development tasks and the evaluation methodology together.
First, completion is often concentrated in the last step. Tasks like app creation and code migration involve multiple collaborating modules. The first half is often made up of boilerplate code and repetitive rewrites, with difficulty increasing over time. A passing build, correct rendering of key screens, and consistent behavior across different screen sizes and OS versions are usually concentrated in the final acceptance stage. Being only a few percentage points short on completion can mean the core functionality does not run at all.
Second, agents have long-tail failures. Some runs may be interrupted by context-window overflow, a tool call stuck in a loop, an accidental change to the build config, or exhaustion of the budget cap. Such runs usually have high mid-run completion but contribute no passing samples.
Third, the stricter the acceptance gate, the lower the pass rate. If the acceptance set requires zero compile errors, all critical-path tests to pass, and no new static-check warnings, then failing any single step is judged a failure. This is consistent with a team’s real quality culture, but it systematically amplifies the failure ratio.
So a more actionable process is: first run objective gates such as build, static checks, and automated tests; then sort by completion; and finally hand only the top-ranked candidates to human review. This avoids wasting effort on failing plans and lets average completion genuinely act as a “coarse filter.”
5. Selection Logic: A Three-Dimensional Tradeoff of Cost, Quality, and Time
5.1 Build a “Real Delivery Cost” Model
Agent selection should not stop at unit-price comparison; it should build a computable evaluation framework. The code below renames the original model’s retry_times to the clearer total_run_times, representing the total number of runs per task, including the first run—it is not “extra retry count.” If you retry once after the first run, you should enter 2.
def effective_cost(per_task_cost: float, # Per-task cost (CNY)pass_rate: float, # Task pass ratereview_hours: float, # Human review time per task (hours)total_run_times: int = 1, # Total runs per task, including the first runlabor_rate: float = 200.0, # Labor cost (CNY/hour)auto_reject_rate: float = 0.9, # Share of failures blocked automatically by gates) -> dict:"""Estimate the real cost per successful delivery"""compute = per_task_cost * total_run_times / pass_ratemanual = review_hours * labor_rate * (1 - auto_reject_rate * (1 - pass_rate))total = compute + manualreturn {"compute_cost": round(compute, 2),"manual_cost": round(manual, 2),"total_cost": round(total, 2),}
Assume 1 hour of review per task, labor cost of ¥200/hour, 1 total run per task, and gates auto-blocking 90% of failed runs:
Plan A: per-task cost ¥2,123, pass rate 24.0%.
Compute cost = 2,123 ÷ 24.0% ≈ ¥8,845.8.
Manual cost = 1 × 200 × (1 − 0.9 × (1 − 24.0%)) ≈ ¥63.2.
Total cost ≈ ¥8,909.Plan B: per-task cost ¥66.2, pass rate 17.3%.
Compute cost = 66.2 ÷ 17.3% ≈ ¥382.7.
Manual cost = 1 × 200 × (1 − 0.9 × (1 − 17.3%)) ≈ ¥51.1.
Total cost ≈ ¥433.8.
Although the parameters are assumed, the direction is clear: when per-task cost and pass rate are calculated together, the nominal cost gap narrows from about 32x to about 20x; whether it is worth paying for the higher-capability plan depends on task value, failure-blocking capability, and human-replacement cost.
Sensitivity analysis:
- A higher
pass_ratesignificantly lowers the compute cost per successful delivery. For example, if Plan B’s pass rate rises from 17.3% to 22.3%, the per-task compute cost drops from about ¥382.7 to about ¥296.9. - A higher
total_run_timesmultiplies the compute cost. Going from 1 to 2 doubles it directly. review_hoursandauto_reject_ratedetermine the manual cost. If the automatic gate’s block rate drops, more failed results enter human review and the total cost rises quickly.
5.2 How to Choose: Allocate Compute Budget by Scenario
| Scenario | Key constraint | Recommended strategy |
|---|---|---|
| Scaffolding, template migration, bulk rewrites | Clear task boundaries, easy-to-verify correctness | Lightweight fast agents, paired with templates and code-generation tools |
| Large module refactoring, cross-platform migration | Complex logic, high human-replacement cost | Long-horizon agents run offline as nightly jobs, humans gate key points |
| Real-time completion during coding | Response latency directly determines experience | Prefer low-latency solutions; escalate to higher-capability solutions for hard problems |
5.3 Common Mistakes: Do Not Turn Evaluation Numbers Directly into a Procurement Conclusion
- Looking only at a single-point pass rate: ignoring the confidence interval makes it easy to mistake noise for a trend.
- Looking only at average completion: completion does not equal deliverability, and under strict acceptance it severely overestimates real capability.
- Comparing only unit price: you should compare the “total cost per successful delivery,” not the price of a single call.
- Ignoring scaffold differences: different models may pair with different prompts, tool sets, context-compression strategies, and retry mechanisms; the scaffold’s impact can be no less than that of the base model.
- Overlooking failure structure: functional errors, build failures, missing dependencies, and budget exhaustion consume development resources very differently.
- No version records: after model, prompt, toolchain, and evaluation-set changes, historical results may no longer be reproducible.
5.4 What to Watch: Data, Compliance, and Reproducibility
Before submitting private code to an external agent service, confirm the data-handling policy, data-retention period, log-audit requirements, and deletion mechanism. When trade secrets, user privacy, or regulated data are involved, prefer private deployment, dedicated networks, or an auditable internal gateway. Before cross-vendor comparison, also unify the cost basis and confirm whether it includes retries, cache-hit discounts, image-build time, sandbox resource usage, and human review cost.
6. Implementation Practice: Turning Evaluation Results into Team Capability
6.1 Version Locking and Reproducible Evaluation
Model versions keep iterating, so today’s conclusion may be invalid next month. Teams should record reproducible metadata for each evaluation and archive it together with artifacts and logs.
{"model_id": "<model-identifier>","model_version": "<version-number>","scaffold_version": "1.4.0","prompt_template_hash": "sha256:9f2c...","tool_config": {"allow_network": false, "max_steps": 80},"dataset": {"name": "mobile-dev-tasks", "version": "2026-03", "size": 30},"budget": {"token_limit": 400000, "wall_time_hours": 8},"sandbox_image": "registry.internal/sandbox:mobile-sdk-2.4","seeds": [42, 43, 44],"metrics": {"pass_rate": 0.24, "completion": 0.823, "avg_hours": 27.4}}
Only with complete metadata can results be compared longitudinally; when a regression appears, you can then pinpoint whether the model, prompt, toolchain, or evaluation set changed.
6.2 Sandboxing: Get It Running First, Then Merge
Mobile tasks are naturally suited to sandbox execution: each task can be wrapped in a separate container preloaded with build toolchains and device emulators, with network access and system privileges restricted.
# Example assumes MODEL_ENDPOINT points to a local model or local proxy.# With --network none the container cannot reach a remote model; if a remote# model is required, drop --network none and restrict access via an egress# firewall, proxy, or gateway.docker run --rm \--network none \--cpus 2 --memory 4g \--user 1000:1000 \-v "$(pwd)/tasks/task-017:/workspace:ro" \-v agent-cache:/cache \-v agent-output:/tmp/output \-e MODEL_ENDPOINT \registry.internal/sandbox:mobile-sdk-2.4 \agent-runner --task /workspace/task.json --max-steps 80 --budget 400000# If the agent needs to write into the repository, make /workspace a writable# volume, or allow writes only to /tmp and /cache, then do a controlled copy# and code review before merging.
By scheduling a scaled sandbox pool in parallel, you can compress a long-horizon evaluation that would take 34 days serially down to an acceptable duration. At runtime, it is best to decouple each tool call and inference round through a message queue, and store task state and intermediate artifacts in persistent storage to avoid a full re-run caused by a single interruption.
6.3 Context Engineering: Help the Agent Understand Your Codebase
The common cause of repository-level task failures is not that the model cannot write code, but that it “does not understand the project.” Improving context quality is usually more cost-effective than swapping the model.
You can maintain a collaboration-rules file at the repo root that spells out the tech stack, directory conventions, coding standards, test and build commands, and prohibited actions.
# Project Collaboration Rules- Tech stack: language versions and frameworks follow the build scripts; do not upgrade dependencies arbitrarily- Must pass before commit: code formatting, static checks, unit tests- New public interfaces must include unit tests- Never commit any keys or credentials; inject sensitive config via environment variables
At the same time, use repository search or code-indexing capabilities to fetch relevant snippets on demand rather than dumping the whole repository into the context. For very long tasks, you can produce a structured status summary so each step focuses only on the current goal and verified facts.
6.4 Tiered Routing: Balancing Cost and Quality
Routing by task complexity is the most direct way to control cost. Simple tasks are handled by a lightweight fast agent and escalated level by level on failure; complex tasks go directly to a long-horizon agent with a clear budget and timeout cap.
TASK_CLASSES = {"format_lint": {"models": ["fast"], "max_runs": 1},"unit_test": {"models": ["fast", "strong"], "max_runs": 2},"module_migrate": {"models": ["strong"], "max_runs": 1},}def run_task(task, runner, gates, budget):policy = TASK_CLASSES[task.cls]for attempt in range(policy["max_runs"]):model = policy["models"][min(attempt, len(policy["models"]) - 1)]if budget.remaining() < model.estimate(task):return "budget_exhausted"result = runner.run(model, task, budget)if gates.build(result) and gates.test(result) and gates.lint(result):return resultreturn "manual_review"
6.5 Evaluation Operations: Logs, Monitoring, and Alerts
It is best to build a unified runtime observability system:
- For each run, record the model, time, token usage, tool-call sequence, and failure reason.
- Schedule the sandbox pool through a container platform, and use a logging service to search failure stack traces centrally.
- Set monitoring and alerts for abnormal durations and cost overruns.
- In the cloud, this capability can be assembled from general-purpose products such as a container platform, object storage, message queues, a logging service, and monitoring and alerting.
7. Risks and Limitations
- The gap between leaderboards and real workloads: benchmark tasks are usually single-point tasks with clear boundaries, whereas real projects are full of cross-service dependencies, legacy code, and implicit business rules. Leaderboard results can only serve as a starting point for filtering, not a substitute for your team’s own validation set.
- The cost basis needs confirmation: whether the per-run cost includes retries, whether it counts cache-hit discounts, and whether different vendors’ pricing units are consistent all affect horizontal comparison.
- The scaffold affects results: after swapping the scaffold, the same base model can show noticeably different scores. Before cross-model comparison, unify the scaffold or disclose the configuration.
- Compliance and data security cannot be ignored: before submitting private code, confirm the data-handling policy, retention period, audit requirements, and deletion mechanism; deploy in a private environment when necessary.
8. Conclusion
The comparison of Plan A and Plan B reveals one clear pattern: in agent-assisted development, there is no “across-the-board best” model—only a combination matched to the task structure, budget constraints, and delivery cadence. For a development team, a more effective approach than chasing leaderboards is to:
- Define metric bases clearly and judge differences with confidence intervals rather than single-point numbers;
- Build version locking and a sandbox-based evaluation mechanism to ensure reproducible results;
- Replace unit-price comparison with “total cost per successful delivery”;
- Use collaboration rules, context retrieval, automated gates, and tiered routing to turn model capability into stable engineering output.
As agents move from “completing code” to “completing tasks,” the developer’s core skill also shifts from “writing code” to “defining tasks, designing constraints, and validating results.” Whoever builds the evaluation, gating, and scheduling system more solidly will be better able to convert agent capability into delivery efficiency.
