Mobile AI Coding Benchmark Results: Metrics, Model Trade-offs, and Selection Guide
This article analyzes a new benchmark for AI coding in mobile development. It explains the four task categories and four core metrics, compares two anonymized models against a reference baseline, and discusses why long-horizon coding tasks often fail during verification rather than code generation. It then provides practical guidance for building an internal evaluation loop, tiering models by task value, applying cost controls, and maintaining human review for production use.
A new mobile AI coding benchmark has released a new round of results. The findings show that the capability gap among mainstream large models is narrowing, while differences in cost and efficiency are becoming more pronounced. Based on public data, this report anonymizes two newly evaluated models: Model A refers to a high-quality, long-horizon model, and Model B refers to a low-cost, fast-response model. The data are used only to illustrate the trade-offs among metrics.
The sections below first explain the task composition and metric system of the benchmark, then compare the results, and finally offer practical recommendations for enterprise model selection and engineering implementation.
1. What the Benchmark Tests: Four Core Tasks in Mobile AI Coding
This benchmark is designed for AI coding evaluation in mobile application development. It covers four types of real engineering tasks:
- Application creation: Generate a runnable application skeleton and basic functionality from requirements.
- Feature development: Add new business features to an existing project while maintaining architectural consistency.
- Code migration: Gradually migrate implementations from legacy technology stacks or architectures to new approaches.
- Application conversion: Convert applications across platforms or form factors, such as extending a phone app to tablets, foldable devices, and other form factors.
Compared with ordinary code completion evaluations, this benchmark is closer to real delivery scenarios. It does not focus only on how many lines of correct code a model can produce. Instead, it evaluates whether a model can maintain context consistency across multiple interaction turns, correctly invoke tools, locate and fix defects, and ultimately make the project buildable, runnable, and verifiable. Therefore, the leaderboard data should be treated as a capability reference rather than a direct selection conclusion.
2. Four Core Metrics: What Each Number Answers
| Metric | Definition | Engineering question answered |
|---|---|---|
| Pass rate | Percentage of tasks that fully pass acceptance criteria | How many tasks can be handed off |
| Average completion | Average task completion progress, indicating how far the code has been written | How much remaining work requires human intervention |
| Average elapsed time | Average elapsed time per task from start to finish | Whether it fits the iteration cadence |
| Cost per benchmark run | Total cost of running all tasks, usually including model calls, tool calls, and retries | Whether it is affordable for regular use |
Three points are important when interpreting these metrics.
First, pass rate is more honest than completion. A task with high completion but blocked at the final acceptance stage is statistically treated as a failure. This is why high completion does not necessarily mean high usability.
Second, elapsed time and cost are not the same thing. Elapsed time depends on task complexity, retries, and tool-call frequency, while cost mainly depends on billing methods, such as input/output token-based billing or duration-based billing.
Third, single-run cost is not the same as production cost. In this benchmark, long-horizon tasks usually involve much larger input and output volumes than everyday coding requests. Therefore, single-run cost cannot be directly extrapolated proportionally to real request volume.
3. Anonymized Comparison: Three Patterns Behind the Data
The benchmark includes 30 long-horizon tasks. The historical best pass rate was 28.7%, which can serve as a reference baseline.
| Item | Model A | Model B | Reference baseline |
|---|---|---|---|
| Agent framework used | Agent Framework A | Agent Framework B | Not disclosed |
| Pass rate | 24.0% | 17.3% | 28.7% |
| Average completion | 82.3% | 71.2% | Not disclosed |
| Average elapsed time | 27.4 hours | 12.0 hours | Not disclosed |
| Cost per benchmark run | $317.4 | $9.9 | Not disclosed |
Pattern 1: The quality gap is narrowing, while the cost gap is widening.
- Model A’s pass rate and completion are 6.7 and 11.1 percentage points higher than Model B, respectively. Both models fall within the 17%–24% pass-rate range, meaning more than half of the tasks cannot be fully passed.
- In terms of cost, Model A is about 32 times more expensive than Model B, and its elapsed time is about 2.3 times longer.
- If measured by the cost required to improve pass rate by 1 percentage point, Model A costs about $13.2, while Model B costs about $0.6. Model A is about 23 times more expensive. This high marginal cost suggests that simply switching to a stronger model is unlikely to significantly improve pass rate in a short period.
Pattern 2: The bottleneck for some tasks is verification, not code generation.
- Both models show completion rates that are much higher than pass rates.
- For Model A, if completion could be linearly converted into pass rate, approximately 24.7 tasks would have their main code completed, but only about 7.2 tasks fully pass acceptance. About 17 tasks still fail acceptance.
- These tasks often look complete, but they fail during build, UI adaptation, or testing. Therefore, engineering-oriented automated verification and defect repair are closer to the real bottleneck than code generation alone.
Pattern 3: The lower-cost model has higher throughput per unit of time.
- Model B takes about 0.7 hours on average to pass one task, while Model A takes about 1.1 hours.
- Model B is better suited for batch tasks that can be quickly verified, while Model A is better suited for structurally complex long-horizon tasks with little room for error.
Derived Metrics
| Derived metric | Calculation method | Model A | Model B |
|---|---|---|---|
| Average elapsed time per passed task (hours) | Average elapsed time ÷ pass rate | About 1.1 | About 0.7 |
| Cost per passed task (USD) | Cost per benchmark run ÷ number of passed tasks | About 44 | About 0.6 |
| Hourly run cost (USD) | Cost per benchmark run ÷ average elapsed time | About 11.6 | About 0.8 |
Derived metrics also have boundaries.
- “Cost per passed task” includes spending on unfinished tasks, so it is only suitable for horizontal comparison within the same task set. It should not be interpreted as the average price of a real task.
- “Hourly run cost” is only meaningful when elapsed time is mainly spent on model inference. If most of the time is spent waiting for builds or tests, this value may be distorted.
- The table only reflects a one-time benchmark run. It cannot capture fluctuations caused by retries, parallelism, or caching. Enterprises should recalculate these values using their own task data.
4. Why Long-Horizon Tasks Are Difficult: Four Typical Failure Modes
To understand the benchmark results, it is also necessary to understand why long-horizon coding tasks are prone to failure. In practice, there are four typical failure modes:
- Context drift. After dozens of interaction turns, early constraints such as naming conventions and interface agreements gradually lose influence, and new code may conflict stylistically with the existing architecture.
- Broken tool-call chains. Actions such as reading and writing files, running builds, and executing tests depend on each other. If an error returned at one step is not correctly interpreted, subsequent operations may fail in sequence.
- Environment and dependency mismatches. Long-horizon tasks may repeatedly adjust dependency versions, eventually leading to version conflicts or non-reproducible builds.
- “Looks complete” half-finished results. An agent may stop early when approaching acceptance conditions, or repeatedly make small fixes to incomplete results. This appears as high completion but low pass rate.
Possible improvements include maintaining persistent task state and decision records for agents, forcing builds and tests after every work round, and introducing human review for critical changes.
5. From Public Benchmark to Internal Evaluation: Building an Enterprise Evaluation Loop
A public leaderboard is an industry reference point, but it cannot replace an enterprise’s own evaluation system. The following four steps are recommended.
1. Stratified Sampling: Build a Task Set
- Sample tasks from a real requirement pool and categorize them into application creation, feature development, code migration, and application conversion.
- Tasks can be divided into three difficulty levels: single-point fixes, module-level development, and end-to-end long-horizon tasks.
- A total size of 20 to 40 tasks is recommended to balance coverage and running cost.
2. Unified Task Description: Ensure Reproducibility
Each task should include its goal, constraints, acceptance criteria, and initial code snapshot. All evaluated models should use the same task descriptions and the same tool interfaces.
# Example evaluation task description (fields are illustrative)task_id: task_014category: Code Migration # Options: Application Creation / Feature Development / Code Migration / Application Conversiondifficulty: hard # Options: easy / medium / hardrepo_snapshot: baseline@a1b2c3d # Initial code snapshot identifiergoal: Migrate the legacy list page to the new list component while preserving data loading and pagination behaviorconstraints:- Preserve the existing directory structure and naming conventions- Do not introduce dependencies that conflict with the existing dependency systemacceptance:- The project builds successfully- All existing unit tests pass- New pagination logic has corresponding test casesbudget:max_wall_clock_hours: 30 # Maximum elapsed time per taskmax_cost_usd: 50 # Maximum cost per task
3. Multi-Dimensional Scoring: Automate Where Possible
def evaluate(result, task, spec):"""Inputs:result: Agent run result (generated code, logs, elapsed time, cost)task: Task object (including initial snapshot and test sets)spec: Task specification (including acceptance criteria)Returns:Multi-dimensional score and pass/fail status"""# 1. Build: ensure the artifact compiles and can be installedbuild_ok = run_command(["build"], cwd=result.workspace, timeout=1800)# 2. Tests: run existing tests to prevent regressions and acceptance tests to verify new functionalityunit_pass = run_tests(task.existing_tests, cwd=result.workspace)accept_pass = run_tests(task.acceptance_tests, cwd=result.workspace)# 3. Static quality: cyclomatic complexity, duplication rate, etc. (illustrative; configure by tech stack)quality = static_analysis(result.workspace)weighted_score = (0.4 * accept_pass.ratio+ 0.3 * unit_pass.ratio+ 0.2 * quality.score+ 0.1 * build_ok)# Hard gate: build failure or regression in existing tests results in failurepassed = build_ok and unit_pass.no_regression and accept_pass.all_passedreturn {"score": weighted_score, "passed": passed}
4. Unified Constraints: Record Everything
- Fix timeout and cost limits, fix the runtime environment, and evaluate under the same hardware configuration.
- Run multiple times and use the average to reduce the impact of randomness.
- Record elapsed time, cost, retry count, and failure stage for each task to build a data asset that can be accumulated over time.
6. Model Selection in Practice: From One Model to a Strategy
1. Capability Tiering: Configure Models by Task Value
Based on the benchmark data, the following preliminary division can be formed. Thresholds should be calibrated with internal evaluation data.
| Task tier | Task characteristics | Suggested configuration | Human involvement |
|---|---|---|---|
| L1 lightweight tasks | Code completion, single-point fixes, copy generation | Low-cost, fast-response model with high-volume concurrent execution | Merge automatically after verification passes |
| L2 module-level tasks | Feature development, partial refactoring | Cost-effective model first; escalate to a high-quality model after failure | Human review at critical steps |
| L3 complex long-horizon tasks | Application creation, application conversion, core flow migration | High-quality long-horizon model with sufficient budget and verification | Full review and acceptance |
2. Tiered Routing: Spend the “Expensive Model” Budget Where It Matters
# Task tiering and model routing pseudocode (thresholds should be calibrated with internal evaluation data)def build_plan(task, models, budget):tier = classify(task) # light / medium / heavy, based on complexity and business impactif tier == "light":route = [models["fast"]]elif tier == "medium":route = [models["balanced"], models["strong"]] # Start with the cost-effective option; escalate after failureelse:route = [models["strong"]]return plan(route=route, budget=budget, require_human_review=(tier == "heavy"))def run_with_fallback(plan, task):for model in plan.route:result = invoke_agent(model, task) # Invoke the agent to execute the taskif verify(result, task).passed:return result # Return immediately after verification passes to avoid unnecessary spendif result.cost >= plan.budget.task_cap:break # Stop when the per-task budget cap is reached and hand off to humansreturn handoff_to_human(task) # Fallback: hand off to humans
3. Budget and Efficiency: Four Cost Guardrails
- Budget caps: Set spending limits at task, daily, and project levels.
- Cache reuse: Cache similar context prompts and retrieval results.
- Prompt trimming: Compress redundant context to reduce input token consumption.
- Incremental execution: For migration tasks, execute by module to avoid full reruns.
4. Resource Elasticity: Batch Strategies for Peak Shaving
- Schedule by elapsed time: The benchmark shows that long-horizon tasks can take from ten to several dozens of hours. These tasks are suitable for batch queuing at night or on weekends. Token-based billing models are preferable because cost is independent of duration.
- Duration-based billing models: These are better suited for synchronous interaction scenarios. In long-horizon tasks, tool execution and model inference often cannot fully overlap within the same time window, so actual elapsed time and cost may be higher.
- Resource capacity: The execution environment needs appropriate elasticity. Container platforms can be used to isolate tasks and scale elastically, while monitoring and alerting services can track task queue time and failure rates.
7. Risk and Compliance: The Final Gate for AI Coding
- Data boundaries: Before sending private code or business data to external services, complete anonymization and access control, and clarify the service provider’s data usage terms.
- License compliance: Generated code may introduce restricted licenses. Automated scanning should be included in admission checks.
- Human responsibility: Final quality responsibility for build artifacts always remains with humans. Model A takes an average of 27.4 hours for long-horizon tasks. When running unattended overnight, threshold alerts for completion, elapsed time, and cost should be configured, with timely interruption and human handoff when anomalies occur.
- Traceability: Preserve task descriptions, model invocation records, and artifact diffs for review and auditing.
8. Conclusion
The real value of this benchmark update is not a few percentage points on the leaderboard. It reveals three structural facts:
- The capability gap among top models is shrinking.
- Differences in cost and efficiency are amplified by multiples.
- The bottleneck in long-horizon tasks is more often verification rather than generation.
The implication for enterprises is clear: evaluations should be built on internal tasks, models should be configured by task tier, cost guardrails should be set in advance, and human review should always be present. If a team is looking for a deployable AI coding solution, general capabilities provided by cloud platforms such as Baidu AI Cloud, including model services, container platforms, and monitoring and alerting, can serve as a technical foundation for building such evaluation and development pipelines.
