Mobile AI Coding Benchmarks: Long-Horizon Tasks and the Cost of Delivery
Mobile coding benchmarks are moving beyond code completion toward end-to-end engineering delivery. This article explains how to interpret pass rate, completion, duration, and cost; why long-horizon tasks are expensive; how to calculate cost per successful run; and how teams can build realistic internal benchmarks and use hybrid strategies for quality, speed, and budget control.
1. Evaluation Is Moving from “Can Write Code” to “Can Deliver the Task”
Early evaluations of AI-assisted mobile development often focused on short problems, code completion, or single-file edits. These tests can measure syntax, API recall, and local reasoning, but they rarely reflect the complexity of real software delivery.
Real projects typically involve multi-module dependencies, build configuration, resource management, permission declarations, lifecycle handling, compatibility constraints, test cases, and validation of release artifacts. A feature that appears simple may require the agent to understand the project structure, modify multiple files, adjust dependencies, fix build failures, run tests, and perform regression validation.
As a result, mobile development benchmarks are increasingly adopting long-horizon tasks, such as creating an application skeleton, implementing a complete feature, migrating a legacy module, or converting a project structure. The key capability is no longer one-shot code completion. It is the ability to perform multiple tool calls, maintain context, manage state, recover from errors, and complete a closed-loop validation process.
The closer an evaluation is to real engineering work, the more clearly it exposes differences in the final mile of delivery. Can the system understand an existing project? Can it identify and fix problems after a failure? Can it make the build, tests, and acceptance checks pass at the same time?
2. Four Core Metrics Should Not Be Confused
At least four types of metrics should be distinguished when analyzing these benchmarks:
- Pass rate: The percentage of tasks that pass end-to-end acceptance. This is usually the strictest metric. The build must succeed, tests must pass, and the required functionality must be delivered. Code generation alone is not sufficient.
- Average completion ratio: The proportion of task subgoals completed. A high completion ratio combined with a low pass rate suggests that the solution can make progress across multiple steps but may fail to close the loop during integration, acceptance, or environment adaptation.
- Average duration: The average time required to complete a task. This directly affects developer feedback cycles and pipeline utilization.
- Cost per run: The combined cost of inference, tool execution, sandbox operation, and failed retries. Looking only at per-run cost without considering success rate can substantially underestimate the real cost.
A more useful metric is cost per successful run:
Cost per successful run = Total operating cost / Number of successful tasks
If Solution A has a lower per-run cost but a lower pass rate, while Solution B costs more per run but completes more tasks successfully, the comparison should consider cost per successful run, human review cost, and business impact.
For high-frequency feedback, response speed and low cost may matter more. For complex migrations and production delivery, end-to-end pass rate and completion ratio are more important.
3. How to Interpret Public Benchmark Data
Publicly available results from one mobile development benchmark update show three typical types of solutions. A leading solution achieved a pass rate of approximately 29%. A long-running solution reached a pass rate of about 24%, with an average completion ratio of roughly 82%, an average duration of approximately 27 hours, and a per-run cost in the hundreds of U.S. dollars. A lightweight solution achieved a pass rate of about 17%, an average completion ratio of approximately 71%, an average duration of around 12 hours, and a per-run cost in the tens of dollars.
The engineering implications of these figures are more valuable than the ranking itself.
The long-running solution’s high completion ratio indicates that it can continue progressing through multiple subgoals and may be suitable for cross-module, long-chain tasks. However, its end-to-end pass rate remains significantly below its completion ratio. This gap often comes from final acceptance checks, cross-module integration, environment differences, implicit requirements, and regression fixes.
The lightweight solution has a lower pass rate but clear advantages in speed and cost. It is better suited to frequent experimentation, localized changes, boilerplate generation, and low-risk tasks.
The leading solution’s slightly higher pass rate suggests that long-horizon capability and cost efficiency can be improved together. However, the available optimization space is constrained by task complexity, the strictness of acceptance checks, and the maturity of the toolchain.
4. The Engineering Trade-Off Behind High Completion and Long Duration
High completion is rarely free. Long-horizon tasks require more interaction rounds, larger contexts, more complex tool calls, and more thorough validation. As a result, both duration and cost increase.
The main trade-offs include:
- Context and state management: Cross-file tasks require the system to maintain project state over time while avoiding conflicting edits and context loss.
- Tool calls and sandbox execution: Builds, tests, and static checks must run in a real environment, making execution time and resource consumption significant.
- Failure recovery and retries: In a long task chain, one failure may trigger multiple rounds of diagnosis and repair, amplifying retry costs.
- Strictness of acceptance criteria: The closer acceptance checks are to production requirements, the lower the pass rate may be, but the more credible the results become.
- Parallelism and caching: Dependency caches, incremental builds, and parallel execution can reduce cost while increasing the complexity of the evaluation platform.
Teams should therefore avoid optimizing only for the highest pass rate or the lowest cost. Task types should be handled differently. Lightweight solutions can support rapid experimentation for exploratory work, while long-horizon solutions can handle complex production workflows with human review, incremental validation, and continuous integration gates.
A hybrid strategy is often more reliable than relying on a single solution. A lightweight system can handle initial screening, boilerplate, and localized fixes. A long-horizon system can take responsibility for migrations, cross-module changes, and complex defect resolution. Executable acceptance criteria should provide the final quality gate.
5. Cost Models and Optimization Practices
Evaluation and deployment costs include more than model inference fees. A more complete model is:
Total cost = Inference cost+ Tool and sandbox execution cost+ Failed retry cost+ Human review cost+ Opportunity cost
Potential optimization practices include:
- Route tasks according to difficulty.
- Set budget limits and early-stop conditions.
- Cache dependencies and build artifacts.
- Use incremental testing.
- Classify failed tasks and reuse diagnostic results.
- Run independent tasks in parallel.
- Schedule high-cost long-horizon tasks during off-peak periods.
At the infrastructure level, teams can use general-purpose container platforms, object storage, logging, monitoring, and alerting capabilities provided by Baidu AI Cloud to build isolated and reproducible evaluation environments. This can reduce environment drift and make results easier to trace.
The objective is not simply to allocate more resources. It is to make the cost of each successful run measurable and optimizable.
6. A Practical Checklist for Building an Internal Benchmark
For internal evaluation, teams should build a benchmark that reflects their own repositories instead of relying only on public leaderboards:
- Source tasks from real repositories and cover application creation, feature development, migration, project conversion, and defect fixing.
- Make acceptance criteria executable, including builds, unit tests, smoke tests, and key business assertions.
- Fix the runtime environment and record dependency versions, build parameters, and toolchain configuration.
- Preserve complete logs and diffs to support failure analysis and result reproduction.
- Group tasks by length and risk, and report pass rate, completion ratio, duration, and cost separately.
- Set budget and timeout limits to prevent unlimited retries on a single task.
- Add manual sampling to calibrate the gap between automated acceptance and actual deliverable quality.
- Maintain a regression set to track stability after each solution update rather than evaluating only a single benchmark run.
7. Example Evaluation Pipeline
A general benchmark configuration can be abstracted as follows:
benchmark:suite: mobile_long_horizontasks:- create_app_skeleton- implement_feature- migrate_module- convert_project_structureacceptance:build: requiredunit_tests: requiredsmoke_tests: requiredmetrics:- pass_rate- completion_ratio- duration_hours- cost_per_run- cost_per_success
The scoring logic should also focus on end-to-end results rather than generated content alone:
function score(task):completion = task.completed_subgoals / task.total_subgoalspassed = task.build_ok and task.tests_ok and task.smoke_okcost_per_success = task.total_cost / max(task.success_count, 1)return { completion, passed, cost_per_success }
8. Conclusion
The evolution of mobile AI coding benchmarks reflects an important shift: evaluation is moving from code-fragment capability to engineering task delivery.
Pass rate, completion ratio, duration, and cost must be interpreted together. A high-completion, long-running solution demonstrates the ability to advance complex tasks, but also exposes the cost of long-chain validation. A lightweight solution trades some success probability for faster and less expensive experimentation.
For development teams, the more reliable path is to build an internal benchmark based on real tasks, select solutions by scenario, and use end-to-end acceptance and cost per successful run as core metrics. Continuous calibration across quality, efficiency, and budget is more valuable than optimizing for a single benchmark score.
