Long-Horizon Trade-Offs in AI Coding Benchmarks for Mobile Applications
AI coding benchmarks for mobile app development are moving beyond code completion to evaluate end-to-end engineering work. This article examines pass rate, completion ratio, time, and cost, explains why long-running tasks magnify differences in context management and failure recovery, and outlines reproducible evaluation, budget controls, and task-based model selection strategies.
1. From Code Completion to Long-Horizon Engineering Tasks
AI coding capability was initially understood mainly as code completion or snippet generation: given a function signature or comment, a model produces several lines of code for a developer to copy, modify, and run. These tasks have short feedback cycles and relatively simple evaluation criteria.
Real-world mobile application development is different. Engineering tasks often involve project creation, dependency configuration, page and business-logic development, API integration, data migration, building, testing, and error correction. A model or AI agent may need to work in an environment for several hours or even tens of hours, repeatedly reading code, modifying files, executing commands, analyzing logs, and adjusting its strategy based on failures.
As a result, AI coding benchmarks for mobile development should not focus only on whether generated code appears plausible. They should evaluate end-to-end task completion: whether the application can be created and run, whether the requested features work as expected, whether a migrated or converted project can be built and tested, and whether the system can identify and recover from failures. These benchmarks are closer to real software development workflows and better expose the engineering trade-offs involved in long-horizon tasks.
2. Four Core Metrics: Pass Rate, Completion, Time, and Cost
Public evaluations commonly report task pass rate, average completion ratio, average execution time, and cost per run. These are not simply ranking scores. Each metric describes a different aspect of system capability.
Pass rate: A task typically passes only when all acceptance criteria are satisfied, such as successful builds, passing tests, and correct functional behavior. Pass rate is strict and useful for measuring deliverability. However, long-horizon tasks often have high variance, and small differences in the environment or dependency resolution can cause failure.
Average completion ratio: This measures the proportion of work completed or the score across acceptance items. Even when a task does not fully pass, completion can show how effectively the model decomposed the task, modified the code, and fixed errors. High completion with a low pass rate may indicate that a system makes substantial progress but is unreliable during final verification and delivery.
Average execution time: This includes model inference, tool calls, file operations, builds, tests, and failure retries. For long-horizon tasks, execution time depends not only on model speed but also on agent orchestration, context management, parallelization, and failure-recovery efficiency.
Cost per run: This generally includes model calls, tool execution, retries, and build resources. Low-cost systems are suitable for frequent evaluations and budget-sensitive teams. Higher-cost systems must demonstrate enough quality improvement to justify the additional expense.
For example, in one public sample evaluation, one type of system achieved a pass rate of approximately 24% across 30 long-horizon tasks, an average completion ratio of 82.3%, an average execution time of 27.4 hours, and an estimated cost of $317.4 per benchmark run. Another type achieved a pass rate of approximately 17.3%, an average completion ratio of 71.2%, an average execution time of 12.0 hours, and a cost of approximately $9.9 per run.
These results illustrate a typical trade-off. The first approach delivered higher completion and a relatively higher pass rate, but required substantially more time and money. The second was faster and much less expensive, but produced lower pass and completion rates. Engineering selection should therefore consider task value, delivery deadlines, and the cost of failure rather than focusing only on pass rate or unit price.
3. The Engineering Trade-Off Behind High Completion and Long Execution Time
High completion ratios often result from more detailed task decomposition, additional rounds of validation and repair, longer context maintenance, and more conservative decision-making. For example, an agent may build and test after every modification, inspect logs after failures, locate the relevant files, generate a new patch, and repeatedly verify dependencies and configuration. These actions can improve completion, but they also increase time and cost.
The main disadvantages are:
- Higher compute and API costs: Multiple reasoning rounds, tool calls, and retries all increase resource consumption.
- Longer feedback cycles: A slow approach is unsuitable for interactive development that requires minute-level feedback, as well as for high-frequency regression evaluations.
- Greater sensitivity to environment stability: The longer a task runs, the greater the risk of dependency download problems, cache invalidation, temporary-file pollution, and other forms of environment drift.
By contrast, low-cost, short-running approaches typically use more aggressive pruning, fewer validation rounds, or shorter contexts. They respond quickly and are easier on the budget, but they are more likely to stop halfway through complex migrations, cross-file refactoring, or subtle defect repairs. This is not a simple matter of one approach being better than the other. It is an engineering trade-off among quality, speed, and cost.
4. Why Long-Horizon Tasks Magnify Differences
Long-horizon tasks amplify differences in context management, failure recovery, and tool use. A reproducible evaluation pipeline should standardize the base environment, record every tool call and build log, and divide tasks into difficulty levels. Evaluation preferences can be described with a general configuration such as:
{"task_suite": "mobile_app_long_horizon","metrics": {"pass_rate": 0.45,"completion_ratio": 0.25,"time_cost": 0.15,"monetary_cost": 0.15},"budget": {"max_hours_per_task": 8,"max_cost_per_run": 50},"reproducibility": {"fixed_dependencies": true,"clean_workspace": true,"collect_build_logs": true}}
These weights are not universal standards. They simply make business priorities explicit. Quality-focused teams can assign more weight to pass rate. Budget-sensitive teams can give more weight to time and cost. Platform teams may use tiered scoring to prevent a single overall score from concealing different failure types.
5. Reproducible Evaluation and Budget Control
For benchmark results to be useful, they must be reproducible, interpretable, and subject to budget controls.
First, dependency versions and base images should be fixed, while network access and caches should be isolated so that every task starts from the same conditions. Second, failures should be categorized as build failures, test failures, dependency-resolution failures, timeouts, or budget overruns instead of being reduced to a single pass-rate figure. Third, time and cost limits should be enforced. A run should be stopped after exceeding the defined threshold so that long-tail tasks do not consume the entire evaluation budget. Fourth, pass rates should be reported with confidence intervals or statistics from multiple runs because long-horizon tasks can have substantial variance.
The following pseudocode illustrates a possible scoring function:
function score(run):pass = run.passed ? 1 : 0completion = clamp(run.completed_items / run.total_items, 0, 1)time_score = 1 - min(run.hours / budget.max_hours, 1)cost_score = 1 - min(run.cost / budget.max_cost, 1)return w_pass * pass+ w_completion * completion+ w_time * time_score+ w_cost * cost_score
In production, an evaluation pipeline can use general capabilities such as object storage, container platforms, logging, monitoring, and alerting—for example, those provided by Baidu AI Cloud—to manage task artifacts, isolate execution, collect logs, and trigger metric alerts. However, the core of the evaluation remains the protocol and data governance, not simply the accumulation of infrastructure resources.
6. Team Selection: Match Solutions to Task Value
Teams that prioritize budget and response time can use low-cost, short-running systems for frequent tasks with clear boundaries, such as boilerplate generation, simple page modifications, configuration fixes, and initial migration work.
For tasks with high delivery requirements and high failure costs—such as cross-module refactoring, complex business migrations, and critical defect repairs—teams can use higher-completion, longer-running systems. Another option is a hybrid workflow: begin with a low-cost exploratory approach and use a more reliable system for finalization.
The key to hybrid routing is to define escalation conditions. A task can be automatically switched to a more robust solution when the low-cost approach fails repeatedly, falls below a completion threshold, or reaches its time or cost limit.
Teams should also establish observability by recording execution time, cost, failure reasons, and the number of manual interventions for every task. Regular regression evaluations are equally important. Only by tracking pass rate, completion, time, and cost in the same engineering ledger can teams avoid being misled by a single leaderboard score and make technology choices that fit their delivery cadence.
