AI Coding Agent Evaluation for Mobile Long-Horizon Tasks: Pass Rate, Completion, and Cost
This article explains how to evaluate AI coding agents on long-horizon mobile development tasks. It breaks down pass rate, average completion, average elapsed time, and per-run cost, discusses why higher completion often requires more time and expense, and offers practical guidance for task layering, budget guardrails, observability, and reproducible internal evaluation.
1. Introduction: From Code-Generation Evaluation to Engineering-Delivery Evaluation
AI-assisted programming evaluation is shifting from “writing a correct piece of code” to “delivering a working feature.” The former focuses on code completion accuracy, unit test pass rate, and syntactic correctness, usually at the function or single-file level. The latter requires an AI agent to work continuously for tens of minutes or even hours, covering requirement understanding, code modification, build and compilation, and runtime verification.
A recent mobile development benchmark update added results for multiple AI coding agents and showed an interesting split:
- The leading performer achieved an average completion rate of more than 80%, ranked second in pass rate among all evaluated models, but took more than one day on average and cost several hundred USD per evaluation run.
- Lightweight, fast models had clearly lower pass rates and average completion, but their average runtime was only a fraction of the leader’s and their per-run cost was less than 3% of it.
These two profiles, “high completion, high runtime, high cost” versus “lower pass rate, low runtime, low cost,” map directly to three constraints that engineering teams cannot avoid: quality, time, and cost. Using this data as a starting point, this article does not focus on any specific product. Instead, it addresses three questions:
- What does each of the four metrics actually measure?
- Why do completion and elapsed time often move in the same direction?
- How should teams choose tools based on their own constraints and build a reproducible internal evaluation system?
1.1 Why Long-Horizon Tasks Are Difficult to Evaluate
Mobile application development is a typical long-chain engineering activity. Even a seemingly simple request, such as “add a user login page and integrate authentication,” usually involves:
- Reading the existing code structure.
- Creating new page and component files.
- Modifying dependency declarations and build configuration.
- Implementing business logic.
- Handling network requests and local data persistence.
- Handling permissions and lifecycle behavior.
- Compiling and running the app on a simulator or real device.
- Fixing compilation errors and runtime exceptions.
If any step fails, the user does not receive “partial results.” They receive a feature that cannot run. Therefore, an evaluation designed for engineering tasks must satisfy the following requirements:
- Deterministic: there are objective acceptance criteria that can be executed automatically.
- Reproducible: the environment, dependencies, and timeout policies are fixed.
- Measurable: in addition to the result, the evaluation records elapsed time, cost, and failure stage.
The metrics “pass rate, average completion, average elapsed time, and cost per run” are designed around these three requirements.
2. Breaking Down the Metrics: Engineering Meaning Behind Four Numbers
| Metric | What it measures | How it is calculated | Main limitation |
|---|---|---|---|
| Pass rate | Whether the task is truly completed | Number of tasks passing all acceptance criteria divided by total tasks, ranging from 0% to 100% | Binary judgment; cannot show how close a failed task was |
| Average completion | Value delivered by partial completion | Arithmetic mean of each task’s completion ratio | Definitions of completion ratio vary widely and may not be comparable across benchmarks |
| Average elapsed time | Delivery cycle | Mean wall-clock time from task start to task end | Affected by machine performance, build cache, and retry strategies |
| Cost per run | Economics | Total cost consumed by one complete evaluation run | Billing methods differ, and cache or batch discounts are handled differently |
2.1 Pass Rate: The “Last Mile” Problem Under Binary Judgment
Pass rate uses strict judgment: a task counts as passed only if its output passes all acceptance criteria. Partial completion is treated as failure. This most faithfully reflects delivery capability, but it does not show “how far away from completion” the result was.
The example above also shows that when average completion is above 80%, the pass rate is often only 20% to 30%. The gap indicates that most tasks are “almost complete” but fail at the final step. Common causes include:
- Incomplete handling of boundary inputs.
- Runtime failures under specific system versions or screen sizes.
- Code style that does not match existing conventions.
- Generated output that fails a full build or key-path test cases.
For engineering teams, this is more concerning than a completely wrong answer. Near-complete outputs often look correct and are more likely to be overlooked during review, allowing defects to flow into later stages.
2.2 Average Completion: Measuring Partial Value With Non-Uniform Definitions
Average completion gives continuous credit to partial completion. Within the same benchmark, it allows different models to be compared and can distinguish between “only one third completed” and “almost done but missing one part.”
However, completion depends on a specific scoring rubric. Three common approaches are:
- Task milestones: the task is split into steps with dependencies, and scoring is based on the number of passed milestones.
- Acceptance dimensions: scores are weighted across dimensions such as build, functionality, robustness, and code quality.
- Human evaluation: reviewers score outputs according to predefined rules.
Because these methods use different weights, cross-benchmark comparisons of completion should be treated cautiously. Meaningful comparison usually exists only within the same benchmark.
2.3 Average Elapsed Time and Cost: Two Often-Ignored Dimensions
Elapsed time and cost are often treated as secondary metrics, but they directly determine whether an agent can enter daily development workflows. Runs that take more than ten hours and cost several hundred USD per run are difficult to fit into requirement cycles measured in days. In contrast, low-cost runs measured in hours can be executed multiple times before review to compare several candidate approaches.
The cost structure of agent tasks differs from ordinary question-answering calls. It includes at least the following parts:
- Model inference consumption: long contexts are repeatedly reread, and multiple inference steps accumulate.
- Tool execution overhead: resource consumption from file read/write, command execution, builds, and tests.
- Retry and branching consumption: repair attempts after failures and parallel exploration of multiple candidate solutions.
Therefore, cost is usually the sum of inference cost and tool/environment cost. Looking only at a unit price can severely underestimate the real expense.
3. Engineering Trade-offs: Why High Completion Often Comes With Long Elapsed Time
“Higher completion means longer runtime” is not accidental. Several mechanisms are behind this pattern.
3.1 Exploration Depth and Verification Intensity Set the Quality Ceiling
High completion often comes from deeper task understanding and stricter self-verification. A typical agent loop looks like this:
Read task specification → Search relevant code and configuration → Create implementation plan→ Modify code → Run build├─ Build fails → Read error logs → Fix and rebuild (loop)└─ Build passes → Run tests├─ Tests fail → Locate cause → Fix and rerun (loop)└─ Tests pass → Produce change notes and finish
Each “fail, diagnose, fix, reverify” loop involves one round of reasoning, a set of tool calls, and one build or test execution. More loop rounds lead to higher completion, but they also increase elapsed time and cost.
3.2 Serial Waits Are Included in Total Elapsed Time
Mobile builds, dependency downloads, simulator startup and installation, and automated test execution are mostly serial and can each take several minutes. For a task that runs more than twenty verification loops, waiting alone may accumulate to several hours. This means average elapsed time is largely determined by task structure, not only by model generation speed.
3.3 The Quality–Time–Cost Triangle
- Fast trial-and-error: about 70% completion, average elapsed time of several hours, low per-run cost. Suitable for high-frequency, low-risk auxiliary tasks and scenarios where multiple candidate solutions need to be compared quickly.
- Deep polishing: more than 80% completion, average elapsed time of more than twenty hours, and per-run cost of several hundred USD. Suitable for key tasks with clear goals, clear acceptance criteria, and acceptable one-time investment.
Neither approach is absolutely better. The key question is whether the task is worth the investment:
- If requirements are vague and the solution is still being explored, deep polishing can easily lead to “doing the wrong thing at high cost” because of directional errors.
- If only fast trial-and-error is used for critical-path tasks, repeated rework may occur, and total cost may become higher.
3.4 Looking at Cost Through a Single Metric Can Be Misleading
If pass rate is divided by cost per run, the result is the average cost per passed task. For the two profiles above, the calculated values may both be around 20 to 30 USD, but their meanings are different:
- The deep-polishing profile spends most of its budget on a small number of near-complete tasks. If one passes, the output is often more complete and requires less rework.
- The fast trial-and-error profile has a low cost per run, but it may need multiple runs to cover one complete requirement.
Comparing only cost per run can lead to the opposite conclusion.
3.5 Coordinated Optimization of Pass Rate, Completion, and Cost
With a fixed budget, if multiple candidate solutions can be verified in parallel, a reasonable approach is to first use fast trial-and-error models to filter candidates. Use multiple low-cost runs to cover more solutions, keep the most promising few, and then apply deep polishing to them. This controls total budget while still preserving delivery quality. In essence, it uses parallelism to increase effective output per unit cost.
4. Practical Recommendations: From Benchmark Numbers to Team-Level Selection
4.1 Task Layering and Budget Allocation
Tasks can be layered by risk and automatability, with different execution strategies for each layer:
| Task layer | Typical tasks | Acceptance method | Recommended strategy |
|---|---|---|---|
| L1: High-frequency, low-risk | Boilerplate code generation, copy and asset replacement, simple refactoring | Build passes + unit tests | Fast trial-and-error |
| L2: Medium complexity | New feature modules, API integration, permission adaptation | Build + key-path tests | Layered: fast candidate generation first, then deep polishing of the best candidate |
| L3: Critical path | Core transactions, authentication, payments, data migration | Full regression + security scan + human review | Deep polishing + human fallback |
The following example code shows a task scheduling framework with budget guardrails. The configuration values are illustrative and should be adjusted according to actual team constraints:
import timefrom dataclasses import dataclass, fieldfrom typing import Callable@dataclassclass Task:task_id: strlevel: str # L1 / L2 / L3acceptance: Callable # Acceptance function: takes a working directory and returns (passed, completion)@dataclassclass RunConfig:max_turns: int = 40 # Maximum interaction turnsmax_builds: int = 6 # Maximum builds per taskwall_clock_limit: int = 7200 # Wall-clock timeout in secondsbudget_limit: float = 20.0 # Budget limit in normalized cost units@dataclassclass TaskResult:task_id: strpassed: boolcompletion: floatturns: intbuilds: intelapsed: floatcost: floatfail_stage: str = "" # build / test / timeout / budgetdef run_task(task: Task, cfg: RunConfig) -> TaskResult:start = time.monotonic()turns, builds, cost = 0, 0, 0.0workdir = f"/workspace/{task.task_id}"while turns < cfg.max_turns and builds < cfg.max_builds:# Budget and timeout guardrails: terminate immediately if any limit is exceeded, keeping current progressif time.monotonic() - start > cfg.wall_clock_limit:return TaskResult(task.task_id, False, 0.0, turns, builds,time.monotonic() - start, cost, "timeout")if cost >= cfg.budget_limit:return TaskResult(task.task_id, False, 0.0, turns, builds,time.monotonic() - start, cost, "budget")# Each turn: the agent reasons and produces an action; build_and_test is a placeholderturn_cost = 1.0 # Placeholder: in practice, replace this with observable usage statisticscost += turn_costturns += 1try:stage, passed, completion = build_and_test(workdir) # Build and testbuilds += 1except Exception:# Environment or tool exceptions count as failures to avoid silent passesreturn TaskResult(task.task_id, False, completion, turns, builds,time.monotonic() - start, cost, "build")if passed:# The final result uses the task acceptance function to prevent self-validation errorsok, final_completion = task.acceptance(workdir)return TaskResult(task.task_id, ok, final_completion, turns, builds,time.monotonic() - start, cost,"" if ok else "test")return TaskResult(task.task_id, False, 0.0, turns, builds,time.monotonic() - start, cost, "build")# Example configuration: critical-path tasks can have a higher budget limit but must have a hard timeoutKEY_PATH_CFG = RunConfig(max_turns=120, max_builds=20,wall_clock_limit=86400, budget_limit=300.0)
This framework has three design principles:
- Hard constraints take priority over soft goals: once the budget or timeout limit is reached, execution stops immediately and current progress is retained, avoiding “unlimited investment for completion.”
- Independent acceptance beyond self-validation: tests that an agent runs successfully are not the same as passing acceptance. The final judgment should be made by an independent acceptance function.
- Failure stages must be attributable: recording whether a result stopped at build, test, timeout, or budget provides a basis for later tuning.
4.2 Building Cost Guardrails and Observability
- Task-level budget limits: limit the maximum number of interaction turns, tool calls, and single execution duration in the agent configuration. Terminate immediately when limits are exceeded.
- Cost aggregation: aggregate invocation volume by team, project, and task to detect abnormal consumption early.
- Full-chain tracing: record input/output summaries, tool-call sequences, and elapsed time for each round to support later review.
- Logging and alerting: use logging and monitoring alert capabilities to set thresholds for sudden runtime increases or rising failure rates for a single task.
- Artifact archiving and message decoupling: intermediate artifacts such as build outputs and test reports can be archived to object storage, while build and test tasks can be smoothed and decoupled with message queues.
- Container isolation: use container platforms to provide a clean and reproducible environment for each run.
In a cloud environment, teams can combine log services, monitoring and alerting, object storage, message queues, and container platforms to form an end-to-end chain covering execution, observability, and archiving. Components can be selected as needed rather than being tied to a single option.
4.3 Validating Selection Conclusions With an Internal Evaluation Suite
Public leaderboards can only serve as an initial reference. A reliable basis for selection comes from the team’s own task set. It is recommended to extract tasks from real requirements and historical defects, and to pair them with executable acceptance scripts:
import timefrom concurrent.futures import ThreadPoolExecutorfrom statistics import meandef evaluate(candidates: dict, tasks: list[Task], cfg: RunConfig) -> dict:"""candidates: {solution name: adapter for run_task}"""report = {}for name, runner in candidates.items():with ThreadPoolExecutor(max_workers=4) as pool:# Candidate solutions are independent and can run in parallel; the same task can share the environment across solutionsresults = list(pool.map(lambda t: runner(t, cfg), tasks))n = len(results)report[name] = {"pass_rate": sum(r.passed for r in results) / n,"average_completion": mean(r.completion for r in results),"average_elapsed_seconds": mean(r.elapsed for r in results),"average_cost_per_run": mean(r.cost for r in results),"cost_per_passed_task": (sum(r.cost for r in results) /max(1, sum(r.passed for r in results))),"failure_stage_distribution": {s: sum(r.fail_stage == s for r in results)for s in ("build", "test", "timeout", "budget")},}return report
A self-built evaluation set should cover the following task types:
- Application creation and page building.
- Feature development and API integration.
- Legacy code migration.
- Cross-platform conversion.
- Defect fixing.
These task types roughly correspond to the capabilities covered by general mobile development benchmarks. Each solution should be run at least 3 to 5 times to reduce misjudgment caused by single-run variance.
4.4 Common Misreadings of Benchmarks
- Ignoring statistical variance: In a 30-task evaluation, one task changes the pass rate by about 3.3 percentage points, and agents may be stochastic. For example, if a solution passes 9 of 30 tasks, the observed pass rate is 30%; the confidence interval can still be wide, so adjacent rankings may simply be statistical noise. Similarly, average completion is the mean over 30 tasks, so one extreme task can visibly move the overall number.
- Treating completion as usability: 80% completion does not mean “80% usable.” If the missing parts include authentication, exception handling, or other critical components, the output may be completely unusable.
- Comparing only cost per run: It is necessary to calculate “cost per passed task” together with pass rate, and to consider hidden costs such as rework and manual repair.
- Ignoring the impact of the agent framework: The same model paired with different agents can produce very different results because tool-call strategies, context management, and retry logic differ. Benchmark results are a combination of “model + agent” and may not be reproducible after moving to a team’s own toolchain.
- Ignoring environment differences: Machine performance, build cache hit rate, simulator version, and timeout settings all affect elapsed time. Runtime data from different benchmarks should not be compared directly.
- Treating benchmarks as the endpoint: Evaluation sets may be widely cited, creating the risk of overlap with model training data. As a result, the discriminative power of public leaderboards may gradually decline. Continuous internal evaluation remains the reliable basis.
5. Conclusion
An average completion rate of more than 80% with a runtime of one day, or a pass rate of less than 20% with a runtime of only a few hours, is not inherently problematic. The real issue is using one of four metrics that should be considered together as the basis for a conclusion.
A more reasonable interpretation is:
- Pass rate: determines whether something is usable.
- Average completion: determines how far it is from being usable.
- Elapsed time: determines whether it can fit into the development rhythm.
- Cost: determines whether it can be used at scale.
For teams, the correct approach is not to chase leaderboard rankings, but to:
- Clarify priorities among quality, time, and cost.
- Establish task layering and budget guardrails.
- Continuously validate with an internal evaluation set that is close to real business needs.
- Let different types of agents handle tasks at different layers.
Only by turning evaluation metrics into executable engineering constraints can AI coding agents truly move from “demonstration tools” to “stable productivity tools.”
