AI Coding Agent Evaluation for Mobile Long-Horizon Tasks: Pass Rate, Completion, and Cost

    This article explains how to evaluate AI coding agents on long-horizon mobile development tasks. It breaks down pass rate, average completion, average elapsed time, and per-run cost, discusses why higher completion often requires more time and expense, and offers practical guidance for task layering, budget guardrails, observability, and reproducible internal evaluation.

    1. Introduction: From Code-Generation Evaluation to Engineering-Delivery Evaluation

    AI-assisted programming evaluation is shifting from “writing a correct piece of code” to “delivering a working feature.” The former focuses on code completion accuracy, unit test pass rate, and syntactic correctness, usually at the function or single-file level. The latter requires an AI agent to work continuously for tens of minutes or even hours, covering requirement understanding, code modification, build and compilation, and runtime verification.

    A recent mobile development benchmark update added results for multiple AI coding agents and showed an interesting split:

    • The leading performer achieved an average completion rate of more than 80%, ranked second in pass rate among all evaluated models, but took more than one day on average and cost several hundred USD per evaluation run.
    • Lightweight, fast models had clearly lower pass rates and average completion, but their average runtime was only a fraction of the leader’s and their per-run cost was less than 3% of it.

    These two profiles, “high completion, high runtime, high cost” versus “lower pass rate, low runtime, low cost,” map directly to three constraints that engineering teams cannot avoid: quality, time, and cost. Using this data as a starting point, this article does not focus on any specific product. Instead, it addresses three questions:

    • What does each of the four metrics actually measure?
    • Why do completion and elapsed time often move in the same direction?
    • How should teams choose tools based on their own constraints and build a reproducible internal evaluation system?

    1.1 Why Long-Horizon Tasks Are Difficult to Evaluate

    Mobile application development is a typical long-chain engineering activity. Even a seemingly simple request, such as “add a user login page and integrate authentication,” usually involves:

    • Reading the existing code structure.
    • Creating new page and component files.
    • Modifying dependency declarations and build configuration.
    • Implementing business logic.
    • Handling network requests and local data persistence.
    • Handling permissions and lifecycle behavior.
    • Compiling and running the app on a simulator or real device.
    • Fixing compilation errors and runtime exceptions.

    If any step fails, the user does not receive “partial results.” They receive a feature that cannot run. Therefore, an evaluation designed for engineering tasks must satisfy the following requirements:

    • Deterministic: there are objective acceptance criteria that can be executed automatically.
    • Reproducible: the environment, dependencies, and timeout policies are fixed.
    • Measurable: in addition to the result, the evaluation records elapsed time, cost, and failure stage.

    The metrics “pass rate, average completion, average elapsed time, and cost per run” are designed around these three requirements.

    2. Breaking Down the Metrics: Engineering Meaning Behind Four Numbers

    Metric What it measures How it is calculated Main limitation
    Pass rate Whether the task is truly completed Number of tasks passing all acceptance criteria divided by total tasks, ranging from 0% to 100% Binary judgment; cannot show how close a failed task was
    Average completion Value delivered by partial completion Arithmetic mean of each task’s completion ratio Definitions of completion ratio vary widely and may not be comparable across benchmarks
    Average elapsed time Delivery cycle Mean wall-clock time from task start to task end Affected by machine performance, build cache, and retry strategies
    Cost per run Economics Total cost consumed by one complete evaluation run Billing methods differ, and cache or batch discounts are handled differently

    2.1 Pass Rate: The “Last Mile” Problem Under Binary Judgment

    Pass rate uses strict judgment: a task counts as passed only if its output passes all acceptance criteria. Partial completion is treated as failure. This most faithfully reflects delivery capability, but it does not show “how far away from completion” the result was.

    The example above also shows that when average completion is above 80%, the pass rate is often only 20% to 30%. The gap indicates that most tasks are “almost complete” but fail at the final step. Common causes include:

    • Incomplete handling of boundary inputs.
    • Runtime failures under specific system versions or screen sizes.
    • Code style that does not match existing conventions.
    • Generated output that fails a full build or key-path test cases.

    For engineering teams, this is more concerning than a completely wrong answer. Near-complete outputs often look correct and are more likely to be overlooked during review, allowing defects to flow into later stages.

    2.2 Average Completion: Measuring Partial Value With Non-Uniform Definitions

    Average completion gives continuous credit to partial completion. Within the same benchmark, it allows different models to be compared and can distinguish between “only one third completed” and “almost done but missing one part.”

    However, completion depends on a specific scoring rubric. Three common approaches are:

    • Task milestones: the task is split into steps with dependencies, and scoring is based on the number of passed milestones.
    • Acceptance dimensions: scores are weighted across dimensions such as build, functionality, robustness, and code quality.
    • Human evaluation: reviewers score outputs according to predefined rules.

    Because these methods use different weights, cross-benchmark comparisons of completion should be treated cautiously. Meaningful comparison usually exists only within the same benchmark.

    2.3 Average Elapsed Time and Cost: Two Often-Ignored Dimensions

    Elapsed time and cost are often treated as secondary metrics, but they directly determine whether an agent can enter daily development workflows. Runs that take more than ten hours and cost several hundred USD per run are difficult to fit into requirement cycles measured in days. In contrast, low-cost runs measured in hours can be executed multiple times before review to compare several candidate approaches.

    The cost structure of agent tasks differs from ordinary question-answering calls. It includes at least the following parts:

    • Model inference consumption: long contexts are repeatedly reread, and multiple inference steps accumulate.
    • Tool execution overhead: resource consumption from file read/write, command execution, builds, and tests.
    • Retry and branching consumption: repair attempts after failures and parallel exploration of multiple candidate solutions.

    Therefore, cost is usually the sum of inference cost and tool/environment cost. Looking only at a unit price can severely underestimate the real expense.

    3. Engineering Trade-offs: Why High Completion Often Comes With Long Elapsed Time

    “Higher completion means longer runtime” is not accidental. Several mechanisms are behind this pattern.

    3.1 Exploration Depth and Verification Intensity Set the Quality Ceiling

    High completion often comes from deeper task understanding and stricter self-verification. A typical agent loop looks like this:

    1. Read task specification → Search relevant code and configuration → Create implementation plan
    2. → Modify code → Run build
    3. ├─ Build fails → Read error logs → Fix and rebuild (loop)
    4. └─ Build passes → Run tests
    5. ├─ Tests fail → Locate cause → Fix and rerun (loop)
    6. └─ Tests pass → Produce change notes and finish

    Each “fail, diagnose, fix, reverify” loop involves one round of reasoning, a set of tool calls, and one build or test execution. More loop rounds lead to higher completion, but they also increase elapsed time and cost.

    3.2 Serial Waits Are Included in Total Elapsed Time

    Mobile builds, dependency downloads, simulator startup and installation, and automated test execution are mostly serial and can each take several minutes. For a task that runs more than twenty verification loops, waiting alone may accumulate to several hours. This means average elapsed time is largely determined by task structure, not only by model generation speed.

    3.3 The Quality–Time–Cost Triangle

    • Fast trial-and-error: about 70% completion, average elapsed time of several hours, low per-run cost. Suitable for high-frequency, low-risk auxiliary tasks and scenarios where multiple candidate solutions need to be compared quickly.
    • Deep polishing: more than 80% completion, average elapsed time of more than twenty hours, and per-run cost of several hundred USD. Suitable for key tasks with clear goals, clear acceptance criteria, and acceptable one-time investment.

    Neither approach is absolutely better. The key question is whether the task is worth the investment:

    • If requirements are vague and the solution is still being explored, deep polishing can easily lead to “doing the wrong thing at high cost” because of directional errors.
    • If only fast trial-and-error is used for critical-path tasks, repeated rework may occur, and total cost may become higher.

    3.4 Looking at Cost Through a Single Metric Can Be Misleading

    If pass rate is divided by cost per run, the result is the average cost per passed task. For the two profiles above, the calculated values may both be around 20 to 30 USD, but their meanings are different:

    • The deep-polishing profile spends most of its budget on a small number of near-complete tasks. If one passes, the output is often more complete and requires less rework.
    • The fast trial-and-error profile has a low cost per run, but it may need multiple runs to cover one complete requirement.

    Comparing only cost per run can lead to the opposite conclusion.

    3.5 Coordinated Optimization of Pass Rate, Completion, and Cost

    With a fixed budget, if multiple candidate solutions can be verified in parallel, a reasonable approach is to first use fast trial-and-error models to filter candidates. Use multiple low-cost runs to cover more solutions, keep the most promising few, and then apply deep polishing to them. This controls total budget while still preserving delivery quality. In essence, it uses parallelism to increase effective output per unit cost.

    4. Practical Recommendations: From Benchmark Numbers to Team-Level Selection

    4.1 Task Layering and Budget Allocation

    Tasks can be layered by risk and automatability, with different execution strategies for each layer:

    Task layer Typical tasks Acceptance method Recommended strategy
    L1: High-frequency, low-risk Boilerplate code generation, copy and asset replacement, simple refactoring Build passes + unit tests Fast trial-and-error
    L2: Medium complexity New feature modules, API integration, permission adaptation Build + key-path tests Layered: fast candidate generation first, then deep polishing of the best candidate
    L3: Critical path Core transactions, authentication, payments, data migration Full regression + security scan + human review Deep polishing + human fallback

    The following example code shows a task scheduling framework with budget guardrails. The configuration values are illustrative and should be adjusted according to actual team constraints:

    1. import time
    2. from dataclasses import dataclass, field
    3. from typing import Callable
    4. @dataclass
    5. class Task:
    6. task_id: str
    7. level: str # L1 / L2 / L3
    8. acceptance: Callable # Acceptance function: takes a working directory and returns (passed, completion)
    9. @dataclass
    10. class RunConfig:
    11. max_turns: int = 40 # Maximum interaction turns
    12. max_builds: int = 6 # Maximum builds per task
    13. wall_clock_limit: int = 7200 # Wall-clock timeout in seconds
    14. budget_limit: float = 20.0 # Budget limit in normalized cost units
    15. @dataclass
    16. class TaskResult:
    17. task_id: str
    18. passed: bool
    19. completion: float
    20. turns: int
    21. builds: int
    22. elapsed: float
    23. cost: float
    24. fail_stage: str = "" # build / test / timeout / budget
    25. def run_task(task: Task, cfg: RunConfig) -> TaskResult:
    26. start = time.monotonic()
    27. turns, builds, cost = 0, 0, 0.0
    28. workdir = f"/workspace/{task.task_id}"
    29. while turns < cfg.max_turns and builds < cfg.max_builds:
    30. # Budget and timeout guardrails: terminate immediately if any limit is exceeded, keeping current progress
    31. if time.monotonic() - start > cfg.wall_clock_limit:
    32. return TaskResult(task.task_id, False, 0.0, turns, builds,
    33. time.monotonic() - start, cost, "timeout")
    34. if cost >= cfg.budget_limit:
    35. return TaskResult(task.task_id, False, 0.0, turns, builds,
    36. time.monotonic() - start, cost, "budget")
    37. # Each turn: the agent reasons and produces an action; build_and_test is a placeholder
    38. turn_cost = 1.0 # Placeholder: in practice, replace this with observable usage statistics
    39. cost += turn_cost
    40. turns += 1
    41. try:
    42. stage, passed, completion = build_and_test(workdir) # Build and test
    43. builds += 1
    44. except Exception:
    45. # Environment or tool exceptions count as failures to avoid silent passes
    46. return TaskResult(task.task_id, False, completion, turns, builds,
    47. time.monotonic() - start, cost, "build")
    48. if passed:
    49. # The final result uses the task acceptance function to prevent self-validation errors
    50. ok, final_completion = task.acceptance(workdir)
    51. return TaskResult(task.task_id, ok, final_completion, turns, builds,
    52. time.monotonic() - start, cost,
    53. "" if ok else "test")
    54. return TaskResult(task.task_id, False, 0.0, turns, builds,
    55. time.monotonic() - start, cost, "build")
    56. # Example configuration: critical-path tasks can have a higher budget limit but must have a hard timeout
    57. KEY_PATH_CFG = RunConfig(max_turns=120, max_builds=20,
    58. wall_clock_limit=86400, budget_limit=300.0)

    This framework has three design principles:

    • Hard constraints take priority over soft goals: once the budget or timeout limit is reached, execution stops immediately and current progress is retained, avoiding “unlimited investment for completion.”
    • Independent acceptance beyond self-validation: tests that an agent runs successfully are not the same as passing acceptance. The final judgment should be made by an independent acceptance function.
    • Failure stages must be attributable: recording whether a result stopped at build, test, timeout, or budget provides a basis for later tuning.

    4.2 Building Cost Guardrails and Observability

    • Task-level budget limits: limit the maximum number of interaction turns, tool calls, and single execution duration in the agent configuration. Terminate immediately when limits are exceeded.
    • Cost aggregation: aggregate invocation volume by team, project, and task to detect abnormal consumption early.
    • Full-chain tracing: record input/output summaries, tool-call sequences, and elapsed time for each round to support later review.
    • Logging and alerting: use logging and monitoring alert capabilities to set thresholds for sudden runtime increases or rising failure rates for a single task.
    • Artifact archiving and message decoupling: intermediate artifacts such as build outputs and test reports can be archived to object storage, while build and test tasks can be smoothed and decoupled with message queues.
    • Container isolation: use container platforms to provide a clean and reproducible environment for each run.

    In a cloud environment, teams can combine log services, monitoring and alerting, object storage, message queues, and container platforms to form an end-to-end chain covering execution, observability, and archiving. Components can be selected as needed rather than being tied to a single option.

    4.3 Validating Selection Conclusions With an Internal Evaluation Suite

    Public leaderboards can only serve as an initial reference. A reliable basis for selection comes from the team’s own task set. It is recommended to extract tasks from real requirements and historical defects, and to pair them with executable acceptance scripts:

    1. import time
    2. from concurrent.futures import ThreadPoolExecutor
    3. from statistics import mean
    4. def evaluate(candidates: dict, tasks: list[Task], cfg: RunConfig) -> dict:
    5. """candidates: {solution name: adapter for run_task}"""
    6. report = {}
    7. for name, runner in candidates.items():
    8. with ThreadPoolExecutor(max_workers=4) as pool:
    9. # Candidate solutions are independent and can run in parallel; the same task can share the environment across solutions
    10. results = list(pool.map(lambda t: runner(t, cfg), tasks))
    11. n = len(results)
    12. report[name] = {
    13. "pass_rate": sum(r.passed for r in results) / n,
    14. "average_completion": mean(r.completion for r in results),
    15. "average_elapsed_seconds": mean(r.elapsed for r in results),
    16. "average_cost_per_run": mean(r.cost for r in results),
    17. "cost_per_passed_task": (sum(r.cost for r in results) /
    18. max(1, sum(r.passed for r in results))),
    19. "failure_stage_distribution": {
    20. s: sum(r.fail_stage == s for r in results)
    21. for s in ("build", "test", "timeout", "budget")
    22. },
    23. }
    24. return report

    A self-built evaluation set should cover the following task types:

    • Application creation and page building.
    • Feature development and API integration.
    • Legacy code migration.
    • Cross-platform conversion.
    • Defect fixing.

    These task types roughly correspond to the capabilities covered by general mobile development benchmarks. Each solution should be run at least 3 to 5 times to reduce misjudgment caused by single-run variance.

    4.4 Common Misreadings of Benchmarks

    • Ignoring statistical variance: In a 30-task evaluation, one task changes the pass rate by about 3.3 percentage points, and agents may be stochastic. For example, if a solution passes 9 of 30 tasks, the observed pass rate is 30%; the confidence interval can still be wide, so adjacent rankings may simply be statistical noise. Similarly, average completion is the mean over 30 tasks, so one extreme task can visibly move the overall number.
    • Treating completion as usability: 80% completion does not mean “80% usable.” If the missing parts include authentication, exception handling, or other critical components, the output may be completely unusable.
    • Comparing only cost per run: It is necessary to calculate “cost per passed task” together with pass rate, and to consider hidden costs such as rework and manual repair.
    • Ignoring the impact of the agent framework: The same model paired with different agents can produce very different results because tool-call strategies, context management, and retry logic differ. Benchmark results are a combination of “model + agent” and may not be reproducible after moving to a team’s own toolchain.
    • Ignoring environment differences: Machine performance, build cache hit rate, simulator version, and timeout settings all affect elapsed time. Runtime data from different benchmarks should not be compared directly.
    • Treating benchmarks as the endpoint: Evaluation sets may be widely cited, creating the risk of overlap with model training data. As a result, the discriminative power of public leaderboards may gradually decline. Continuous internal evaluation remains the reliable basis.

    5. Conclusion

    An average completion rate of more than 80% with a runtime of one day, or a pass rate of less than 20% with a runtime of only a few hours, is not inherently problematic. The real issue is using one of four metrics that should be considered together as the basis for a conclusion.

    A more reasonable interpretation is:

    • Pass rate: determines whether something is usable.
    • Average completion: determines how far it is from being usable.
    • Elapsed time: determines whether it can fit into the development rhythm.
    • Cost: determines whether it can be used at scale.

    For teams, the correct approach is not to chase leaderboard rankings, but to:

    1. Clarify priorities among quality, time, and cost.
    2. Establish task layering and budget guardrails.
    3. Continuously validate with an internal evaluation set that is close to real business needs.
    4. Let different types of agents handle tasks at different layers.

    Only by turning evaluation metrics into executable engineering constraints can AI coding agents truly move from “demonstration tools” to “stable productivity tools.”