AI Coding Agent Benchmarks for Long-Horizon Tasks: Metrics, Trade-offs, and a Selection Guide

    A public mobile app development benchmark put three representative models through end-to-end engineering work, and the highest pass rate was only 28.7%, with the other two at 24.0% and 17.3%. One agent reached 82.3% average completion but needed roughly 27.4 hours and about $317.4 per run. This article breaks down what long-horizon coding benchmarks actually measure — task pass rate, average completion, average wall-clock time, and cost per run — plus derived metrics such as cost per passing task. It also explains why a ~30% ceiling and slow, expensive runs are expected rather than anomalous, how to build an objective and reproducible internal evaluation harness (task set design, isolated sandboxes, executor flow, observability), and how to turn benchmark numbers into a tiered agent selection strategy instead of defaulting to "the strongest model."

    1. Evaluation context: AI coding agents are moving from code completion to engineering delivery

    The capability boundary of AI coding tools has shifted noticeably over the past two years. What started as single-line completion and function-level generation has evolved into cross-file modification, automated test authoring and repair, and even building a complete application from an empty repository. The evaluation focus has shifted with it: the question is no longer “is the code correct?” but “was the task actually completed?”

    Against this backdrop, a public mobile app development benchmark reported results for three representative models. The highest pass rate was 28.7%; the other two scored 24.0% and 17.3%. That raises two core questions:

    1. Why does even the strongest model clear only about a third of tasks on long-horizon work?
    2. Why did one model reach 82.3% average completion while averaging close to 27.4 hours and $317.4 per run?

    Answering these requires first pinning down what the benchmark measures, how each metric is computed, and what those metrics say about engineering reality.

    1.1 Why mobile development is a canonical long-horizon scenario

    Mobile development works well as a long-horizon evaluation setting mainly because it closely mirrors real engineering delivery:

    1. Deep toolchain dependencies. Developers juggle build systems, dependency management, lint and style checks, test frameworks, and simulators at the same time. An agent must not only write code but also invoke tools and interpret their output.
    2. Strict environment constraints. Platform versions, device screen sizes, permission models, and signing and installation rules all affect the final result. Code that looks logically sound can still break during a real build or at runtime.
    3. Long feedback loops. Writing code, adjusting configuration, compiling, running, and reading logs — a failure in any single stage can block everything downstream, so the agent has to locate and correct problems repeatedly.
    4. Dense domain knowledge. UI layout, lifecycle management, data storage, and background execution all come with explicit platform conventions. Cross-version migration additionally involves deprecated APIs and compatibility concerns.

    These traits align closely with enterprise software engineering, which is why mobile development is a good test of long-horizon agent capability.

    1.2 Task categories covered by the benchmark

    Benchmarks of this kind are typically built around four categories of real engineering work:

    Task type Typical content Core capabilities tested
    App creation Scaffolding a project from scratch, with a runnable main screen and basic interactions Planning, scaffolding, environment setup
    Feature development Adding modules to an existing project: lists, forms, data persistence, network requests Code comprehension, module design, local/global consistency
    Code migration Moving old implementations to new APIs or new idioms, across multiple files and versions Global search, compatibility judgment, regression safety
    App conversion Rewriting across stacks or platforms, translating one UI system into an equivalent other Semantic understanding, architectural refactoring, feature parity

    2. Reading the core metrics: pass rate is not the whole story

    The benchmark treats a set of long-horizon tasks as the unit of work, focusing on whether an agent can independently complete app creation, feature development, code migration, and app conversion. Four headline metrics are reported: task pass rate, average completion, average wall-clock time, and cost per run.

    2.1 Pass rate: a measure of end-to-end usability

    Pass rate is the share of tasks an agent completes on its own while satisfying every acceptance criterion. “Passing” here does not mean “the code looks fine.” It means usable in an engineering sense, for example:

    • The build succeeds with no errors and no blocking warnings;
    • All key tests or acceptance scripts pass;
    • Functional behavior matches what the task specified.

    Why is a pass rate around 30% not surprising?

    • Error accumulation. The success probability of a long-horizon task approximates the product of per-step success probabilities. Chain 20 critical steps together, and even a 95% success rate per step yields an overall success rate of roughly 36%.
    • Long-tail failures. Agent mistakes cluster in a handful of places: dependency version conflicts, improper lifecycle handling, async and threading issues, missing edge-case configuration. Each individual failure is unlikely, but over dozens of steps it gets triggered.
    • Strict judgment. A single blocking issue fails the whole task. A high failure count does not mean the output has no value.

    Note also that pass rate depends on boundary conditions: task count, how strict the acceptance scripts are, timeout thresholds, and retry allowances. Results from different benchmarks therefore cannot be compared directly. Pass rate is best read as a signal of the capability ceiling, not as a direct proxy for production readiness.

    2.2 Average completion: how close is “nearly usable”?

    Average completion is a graded metric between 0% and 100%. It is typically computed by scoring agent output against a predefined set of checkpoints or an acceptance checklist, weighted to express how close the result is to the intended deliverable.

    One model in the benchmark scored 82.3% average completion against a 24.0% pass rate — a 58.3-point gap. That points to one important fact:

    A large share of tasks are not outright failures; most of the work is correct, and only a few steps remain unresolved.

    To avoid misinterpretation, completion should not be read as the model’s subjective self-assessment. A sounder approach is for the evaluator to define the checkpoints up front and publish where the weights come from. For example:

    • Build succeeds: 20%;
    • Core screens run: 20%;
    • Key interactions work: 25%;
    • Data persistence or network requests work: 15%;
    • Tests pass: 10%;
    • Code style, logging, and basic documentation: 10%.

    For failed samples, a manual review rule can further classify whether the cause was the environment, the acceptance script, or a genuine gap in agent work.

    For engineering teams, high-completion output often still carries value. Finishing the last step usually costs far less than starting from zero — which is why some teams are willing to pay a premium for higher completion.

    2.3 Average wall-clock time and cost per run: two yardsticks for efficiency

    • Average wall-clock time is the mean time from agent start to finish across all required tasks. It includes model inference, tool invocation, build and test execution, and failed retries — not just model response latency.
    • Cost per run is the total spend of one complete evaluation pass, dominated by model invocation fees. Model pricing, context length, retry counts, and tool-call frequency all feed directly into this number.

    Side by side, capable models and lightweight models differ sharply:

    Model (anonymized) Pass rate Avg. completion Avg. time Cost per run Cost per passing task
    Highest-pass-rate model among the three 28.7% — — Not published Cannot be computed precisely
    Agent A 24.0% 82.3% 27.4 hours ~$317.4 ~$44.1
    Agent B 17.3% 71.2% 12.0 hours ~$9.9 ~$1.91

    Note: cost per passing task is computed as cost per run ÷ (number of tasks × pass rate). Assuming 30 tasks in this round, Agent A is roughly 317.4 ÷ (30 × 24.0%) ≈ $44.1, and Agent B roughly 9.9 ÷ (30 × 17.3%) ≈ $1.91.

    2.4 Three derived metrics that are easy to overlook

    Raw numbers alone invite one-sided conclusions. Three derived metrics are worth adding.

    1. Cost per passing task

      Cost per passing task = cost per run ÷ (number of tasks × pass rate)

      Given the same task count, this tracks real efficiency more closely than cost per run does. On 30 tasks, Agent A lands near $44.1 and Agent B near $1.91 — about a 23.1x difference.

    2. Average time per task

      With 30 tasks as the base, 27.4 hours works out to roughly 55 minutes per task, while 12.0 hours works out to about 24 minutes. A higher pass rate often means the agent completed more steps and triggered more builds and test runs, which naturally stretches the clock.

    3. Apples-to-apples cost accounting

      If two models have the same cost per run and the same task count, cost per passing task can be compared directly. If one model’s cost per run is unpublished, you should not back-solve for “how much a one-point gain costs.” Without a complete cost basis, forcing a marginal-cost calculation will mislead the selection process.

    3. What high completion and long runtime reveal about engineering trade-offs

    Agent A’s profile captures a set of trade-offs common to today’s long-horizon coding agents: high completion tends to come with high latency and high cost. This is not unique to one product and is worth unpacking from an engineering standpoint.

    3.1 Reasoning budget correlates with completion

    High completion usually comes from a larger reasoning and verification budget. Concretely:

    • More thorough planning: decompose the task first, then implement step by step;
    • More thorough retrieval: read the relevant files and call relationships before editing;
    • Stricter self-verification: use compilation, tests, and static analysis to find and fix errors.

    All of these raise completion — and directly increase time and cost. Agents tuned for high-quality output spend more reasoning steps, so they cost more than lightweight, fast-response models.

    3.2 Long-horizon work has inherent sequential dependencies

    Unlike batch code generation, which can be parallelized, long-horizon tasks are strongly sequential: the project skeleton must exist before UI work begins, and the UI must run before interaction logic can be tested. Each step consumes the previous step’s output, leaving limited room for parallel speedups.

    Each task is also typically a fresh repository, so caches and context from a previous build cannot be reused. Running 30 tasks one after another means total time grows roughly linearly — or worse — with task complexity.

    3.3 The last mile is the hardest and the most expensive

    Failures concentrate in the final verification stage: build errors, runtime crashes, failing tests. Diagnosing them requires:

    • Reading and understanding long logs;
    • Trial and error across several candidate causes;
    • Regression-verifying each change.

    These operations generate a large volume of extra input and output and cannot easily be skipped in advance, which makes them the main source of time and cost.

    3.4 Cost scales with context and step count

    Long contexts and multi-step execution amplify cost twice over:

    • Every call carries the accumulated history, so input consumption grows with each turn;
    • Every additional tool call plus result readback means another full model inference.

    Across 30 tasks with dozens of interactions each, this adds up to hundreds of hours and hundreds of invocations. The $317.4 cost per run comes directly from this long-chain execution.

    3.5 Hidden risk: long-tail waiting and observability

    An average of 27.4 hours does not mean every task takes 27.4 hours — a substantial portion may run far longer. Dropped straight into a CI pipeline, this can leave jobs hanging for long periods and block the queue.

    To reduce that risk, an evaluation system should support:

    • Checkpoint and resume;
    • Failure snapshots;
    • Log correlation to task IDs;
    • Timeout state marking;
    • Alerts on retry counts and abnormal cost.

    Without these observability capabilities, the cost of diagnosing failures grows further.

    3.6 What this means for teams

    • High-capability agents suit quality-first, time-tolerant work, such as cross-version migration and complex module refactoring.
    • Lightweight agents suit budget-sensitive, responsiveness-first work, such as routine iteration, batch boilerplate generation, and first drafts.
    • The two are not substitutes; layer them according to task value and timing requirements.

    4. How to build a long-horizon evaluation system in practice

    Teams that want to run agent evaluations internally should focus on making results objective, reproducible, and comparable.

    4.1 Task set design

    • Cover typical scenarios. Balance the four categories — creation, development, migration, conversion — and vary difficulty within each.
    • Stay close to real business needs. Derive tasks from actual requirements and production defects rather than textbook examples.
    • Make acceptance criteria explicit. Pair every task with an automatically executable acceptance script and define a quantifiable completion-scoring rule.
    • Control the size. Aim for at least 20–30 tasks and refresh them periodically to guard against benchmark-gaming optimizations.

    4.2 An isolated, reproducible sandbox environment

    • Pin the image. Standardize build tooling, runtimes, and device image versions so environment differences do not skew results.
    • Control the network. Pre-seed dependency caches and, where necessary, restrict outbound access to reduce nondeterminism.
    • Isolate resources. Give each task its own container or VM to prevent cross-contamination.
    • Keep snapshots. Preserve the initial repository snapshot and the final artifacts for review and reproduction.

    4.3 A well-specified execution flow

    An evaluator can follow a structure like this pseudocode:

    1. def evaluate(agent, tasks, policy):
    2. results = []
    3. for task in tasks: # run each task in isolation
    4. sandbox = create_sandbox(task.image)
    5. budget = Budget(
    6. max_steps=policy.max_steps, # step limit
    7. max_tokens=policy.max_tokens, # token budget
    8. timeout_hours=policy.timeout, # timeout threshold
    9. )
    10. with sandbox:
    11. repo = seed_repository(task.snapshot)
    12. transcript = agent.solve(repo, task.prompt, budget)
    13. build = run_build(repo) # unified build
    14. tests = run_acceptance(repo) # unified acceptance
    15. passed = build.ok and tests.all_passed
    16. completion = score_checklist(task.checklist, repo, tests)
    17. results.append({
    18. "task": task.id,
    19. "passed": passed,
    20. "completion": completion,
    21. "wall_clock_h": budget.elapsed_hours,
    22. "tokens": budget.tokens_used,
    23. "steps": budget.steps_used,
    24. "cost_usd": estimate_cost(budget),
    25. "timed_out": budget.timed_out,
    26. "transcript": transcript, # full execution trace
    27. })
    28. destroy_sandbox(sandbox)
    29. return summarize(results) # summarize the four core metrics

    The cost accounting convention matters here:

    • estimate_cost(budget) can be estimated as tokens_used × unit price + tool invocation fees + retry costs;
    • cost_usd is the aggregated cost of the whole task round, not the cost of a single model response;
    • budget.elapsed_hours records actual wall-clock time from agent start to finish;
    • tokens_used should record the sum of input and output tokens;
    • steps_used should record tool calls, command executions, or action steps.

    Other design points to keep in mind:

    • Timeout and budget control: prevent any single task from running indefinitely; timed-out tasks must be explicitly recorded as “not completed” and tracked separately.
    • Retry limits: prevent repeated trial and error from blowing up costs.
    • Concurrency control: decide how many tasks run in parallel based on available compute and cost budget.
    • Complete tracing: record every prompt, tool call, and result to support failure attribution and model iteration.
    • Average over multiple runs: for agents with high variance, run several times and report the mean or median to damp out random fluctuations.

    4.4 Observability and data management

    • Use a centralized logging service to collect execution traces and error output;
    • Use monitoring and alerting to flag timeouts, sudden spikes in failure rate, and abnormal cost;
    • Archive task repositories, build artifacts, and evaluation reports together for versioning and traceability.

    For long-running, large-scale evaluation, the evaluator can be containerized and deployed on a container orchestration platform with elastic scaling, alongside a message queue for task distribution. General-purpose cloud services of this kind help teams build a stable, observable agent evaluation pipeline.

    5. From evaluation to selection: practical guidance for teams

    How do you choose an AI coding agent?

    Use a tiered selection strategy rather than simply picking “the strongest model”:

    1. High-value, long-cycle tasks. For work such as cross-version migration and complex module refactoring, prefer a high-capability agent and accept the higher time and cost.
    2. Routine iteration. For small feature work and bug fixes, choose a balanced agent that trades off quality against efficiency.
    3. Batch lightweight tasks. For boilerplate, basic screens, and draft generation, choose a lightweight agent and prioritize response speed and predictable spend.

    What mistakes are common?

    • Looking only at cost per run and ignoring cost per passing task;
    • Looking only at pass rate and ignoring average completion and recoverable value;
    • Comparing results across different benchmarks and acceptance criteria as if they were equivalent;
    • Setting no timeout, token cap, or retry limit, and losing control of cost;
    • Merging high-risk migration work into the main branch without human review.

    What should you watch when rolling this out?

    • Set hard cost limits. Enforce per-task token caps, timeout thresholds, and budget alerts at the toolchain level.
    • Keep a human in the loop. For high-risk work such as migration and conversion, require the agent to produce a change summary and impact-scope list, and have a human review the critical path before merging.
    • Build an internal benchmark. Accumulate a dedicated evaluation set from your own codebase and track how different models and configuration versions perform over time.
    • Invest in context assets. Giving agents clear architecture docs, coding standards, reusable scaffolding, and automated tests often raises pass rate more than swapping models.

    6. Conclusion

    Three takeaways summarize what the benchmark shows:

    • Pass rate measures the usability ceiling. A level around 30% means agents can already complete a meaningful share of mobile development tasks independently, but they remain far from fully automated delivery.
    • Completion measures recoverable value. At 82.3% average completion, most failures are not total failures, and having a human close the last mile remains a highly cost-effective way to collaborate.
    • Time and cost reflect the price of capability. High completion usually costs more time and more money, so selection has to make an explicit trade-off among quality, time, and budget.

    For developers and engineering leaders, building an evaluation and rollout system tailored to your own business matters more than chasing leaderboard position: measure capability with clear metrics, keep results trustworthy with isolated environments, and control cost with a tiered strategy. That is how AI coding agents move from impressive demos to genuine engineering utility.