# AI Coding Agent Benchmarking and Selection: Weighing Pass Rate, Completion, Runtime, and Cost for Mobile and Long-Horizon Tasks

    A practical framework for evaluating AI coding agents on long-horizon engineering work such as app creation, feature development, and cross-stack code migration. It explains the four standard benchmark metrics — pass rate, average completion, average runtime, and cost per run — why they cannot all be optimized at once, how to read the two common agent profiles on leaderboards, and how to build an internal, business-specific evaluation set with automated acceptance checks. It closes with the cloud-native building blocks needed to run repeatable, observable, regression-tested agent evaluations, and the core selection principle: choose the best return on investment under your own constraints, not the highest score.

    1. What the evaluation actually measures

    Over the past few years, AI coding agents have moved from “autocomplete a single line” to “take on a complete engineering task.” The focus of evaluation has shifted with them — away from the accuracy of static code generation and toward whether an agent can finish a long-horizon task.

    A long-horizon task is an engineering goal that requires multiple steps and coordination across files and modules before it can be delivered. Typical cases include:

    • Application creation: from scaffolding to a runnable prototype.
    • Feature development: adding or reworking business capability inside an existing codebase.
    • Code migration: moving an implementation from one technology stack to another.
    • Application conversion: cross-platform or cross-framework reimplementation with functional equivalence.

    Unlike traditional unit tests or function-level benchmarks, long-horizon tasks emphasize planning, decomposition, iteration, and self-correction inside a realistic engineering context. A single benchmark run often requires dozens of tool calls and code changes performed autonomously by the agent. As a result, the metrics have grown beyond “is it correct” to cover three further questions: can it deliver, how well does it deliver, and at what cost.

    2. Four key metrics and the tension between them

    A serious coding benchmark usually reports four dimensions at once. They are naturally coupled, and they pull against each other:

    1. Pass rate: the share of all long-horizon tasks judged to have fully met the acceptance criteria. It reflects delivery reliability and is the hardest of the metrics.

    2. Average completion: how far the model advanced the goal even when the task did not pass outright. A model with a mediocre pass rate but high completion often “missed only the last step,” which still carries considerable engineering value.

    3. Average runtime: wall-clock time to finish a batch of tasks. It directly determines the speed of the development feedback loop and has a large effect on interactive iteration.

    4. Cost per run: the compute or API spend required to complete one benchmark round. For teams that evaluate repeatedly or call agents frequently in continuous integration, cost is the constraint that decides whether an approach can scale.

    The most important thing to internalize is that these four metrics cannot all be optimal at the same time. Pass rate and completion typically demand more exploration, more verification, and more rollbacks, which naturally means longer runtimes and higher costs. Strategies that optimize for speed and low cost usually give up some pass rate on complex long-horizon tasks. The value of a leaderboard lies in quantifying and normalizing this trade-off — not in claiming that some universally superior model exists.

    3. Reading the “high completion, long runtime” trade-off

    Leaderboards commonly show two very different profiles, and each deserves its own reading.

    Profile A: high completion, long runtime, comparatively high cost. Agents of this type tend to decompose and verify problems thoroughly and are willing to spend many rounds closing in on the goal. Their high average completion means they are usually “one step away from delivery,” but the pass rate is held back by judgment errors in the final stage, and because they retry repeatedly, runtime can reach tens of hours and cost per run can reach the hundreds.

    Profile B: moderate pass rate, faster, low cost. Agents of this type produce usable results quickly within a limited number of steps, with runtimes measured in hours and costs of only a few to a few tens per run. Their advantage is return on investment per unit of output and feedback speed, which suits scenarios sensitive to budget and response time.

    Which profile to choose depends on the nature of the task, not on the leaderboard rank:

    • For one-off, low-tolerance, long-maintained core engineering (for example, infrastructure migration), the high-completion profile carries more value, because a single successful delivery amortizes the time cost.
    • For high-frequency, quickly verifiable, human-backstopped day-to-day development (for example, scaffolding generation or prototype validation), the speed and cost advantages of Profile B are often more useful in practice.

    4. Evaluating and selecting in a disciplined way

    Rather than chasing a single leaderboard position, make the evaluation method part of your own engineering system.

    4.1 Build an evaluation set close to your own business

    Generic leaderboard tasks may not reflect your technology stack. Extract a set of representative long-horizon tasks from real requirements, annotate clear acceptance criteria for each, and turn them into an internal benchmark. A schematic evaluation pipeline looks like this:

    1. # Conceptual sketch: skeleton of a long-horizon benchmark pipeline
    2. def run_benchmark(agent, tasks):
    3. results = []
    4. for task in tasks:
    5. start = time.time()
    6. attempt = agent.execute(task.prompt) # agent runs multiple rounds autonomously
    7. elapsed = time.time() - start
    8. passed = task.verify(attempt.diff) # automated acceptance check
    9. completion = task.grade(attempt.diff) # completion score from 0 to 1
    10. results.append({
    11. "task_id": task.id,
    12. "passed": passed,
    13. "completion": completion,
    14. "elapsed": elapsed,
    15. "cost": estimate_cost(attempt.tokens, attempt.steps),
    16. })
    17. return summarize(results) # aggregates pass rate / avg completion / avg runtime / cost

    4.2 Set a threshold for each of the four dimensions

    Do not let a single aggregate score hide the trade-off. Define a lower bound that your team can accept for each dimension, then rank the candidates that clear those bounds by business weighting. For example, weight pass rate fully for a core library migration, and weight runtime and cost fully for an internal tool prototype.

    4.3 Replace subjective judgment with automated acceptance

    Long-horizon tasks are the easiest to fool with work that merely “looks finished.” Always use automatable acceptance checks — compilation, unit tests, integration tests, static analysis — to decide whether a task truly passed, so that evaluation results are not skewed by human opinion.

    5. Running it in the cloud: keeping evaluation and agents stable

    To run this kind of evaluation and these agents in your engineering system over the long term and at scale, cloud-native capabilities matter. Only general capability categories are discussed here; nothing vendor-specific is implied.

    • Container platform: package the agent execution environment as reproducible sandbox containers that isolate the codebase, dependencies, and toolchain, keeping concurrency and results consistent across tasks.
    • CI/CD pipeline: attach the benchmark to your continuous integration flow so that every model or prompt update automatically triggers a regression run and capability degradation surfaces early.
    • Object storage: persist the code patches, build artifacts, and logs produced by every run, enabling side-by-side comparison and incident retrospection.
    • Logging, monitoring, and alerting: collect runtime metrics such as elapsed time, token consumption, and failure reasons, and surface abnormal fluctuations through dashboards and alerts.
    • Message queue: smooth out peaks and schedule asynchronous work for large volumes of evaluation tasks, balancing resource usage against execution throughput.

    As an example, Baidu AI Cloud lets a team combine general cloud products for containers, continuous delivery, logging, and monitoring to build a reproducible, observable, regression-tested agent evaluation pipeline — turning “selection” from a one-time decision into a capability that is operated continuously.

    6. Conclusion

    What an AI coding agent can do on mobile and long-horizon engineering tasks can no longer be summarized by a single score. Pass rate, completion, runtime, and cost form a set of trade-offs that cannot all be optimal at once: chasing high completion usually costs time and money, while chasing speed and low cost requires some compromise on complex tasks. The professional approach is to understand the engineering trade-off behind each profile, build an internal evaluation set and automated acceptance mechanism that fit your own business, and use general cloud capabilities to make the evaluation process repeatable and scalable. The goal of selection is not to find the “strongest model,” but to find the solution with the best return on investment under your specific business constraints.