AI Coding Agent Benchmarks: How to Read the Data for Mobile App Development
AI coding agent benchmarks now evaluate long-horizon engineering tasks with metrics such as pass rate, average completion, average runtime, and cost per run. This article explains how to interpret these scores, understand their limitations, and turn public benchmark results into a practical internal evaluation and model-selection workflow for development teams.
Several major mobile app development platforms have recently published benchmark evaluations for AI coding agents. The reports use a set of long-horizon engineering tasks to assess how agents perform across the full workflow, from creating an app and developing features to migrating code and converting an application. They also report metrics such as pass rate, average completion, average runtime, and cost per run. For teams evaluating AI coding tools, these scores are useful, but how to read them—and what they can and cannot prove—often matters more than the numbers themselves. Using mobile app development as a representative scenario, this article explains the methodology behind agent benchmarks and helps teams turn “the model can write code” into “the team can deliver on time.”
1. From Code Completion to Long-Horizon Tasks: The Shift in AI Coding Evaluation
Traditional evaluations of AI coding ability mostly focus on function-level or file-level tasks, such as generating code from a problem description or completing a function based on preceding code. These tasks have clear boundaries and usually rely on unit tests to decide success. In real engineering work, however, developers spend a large portion of their time understanding requirements, mapping dependencies, editing multiple places in a codebase, fixing build errors, and running regression checks. These are exactly the areas that single-file benchmarks cannot cover.
Agents change this landscape. An agent can not only generate code, but also plan steps autonomously, invoke terminal and editing tools, inspect project structure, and retry or correct itself when something fails. This allows it to complete complex tasks over a longer period. Benchmarks have evolved accordingly: instead of “write a function,” the task becomes “build a runnable application from scratch,” or “add a persistent login feature to an existing project.”
This type of benchmark is often called long-horizon evaluation. One key characteristic is that tasks can take hours. A complete benchmark run may require continuous execution for tens of hours or longer. For an agent, this means it must maintain the goal over a long period, remember what it has already done and what remains, and diagnose or fix issues when tool calls fail or tests behave unexpectedly. The following examples are drawn from public agent benchmark tasks and belong to this category:
- Add login to a sample app with four tabs. The agent must create a login page and mark tabs and detail pages as requiring login.
- Add a daily check-in feature. After seven consecutive check-ins, the user receives an item, and this state can be read across the entire app.
These descriptions are short, but they imply multiple constraints: page construction, conditional navigation, shared state across pages, persistent storage, and UI feedback. Any one item alone is not difficult; the combination tests the agent’s planning ability and long-horizon memory.
For engineering teams, these tasks closely resemble everyday work: delivering a relatively independent, verifiable feature inside an existing codebase. That is where the value of long-horizon benchmarks lies. However, it is also important to recognize that tasks in public benchmarks come from multiple open-source projects with different languages, frameworks, and technology stacks. If a team’s technical environment differs significantly, the benchmark’s reference value will decrease.
2. Reading the Four Key Metrics
Common metrics in public evaluations can be grouped into four categories: pass rate, average completion, average runtime, and cost per run. They correspond to four questions: Is it correct? How far did it get? How long did it take? How much did it cost?
1. Pass Rate: The Strictest Metric and Closest to Delivery Criteria
Pass rate is the proportion of tasks that ultimately pass acceptance. It usually follows an all-or-nothing rule: only when the build succeeds, all automated tests pass, and every acceptance criterion is satisfied is the task counted as completed. If any condition is not met, the entire task is marked as failed.
Its advantage is that it is closest to real delivery standards and is easy to compare across models or versions. Its disadvantage is significant information loss. A model with a pass rate of just over 20% does not mean the remaining 70% of tasks show no progress. Many of those tasks may be mostly complete and blocked only by one or two final acceptance conditions.
2. Average Completion: Measuring How Far a Task Gets
Average completion measures how far tasks have progressed. A practical engineering approximation is to split acceptance criteria into independently verifiable checks, calculate the proportion of passed checks, and then average this proportion across all tasks.
def completion_rate(checks: list[bool]) -> float:"""checks: per-item acceptance results; True means the item is satisfied."""if not checks:return 0.0return sum(checks) / len(checks)
Different benchmarks may calculate completion differently, so it is best used for longitudinal comparison within the same evaluation system. Changes in capability between model versions are often more useful than a single absolute value.
3. Average Runtime: The Agent’s Delivery Cycle
Average runtime is the time from the start to the end of a task. It measures the agent’s end-to-end throughput and is influenced by three factors: the model’s inference speed, the task execution strategy (such as parallelism, retries, and backtracking), and waiting time for tool calls such as builds, compilation, and tests.
Runtime has a direct impact on team collaboration. If a task takes hours or longer, the developer’s workflow shifts from real-time conversation to batch dispatch and offline waiting. Teams need to plan task queues, concurrency capacity, and failure retry strategies in advance rather than waiting in front of a screen for results.
4. Cost per Run: An Economic Metric That Must Be Part of the Decision
Cost per run is the total cost of completing one full benchmark task. It usually includes model inference, tool calls, and infrastructure overhead. The cost depends on several factors:
- Model pricing: models with different parameter sizes and service tiers can have significantly different prices.
- Context length: in long-horizon tasks, dialogue history and code context accumulate continuously, causing the input size of each round to grow.
- Retries and backtracking: repeated builds, tests, and fixes generate many additional calls.
For many teams, cost determines whether AI coding capabilities can be scaled. It directly affects the share of AI-assisted work in the total workload.
5. Reading the Four Metrics Together
For example, in one set of publicly available mobile AI coding benchmark results, the top-ranked agent achieved a pass rate of about 29% on long-horizon tasks, compared with about 24% for its previous-generation version. Another model focused on lightweight efficiency had a pass rate of about 17%, an average completion of about 71%, an average runtime of about 12 hours, and a cost of less than 10 USD per run. In contrast, the higher-pass-rate side cost more than 300 USD per run.
The following simplified example shows how to interpret the data:
- Assume task A requires 20 consecutive rounds of tool calls and reasoning to complete.
- Model A costs 0.1 USD per call and has a 25% success rate.
- Model B costs 1 USD per call and has a 60% success rate.
Roughly:
- Model A: on average, one success consumes 20 ÷ 0.25 = 80 calls, so the cost per success is about 8 USD.
- Model B: on average, one success consumes 20 ÷ 0.6 ≈ 33.3 calls, so the cost per success is about 33 USD.
The two differ by about 4x, which does not match the ratio of pass rates (60 ÷ 25 = 2.4). The reason is that failed tasks also consume call costs.
These estimates assume a constant success rate and no residual cost from failed runs. Real scenarios are more complex:
- A failed run may leave behind partially usable code changes, so later retries do not always start from scratch.
- Models with higher per-run costs often have more mature context management, so their per-round cost may not be proportional to their unit price.
- A lightweight model may perform just as well on some local subtasks and only need a stronger model to handle overall planning and validation.
A more reasonable decision is therefore to measure both task success rate and cost per success on the team’s own task set, and then combine them with time requirements. Besides cost per run, teams can also track two derived metrics:
- Task throughput: the number of tasks completed per unit time under the team’s concurrency quota, approximately (concurrency ÷ average runtime) × pass rate.
- Throughput cost: total spend per unit time, calculated as cost per run × task throughput.
The first reflects delivery speed. The second reflects budget scale as the team grows. Together, they determine whether AI coding capability can move from a tool used by a few people to a capability across the whole team.
3. The Cost–Latency Trade-off: No Universal Best Model
Putting these metrics together creates a triangle of quality, cost, and speed.
- High-pass-rate models: higher cost per run and longer runtime, but higher pass rates on long-horizon tasks, reducing repeated debugging and manual rescue work.
- Lightweight, high-speed models: lower cost and faster response, suitable for scenarios with limited budgets or where rapid feedback matters, but with lower pass rates on long and complex tasks.
The key is to return to the team’s own goals. If AI will be used for core-module refactoring or cross-platform migration—tasks where “getting it right the first time” matters—the hidden cost of rework and human review may far exceed model call fees. In that case, a high-pass-rate model may offer better overall value. If a team needs to quickly validate multiple product prototypes or apply AI to large volumes of low-risk changes, a low-cost, fast-feedback model may be easier to scale.
In practice, many teams adopt a combined strategy. For example, lightweight models can handle local tasks such as code completion, unit-test generation, and boilerplate code, while stronger models handle architecture design, complex refactoring, and cross-module migration. Many AI coding tools also provide automatic model selection based on complexity or task type. Teams can use such mechanisms as a reference and combine them with internal evaluation results to define their own task-routing rules.
4. How Long-Horizon Benchmarks Are Designed
Understanding the design of a benchmark helps teams interpret scores more accurately.
1. Task Sets: Covering Key Engineering Activities
Benchmarks for app development usually cover several task categories:
- App creation: generating a runnable app skeleton from an empty project.
- Feature development: adding pages, logic, or data flows to an existing codebase.
- Code migration: moving an implementation from one platform or framework to another technology stack.
- App conversion: transforming one type of app into another, such as adapting an app across platforms.
This classification roughly covers the main work of mobile teams and has broad coverage. However, the difficulty, number, and distribution of specific tasks still affect scores, so different benchmarks should not be compared directly.
2. Acceptance: Build, Test, and Rule Checks
To allow automatic and repeatable evaluation, acceptance criteria are usually written as deterministic checks, such as whether the build succeeds, whether all tests pass, and whether a specific state can be read correctly. This ensures objectivity, but it also means that benchmarks can only cover what can be automatically verified. Code readability, architectural soundness, and alignment with team standards are difficult to reflect fully in this kind of evaluation.
3. Agent Strategy: The Execution Framework Also Matters
The same model configuration can perform very differently under different agent frameworks. The framework determines how the problem-solving process is organized: how subtasks are split, how context is managed, and how the agent backtracks after failures. It also includes domain knowledge and tool integrations, such as platform-specific API conventions and code retrieval capabilities, which can directly affect results. Therefore, a score strictly represents “the performance of a model under a specific agent framework and task set,” not the model’s absolute ability.
5. Turning Benchmark Results into a Team Selection Method
The value of public benchmarks is to provide a starting point, not a final answer. Teams need an internal evaluation system that matches their own business.
1. Build an Internal Evaluation Set
Select a set of representative tasks from historical requirements. Cover different modules, complexity levels, and technology stacks, and define clear acceptance criteria for each task. The task set does not need to be large; a dozen or a few dozen tasks are enough to start. The key is that they come from real work.
# Example internal evaluation task: task description is separated from acceptance criteriatask_id: feature-logintitle: Add login to the sample appcontext:- Project structure and module overview- Current implementation code for relevant pages- Summary of team coding standardsacceptance:- build: success # Build must pass- test: pass # All automated tests must pass- login_gate: true # Accessing the detail page while logged out redirects to the login page- session_persist: true # Login state remains valid after the app restartsbudget:max_rounds: 60 # Maximum number of interaction roundsmax_cost: 5.0 # Cost limit for a single tasktimeout_minutes: 90 # Timeout
2. Make Evaluation Repeatable and Automated
Keep task descriptions, code repositories, execution environments, and acceptance scripts under version control. Run each evaluation in a clean environment to ensure reproducibility. In addition to pass/fail status, record runtime, context length, tool-call count, and cumulative cost for each round to create a traceable run history.
import timefrom dataclasses import dataclass@dataclassclass RunRecord:task_id: strpassed: boolcompletion: float # ratio of passed acceptance checksduration_sec: floatcost: float # cost incurred by this runrounds: int # number of interaction roundsdef run_once(agent, task, checks) -> RunRecord:"""Run one task once and record key metrics."""start = time.monotonic()result = agent.solve(task) # The agent solves the task and returns a resultresults = [check(result) for check in checks]return RunRecord(task_id=task.id,passed=all(results),completion=sum(results) / len(results),duration_sec=time.monotonic() - start,cost=result.total_cost,rounds=result.rounds,)
3. Run Multiple Times and Report Distributions
Long-horizon agents have some randomness. A single success does not prove stable capability. It is recommended to run each task at least 3 to 5 times and report the median and 90th percentile of pass rate, average completion, runtime, and cost. A task that succeeds only after repeated retries and a task that succeeds smoothly have very different implications for a team.
4. Productize the Evaluation Workflow
When evaluation becomes routine, it can be turned into platform capabilities:
- Provide isolated execution environments for each task on a container platform.
- Store code snapshots, run logs, and trace data in object storage.
- Use log services to search failure reasons centrally.
- Use monitoring and alerting to track pass-rate and cost trends.
General cloud capabilities such as those offered by Baidu AI Cloud—containers, storage, logging, and monitoring/alerting—can help teams build and operate an evaluation pipeline quickly and reliably. This is described only as a general infrastructure pattern; teams may also use self-hosted or open-source alternatives.
6. Boundaries of Benchmark Scores: Common Misreadings and Risks
- A high pass rate does not mean code can be merged directly. Passing acceptance does not guarantee code quality. Agent-generated code may contain latent defects, duplicate implementations, or violations of team standards and must be reviewed by humans.
- Benchmark results are not equal to real business performance. Public task sets differ from a team’s technology stack, code size, and collaboration processes. Internal evaluation has more direct reference value.
- A single result does not represent stable performance. Long-horizon tasks are relatively random. One failure may come from an occasional tool-call exception, and one success may include some luck.
- Context length affects both cost and effectiveness. More complex tasks accumulate longer context, which can significantly increase cost and runtime, and may even trigger context limits. Task splitting and context management are needed to control this.
- Regression risk needs a defense line. AI changes can break existing functionality. Automated regression testing and continuous integration are prerequisites for scaling AI coding capabilities.
7. Conclusion
Benchmark evaluation of AI coding agents is essentially a systematic test of whether an agent can complete real engineering tasks without step-by-step human intervention. Pass rate answers “can it deliver,” average completion reveals “where it falls short,” and runtime and cost determine “whether it can scale.” These metrics are interrelated, and judging any one in isolation can easily lead to a one-sided conclusion.
For development teams, a more practical path is to use public benchmarks as the first filter for candidate solutions, then combine the results with the team’s task distribution, quality requirements, time constraints, and budget. Build a sustainable internal evaluation process and expand it gradually through small-scale pilots. Only by putting scores back into the real engineering context can AI coding capability move from “leading on benchmarks” to “reliable delivery.”
