Evaluating AI Coding Assistants for Mobile Development: Four Key Metrics to Watch

    This article explains why mobile app development is a critical testing ground for AI coding agents, outlines the typical structure of benchmark tasks, and breaks down four core evaluation metrics: pass rate, average completion, average time, and cost per run. It then discusses the engineering trade-offs between these metrics, offers a scenario-based selection method with a weighted scoring example, and provides practical advice for building internal evaluation sets and controlling cost and time. The article also highlights the limitations of public benchmarks and emphasizes the need for team-specific validation.

    1. Why evaluate mobile development capabilities separately?

    Mobile app development is a strong test case for AI coding agents. A typical project involves multiple layers simultaneously — UI layout, state management, network requests, local storage, and device permissions — forming a long task chain. A single requirement, from creating a project skeleton to pre-release migration changes, can span multiple files and multiple rounds of edits. General-purpose code completion tests cannot cover such scenarios, which is why industry benchmarks are increasingly designed around real-world engineering tasks.

    The focus of these benchmarks is not whether a model can write a syntactically correct function, but whether it can independently complete an application that compiles, runs, and satisfies the specified requirements within a given environment.

    2. Common structure of benchmark tasks

    A typical mobile development benchmark consists of four task categories:

    • App creation: Generate a project skeleton from a requirement description, including directory structure, dependency declarations, and basic pages.
    • Feature development: Add new modules to an existing project, such as a login flow or a paginated list.
    • Code migration: Rewrite legacy API calls or old syntax to newer versions.
    • App conversion: Rewrite an application implemented in one technology stack to another while preserving original behavior.

    Each task is usually accompanied by an acceptance script. The script runs the build, executes unit tests, or checks key UI behaviors. A task is counted as passed only if all acceptance checks pass.

    3. The four core metrics explained

    3.1 Pass rate

    Pass rate is the ratio of passed tasks to total tasks. It is the strictest metric — if any acceptance check fails, the task is not counted. Pass rate reflects whether the agent can deliver a usable result.

    3.2 Average completion

    Completion measures the proportion of acceptance checks satisfied within a task. A task may have 90% of its functionality implemented, with only one edge case failing. Its completion score is high, but the pass rate remains zero. Looking at both metrics together helps distinguish “almost right” from “fully right.”

    3.3 Average time

    Time is the duration from when the agent receives a task to when it outputs the final result. In long-horizon tasks, the agent may perform multiple rounds of planning, file reading, command execution, and error correction. Therefore, time is influenced by task decomposition strategy, number of retries, and tool-call speed. Time affects both waiting cost and whether a developer can iterate within a single session.

    3.4 Cost per run

    Cost is typically determined by the number of input/output tokens across model calls, the number of tool invocations, and the number of agent loop iterations. In the same task, if the agent repeatedly reads large amounts of context, the cost rises quickly. To measure cost consistently, you need a unified pricing model and must clarify whether failed attempts with retries are included.

    4. Engineering trade-offs between metrics

    The four metrics do not move in the same direction. High completion does not necessarily lead to high pass rate, and longer time does not always correspond to better quality. The following example data illustrates how to interpret them — these numbers are for demonstration only and are not actual benchmark results.

    Option Tasks Passed Pass rate Total cost (CNY) Cost per passed task (CNY) Average time (hours)
    A 30 6 20.0% 300 50 24
    B 30 4 13.3% 40 10 10

    Option A has a higher pass rate, but the cost per deliverable task is five times that of Option B, and the time is more than double. If the team’s tasks are mostly drafts that can be revised multiple times, Option B’s fast response may be more valuable. If tasks require a ready-to-release outcome in one shot, Option A’s higher pass rate might be more cost-effective.

    Such trade-offs usually come from three causes:

    1. To improve correctness, agents often add verification steps, which raises time and cost.
    2. In long-horizon tasks, small early errors amplify in later steps, requiring additional backtracking and correction.
    3. Different models have different tool-call planning capabilities; the number of steps for the same task can vary by several times.

    5. Selecting a method by scenario

    Before selection, clarify the team’s primary scenario. Tasks can be categorized into three types: exploratory, delivery-oriented, and batch migration. Each type assigns different weights to the metrics.

    Below is a simple weighted scoring example for ranking candidate options:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class EvalResult:
    4. name: str
    5. pass_rate: float # value between 0 and 1
    6. completion: float # value between 0 and 1
    7. avg_hours: float # average time in hours
    8. cost_per_pass: float # average cost per passed task in CNY
    9. def score(r: EvalResult, weights: dict[str, float]) -> float:
    10. # Time and cost are lower-is-better; normalize to 0–1 benefit scores
    11. time_score = 1 / (1 + r.avg_hours / 24)
    12. cost_score = 1 / (1 + r.cost_per_pass / 50)
    13. return (
    14. weights["pass_rate"] * r.pass_rate
    15. + weights["completion"] * r.completion
    16. + weights["time"] * time_score
    17. + weights["cost"] * cost_score
    18. )

    The weights should be set by the team according to business needs. Delivery-oriented scenarios may set a high weight for pass_rate, while exploratory scenarios should increase the weights for time and cost. The normalization reference values (e.g., 24 hours, 50 CNY) should also be adjusted based on the team’s actual budget.

    Scoring results only serve as an initial filter. Before making a final decision, run the candidates on your own real tasks sampled from internal projects, because the task distribution in public benchmarks may not align with your in-house workload.

    6. Practical implementation advice

    6.1 Build an internal evaluation set

    Sample 20–50 typical tasks from the requirements or bug tickets of the past six months, covering UI, network, storage, and permission modules. Write acceptance scripts for each task to ensure results can be evaluated automatically.

    6.2 Control time and cost

    Set a maximum wall-clock time and a maximum number of loop iterations for the agent. Kill tasks that exceed the threshold and keep intermediate artifacts. Also set budget alerts and track daily invocation costs per project or team. Here is an example configuration:

    1. agent_run:
    2. max_wall_clock_hours: 12
    3. max_tool_calls: 200
    4. max_retries_per_step: 2
    5. budget_alert_per_task_cny: 80

    6.3 Keep human review

    Generated code should go through code review and automated tests before merging into the main branch. For modules involving payments, permissions, and user privacy, require line-by-line human inspection.

    6.4 Introduce in phases

    Start with low-risk modules such as internal tool pages or admin dashboards. After collecting pass-rate and rework-rate data over time, gradually expand the scope of usage.

    7. Limitations of public benchmarks

    Public benchmarks have three notable limitations. First, the task distribution is chosen by benchmark designers and may differ from the real requirements of a specific project. Second, models and agent frameworks evolve quickly; results from one version may lose relevance within weeks. Third, time and cost depend on pricing, network conditions, and concurrency load, so direct comparisons require consistent measurement criteria.

    Therefore, public benchmarks are useful for narrowing down a set of candidate models, but the final selection should always be validated against your own representative tasks.