Evaluating AI Coding Agents for Mobile App Development: Metrics and Engineering Trade-offs
This article explores a task-level evaluation framework for AI coding agents in mobile application development. It defines four core metrics—pass rate, average completion, average duration, and cost per run—and analyzes the engineering trade-offs between them. Additionally, it provides a practical guide for building a reproducible evaluation pipeline and outlines a decision framework for teams to select the right agent for different development scenarios.
I. The Need for Task-Level Evaluation
In the past, evaluating code generation capabilities typically relied on the accuracy of function-level completions or scores on algorithmic problems. While these metrics indicate a model’s grasp of syntax and local logic, they fail to reflect whether the model can independently complete a full engineering task.
Mobile app development has its own unique characteristics. A single requirement often involves project structure scaffolding, UI layout, state management, network requests, local storage, permission handling, dependency version management, and multi-device adaptation. Task boundaries span multiple files, and acceptance depends on build results and runtime behavior. Therefore, evaluation should focus on “whether the task is truly completed” rather than merely checking if the generated code snippets look reasonable.
Task-level evaluation typically includes the following types of work:
- App Creation: Generating a buildable project skeleton based on requirements, including configuration files, module division, and basic pages.
- Feature Development: Adding new pages or business logic to an existing project while ensuring compatibility with the original code.
- Code Migration: Migrating old API calls, outdated view components, or deprecated dependencies to newer versions.
- App Conversion: Rewriting an implementation from one tech stack to another, such as moving from one declarative UI framework to another.
The commonality among these tasks is that the results must be verifiable automatically. Verification methods include compilation, unit testing, UI testing, and static analysis. Without executable acceptance criteria, evaluation conclusions are difficult to reproduce.
II. Defining the Four Core Metrics
A practical evaluation report typically includes four dimensions. Each answers a different question and cannot substitute for another.
2.1 Pass Rate
Pass rate is a binary metric: whether a task fully meets the acceptance criteria. The calculation is as follows:
Pass Rate = Passed Tasks / Total Tasks
Pass rate is the most intuitive but also the most unforgiving. If even one acceptance condition is unmet, the task is counted as a failure. For example, if all features are correct but a UI layout overflows on a small-screen device, it still fails. Thus, pass rate is suitable for measuring the ability to “deliver directly.”
2.2 Average Completion
Average completion is a partial scoring metric. It breaks down a task into several sub-goals, such as “project builds successfully,” “page renders,” “data saves,” and “exception branches handled.” Each sub-goal carries a weight, and the final score is averaged.
Average completion reveals whether the model is close to the goal in most cases, even if the final step fails. If completion is high but the pass rate is low, it usually means the model can do most of the work but struggles with finishing touches—such as missing boundary conditions, failing to update configurations, or ignoring compatibility issues. These results are valuable for manual review, as repair costs are often concentrated in those final steps.
2.3 Average Duration
Duration refers to the time elapsed from task start to the agent’s final output, including model inference, tool calls, build, and test waiting. Average duration is affected by the following factors:
- Whether the agent uses multi-round self-correction;
- The response speed of the build and test environment;
- Task context length and retrieval frequency;
- The presence of repeated attempts or failure rollbacks.
Duration directly impacts the pace of the development workflow. For interactive development, waiting tens of minutes disrupts the working state; for overnight batch migrations, duration mainly affects resource usage and completion time.
2.4 Cost Per Run
Cost per run includes model API fees, tool execution fees, and computing resource overhead. It should be calculated per task or per full evaluation run, with pricing standards and statistical timeframes clearly stated to ensure comparability across reports.
Cost also has an easily overlooked hidden component: the retry costs of failed tasks and the human hours consumed in manual fixes. If you only look at the bill, a low-priced solution might actually incur a high total cost due to frequent retries.
III. Engineering Trade-offs Between Metrics
Each of the four metrics has limitations on its own; only by examining them together can a sound judgment be formed. The following combinations deserve special attention.
3.1 High Completion vs. Long Duration
When average completion is high but average duration is long, it indicates the agent is willing to invest more steps to approach the goal. This behavior is often valuable in complex migration tasks, as it reduces the manual effort of starting from scratch. However, long durations also mean higher waiting costs and harder-to-debug intermediate states.
These solutions are better suited for:
- Unattended batch tasks, such as overnight code migrations;
- Task queues that can be processed in parallel;
- Work that demands high correctness but low response speed.
Unsuitable scenarios include interactive development requiring instant feedback, and pipelines with strict time constraints per task.
3.2 Low Cost vs. Fast Response
When both cost and duration are low, but the pass rate is also low, the solution should be positioned as a “fast draft.” It is suitable for generating boilerplate code, adding test cases, or handling simple structural changes. Handing its output over for human review is safer than expecting it to deliver independently.
To determine if such a solution is cost-effective, two metrics must be compared:
Cost Per Success = Total Cost / Passed Tasks
Effective Throughput = Passed Tasks / Total Duration
If the cost per success is very low but effective throughput is also low, it means the solution is cheap but slow to produce. If effective throughput is high, it means it can deliver more usable results per unit of time, even if the unit price isn’t the lowest.
3.3 The Gap Between Pass Rate and Completion
The gap between pass rate and average completion reflects the “last mile” problem. When the gap is large, you should typically check the following:
- Are the acceptance criteria too strict, such as pixel-level visual comparisons?
- Are there implicit requirements missing from the task description?
- Is the agent lacking runtime environment information to verify its own results?
- Do failures concentrate on a specific type of task, such as dependency conflicts or platform version differences?
After identifying these causes, the team can determine whether the issue lies in model capability, task design, or the evaluation environment.
IV. Building a Reproducible Evaluation Pipeline
Evaluation conclusions are only valuable if they can be repeatedly verified. Below is a standard workflow.
4.1 Designing the Task Set
The task set should cover the four types of work mentioned earlier and be stratified by difficulty. Each task should include a clear requirement description, input materials, acceptance criteria, and expected outputs. Here is a task definition example:
{"task_id": "feature-042","category": "Feature Development","platform": "Mobile App","description": "Add pull-to-refresh to the existing list page, and display a retry button on network errors","inputs": ["Project source code snapshot", "API documentation"],"acceptance": ["Project builds successfully","All unit tests pass","Pull-to-refresh triggers one data request","A retry button is displayed on network error and re-requests on click"],"difficulty": "medium"}
Task descriptions should avoid ambiguity. If a requirement has multiple reasonable implementations, the acceptance criteria should only constrain behavior, not specific coding styles.
4.2 Fixing the Runtime Environment
Environmental changes directly impact results. The following should be fixed during evaluation:
- Versions of the development toolchain and build tools;
- Locked versions of dependency libraries;
- OS versions and resolutions of emulators or test devices;
- Mock service data used for network requests;
- The list of tools the agent can access and their permission scopes.
Environment configurations are best saved as code and archived alongside the evaluation report.
4.3 Automated Acceptance
Acceptance should be automated as much as possible to reduce subjective judgment. Common acceptance layers include:
- Compilation and Build: Check if the project can be built successfully; this is the basic threshold.
- Static Analysis: Run linters and type checkers to catch obvious errors.
- Unit Testing: Verify the correctness of business logic.
- UI Testing: Simulate user interactions to check page states and interaction results.
- Manual Spot Checks: Conduct sampling reviews for visual and experiential issues that are hard to cover automatically.
4.4 Multiple Runs and Statistics
A single run’s results are heavily influenced by randomness. Each task should be run multiple times, reporting the mean and variance. With a small sample size, differences in pass rates might just be coincidental.
The following code demonstrates how to calculate core metrics from run records:
from dataclasses import dataclass@dataclassclass RunRecord:task_id: strpassed: boolcompletion: float # Completion rate, ranging from 0.0 to 1.0duration_h: float # Duration in hourscost_usd: float # Cost per run in USDdef summarize(records: list[RunRecord]) -> dict:if not records:raise ValueError("Run records are empty")n = len(records)passed = [r for r in records if r.passed]total_time = sum(r.duration_h for r in records)total_cost = sum(r.cost_usd for r in records)return {"pass_rate": len(passed) / n,"avg_completion": sum(r.completion for r in records) / n,"avg_duration_h": total_time / n,"cost_per_pass": total_cost / len(passed) if passed else float("inf"),"passes_per_hour": len(passed) / total_time if total_time else 0.0,}
The output results should be saved along with environment information and task set versions for future comparison.
V. A Decision Framework for Team Adoption
A leaderboard only reflects a model’s performance on a specific task set; it cannot directly replace a team’s own judgment. Adoption can proceed through the following steps.
5.1 Stratify by Task Risk
Different tasks carry different risks. Changes to core business modules require high correctness, making them suitable for solutions with high pass rates and strict manual review. Generating boilerplate code and test cases has looser correctness requirements and can tolerate fast solutions with lower pass rates. Large-scale migrations prioritize effective throughput and overall cost.
5.2 Define Budget and Latency Constraints
Before selecting a solution, the team should establish two hard constraints: the maximum acceptable duration per task, and the maximum budget for the month or project cycle. Filter candidate solutions that meet these constraints, then sort by quality—this aligns better with real-world needs than comparing metrics directly.
5.3 Include Manual Costs in Total Cost
Manual review and repair time is often the largest hidden cost. A solution with a slightly lower pass rate but stable output quality and easy-to-locate issues might save more money overall. This can be estimated as follows:
Total Cost ≈ Model & Tool Fees + Manual Review Hours × Hourly Rate + Retry Costs for Failed Tasks
5.4 Pilot Before Full Rollout
Before formal adoption, it is recommended to conduct a small-scale pilot using a batch of historical tasks from real projects, covering common requirement types. Pilot results are much closer to reality than public leaderboards.
VI. Common Pitfalls
During evaluation, the following practices can easily lead to wrong conclusions:
- Treating completion as pass rate. High completion only means most steps are correct; it doesn’t mean the result is ready for direct delivery.
- Ignoring failure costs. Only calculating the time and cost of successful tasks overestimates the solution’s efficiency.
- Small sample sizes. Results from a few tasks rarely reflect long-term performance, and differences might stem from random fluctuation.
- Task overlap with training data. If evaluation tasks are widely present in public materials, results might be artificially high. Use the team’s own or newly constructed tasks whenever possible.
- Ignoring version changes. Historical results become invalid after toolchain or OS version updates, requiring re-evaluation.
VII. Implementation Recommendations
- Establish a fixed task set for the team and regularly add new cases from real requirements.
- Save environment configurations, task versions, and raw run logs for every evaluation to ensure traceability.
- Report pass rate, completion, duration, and cost simultaneously; avoid ranking based on a single metric.
- Retain a manual review step for high-risk tasks and record the types of issues found to improve task descriptions and acceptance criteria.
- Establish different combinations of solutions based on task types, rather than relying on a single solution for all work long-term.
