Evaluating Mobile AI Coding Agents: Metrics, Trade-Offs, and Engineering Practices
A practical guide to evaluating AI coding agents on realistic mobile development tasks. It explains pass rate, completion rate, time, and per-run cost; examines the trade-offs among these metrics; and outlines how teams can build internal benchmarks, standardize test conditions, analyze results by task difficulty, and avoid common evaluation mistakes.
1. Why Evaluate AI Coding Agents on Real Engineering Tasks?
Over the past two years, AI coding agents have become capable of handling parts of software development independently, including scaffolding projects, adding features, and migrating legacy code. However, there is still a significant gap between “can write code” and “can deliver usable code.”
Unit-test-style code generation benchmarks can show whether a model can implement a correct function, but they do not reveal how well it performs on tasks involving multiple files, long workflows, repeated debugging, and complex validation.
Benchmarks for mobile application development typically model complete engineering scenarios, including:
- Application creation: Generate a compilable and runnable project structure from a requirements description.
- Feature development: Add screens, API calls, or interaction logic to an existing codebase while maintaining compatibility with existing modules.
- Code migration: Update legacy syntax, dependencies, or architecture to comply with newer conventions.
- Application conversion: Map an implementation from one technology stack or UI framework to another.
These tasks share several characteristics: many steps, multiple dependencies, and lengthy validation cycles. A single task may involve dozens of file operations, builds, and test runs. An error at any stage can make the final result unusable.
As a result, an evaluation should answer three questions at the same time:
- Was the task actually completed?
- How much of the task was completed?
- How much time and money did the agent require?
2. Understanding the Four Core Metrics
A useful comparison typically reports all four of the following metrics. Looking at only one of them can easily lead to misleading conclusions.
1. Pass Rate
Pass rate is the proportion of tasks whose final output passes all acceptance checks. These checks may include successful compilation, passing unit tests, passing functional test cases, and the absence of serious issues identified by static analysis.
Pass rate measures the ability to deliver a fully acceptable result, making it the strictest metric.
2. Average Completion Rate
Average completion rate measures how much of each task the agent completed. For example, if a task has ten acceptance requirements and the agent satisfies eight, its completion rate for that task is 80%.
The difference between completion rate and pass rate can be particularly informative. If average completion is high but pass rate is low, the agent usually gets close to the target but fails during final validation or on edge cases. This often points to weaknesses in debugging or self-verification.
3. Average Duration
Average duration is the time from task assignment to final output.
However, duration may represent either productive reasoning or repeated trial and error. When agents are allowed to run for extended periods, their duration is closely related to their execution strategy. For example, an agent may frequently reload context, run a build after every change, or roll back and retry after failures.
4. Cost per Run
Cost per run includes model usage fees and the computing resources consumed by the execution environment. It is usually correlated with duration: longer runs often require more context and more reasoning steps.
For teams running tasks at scale, per-run cost directly affects whether broad adoption is practical.
3. Engineering Trade-Offs Between Metrics
Evaluation results rarely show one solution leading in every metric. More often, the metrics pull in different directions. Understanding these trade-offs is more valuable than memorizing a single ranking.
1. Completion Rate vs. Duration
Higher completion rates often require more exploration and validation, which increases execution time. An agent that runs the full test suite and checks edge cases before submitting will generally be slower than one that delivers a quick but lightly validated result.
This can be a reasonable trade-off. If a task is expensive to repair later, spending more time upfront to improve completion is worthwhile. Conversely, excessive validation may waste resources for a rapid prototype or a one-off script.
2. Pass Rate vs. Cost
The relationship between pass rate and cost depends on task difficulty. For simple tasks, a low-cost approach may already be sufficient. For complex tasks, additional reasoning rounds may be necessary before the agent converges on a correct solution.
When comparing two approaches, teams should analyze results by task difficulty rather than relying only on overall averages.
3. Speed vs. Reliability
In interactive development, response speed affects a developer’s attention and flow. In unattended batch execution, reliability is usually more important than speed.
The optimal configuration for the same agent may therefore differ completely between these two scenarios. Teams should first identify their primary usage pattern and then decide how to balance speed and reliability.
4. The Scope and Limitations of Benchmark Results
A benchmark score reflects performance on a specific task set, with a specific runtime configuration and acceptance policy. Results cannot be compared directly if agents have different tool permissions, context limits, or retry strategies.
When reviewing any evaluation, first verify:
- Where the tasks came from
- How acceptance was determined
- What runtime conditions were used
4. How to Build an Internal Evaluation
Public benchmarks provide useful reference points, but teams ultimately need to determine which approach works best on their own codebases and requirements. The following process provides a practical starting point.
Step 1: Build a Task Set
Extract representative tasks from historical requirements, covering the scenarios described above. Each task should include:
- Clear input instructions
- A defined initial code state
- Automatically executable acceptance scripts
Ideally, each scenario should contain at least ten tasks. Label tasks by difficulty so that results can be analyzed by category.
Step 2: Define Acceptance Criteria
Acceptance criteria should be automated whenever possible. Compilation, unit tests, automated tests for critical UI paths, and static checks can all be included in the evaluation.
For aspects that cannot be judged automatically, such as code readability, teams can conduct manual sampling and record the sampling ratio.
Step 3: Standardize Runtime Conditions
When comparing different agents, keep the following variables consistent:
- Task inputs and initial code must be identical.
- Available tools and permissions must be identical.
- Timeouts and maximum retry counts must be identical.
- Runtime environment versions must be identical.
Step 4: Collect and Calculate Metrics
The following pseudocode illustrates how the metrics can be calculated. The implementation should be adapted to the team’s testing framework:
results = run_benchmark(tasks, agent_config)pass_rate = count(r.passed for r in results) / len(results)avg_completion = mean(r.completion_ratio for r in results)avg_duration_h = mean(r.duration_seconds for r in results) / 3600cost_per_run = sum(r.cost for r in results) / len(results)# Composite score: weights should reflect the team's prioritiesscore = w1 * pass_rate + w2 * avg_completion \- w3 * avg_duration_h - w4 * cost_per_run
There is no universally correct weighting scheme. A team under delivery pressure may prioritize pass rate, while a budget-sensitive team may place more weight on cost.
The important point is to define the weights before running the evaluation, rather than adjusting them after seeing the results.
Step 5: Analyze Results by Difficulty
Group tasks by difficulty and compare performance across groups. If an agent leads only on simple tasks, its advantage may not carry over to complex projects.
Stratified analysis can also help teams determine which tasks are suitable for fully automated execution and which still require human leadership.
5. Common Evaluation Mistakes
Mistake 1: Looking only at pass-rate rankings.
A high pass-rate solution may take significantly longer and cost much more. For teams iterating frequently, it may not be the best overall choice.
Mistake 2: Ignoring average completion rate.
A low completion rate indicates that the agent deviated from the target early in the task. This may not be obvious from pass rate alone, but it can substantially increase the repair effort required from developers.
Mistake 3: Treating per-run cost as the total cost.
Per-run cost does not include rerunning failed tasks, manual review, or subsequent repair work. Total cost should also account for rework and review time.
Mistake 4: Making long-term decisions from a single evaluation.
Models and toolchains change frequently, so evaluation results have a limited shelf life. Teams should periodically rerun the same task set and track performance trends over time.
6. Practical Recommendations
For most teams, a pragmatic adoption process can be divided into three stages:
- Use public benchmarks and a small internal task set to narrow the options to two or three candidates.
- Run a small pilot on real requirements for several weeks, recording pass rate, rework frequency, and developers’ subjective feedback.
- Use the pilot data to identify the primary use cases and establish mandatory human review points, such as code review before merging.
The value of an AI coding agent lies in reducing repetitive work, not replacing engineering judgment. The purpose of evaluation is to find the right combination for a specific team and workflow—not to identify one solution that works best in every situation.
