AI Coding Evaluation for Mobile Development: Metrics, Task Design, and Selection Framework
This article explains how to evaluate AI coding assistants for mobile development using real engineering tasks rather than isolated code snippets or multiple-choice questions. It covers task categories and boundaries, four core metrics, data aggregation, common misinterpretations, and a practical framework teams can use to compare candidates and make selection decisions.
1. Why Use Real Engineering Tasks to Evaluate AI Coding Capability
The capabilities of AI coding assistants are often demonstrated through code snippets or multiple-choice questions, but these formats rarely reflect the real demands of developers’ daily work. A mobile app project requires the model to understand context, follow engineering conventions, and ensure that results compile and run—from creating a new project and building the UI to integrating network requests, migrating legacy code, and adapting to new OS versions.
Evaluation based on real engineering tasks treats each task as an acceptance-testable deliverable. Evaluators typically provide task instructions, an initial code repository, and acceptance tests. An agent completes changes in an isolated environment, and automated tests plus manual spot checks determine the result. This approach tests code correctness, context understanding, and execution stability at the same time, making it closer to actual work than simply comparing generated samples.
2. Task Types and Boundary Definition
Evaluation tasks for mobile app development can be broadly divided into four categories:
- App creation: Starting from a blank template, build a project structure that meets conventions, complete dependency configuration, and implement basic pages.
- Feature development: Add features such as lists, forms, and login flows to an existing project, testing the ability to read and reuse existing code.
- Code migration: Rewrite legacy API calls, state management, or data storage approaches using a new solution while preserving existing behavior.
- App adaptation: Adapt to new OS versions, screen sizes, or permission models, involving configuration files, page lifecycle, and compatibility handling.
Clear boundaries are important. If a task description is vague, different agents may implement different requirements, and scores lose comparability. Each task should therefore specify inputs, outputs, out-of-scope changes, and acceptance criteria.
3. Core Metrics
3.1 Pass Rate
Pass rate is the proportion of tasks that pass all acceptance tests. It is the most intuitive metric, but it is insensitive to failure causes. For example, a task that misses only one boundary condition and a task with no implementation at all are both recorded as failed in the pass rate.
3.2 Average Completion
Completion compensates for the coarse granularity of pass rate. Evaluators can score each acceptance test individually, calculate the pass ratio for each task, and then average across all tasks. High completion with a low pass rate usually indicates that the model completed most of the work but has gaps in finishing or detail handling.
3.3 Average Duration
Duration includes the total time for model inference, tool calls, compilation, and testing. Long-horizon tasks take longer because the agent repeatedly reads files, modifies code, and runs tests. Evaluations should record the duration distribution, not just the average. Median and long-tail tasks affect teams differently, and looking only at the mean can easily hide problems.
3.4 Cost per Run
Cost consists of model invocation fees, tool fees, and compute resources. The measurement should be standardized as “cost per single run per task,” with the currency and exchange-rate date noted to avoid errors when comparing across time.
3.5 Trade-offs Among Metrics
The four metrics often constrain one another. A higher pass rate may require longer reasoning time and higher cost. Teams should first set priorities: when delivery timelines are tight, duration should carry more weight; when code will go directly into production, pass rate and completion should take priority.
4. Engineering Evaluation Data
The following example code shows how to aggregate evaluation records. It only illustrates how metrics are calculated; the data structure should be defined by the evaluator.
from dataclasses import dataclassfrom statistics import mean, median@dataclassclass RunRecord:task_id: strpassed: bool # whether all acceptance tests passedcases_passed: int # number of acceptance tests passedcases_total: int # total number of acceptance testsduration_hours: float # end-to-end duration in hourscost_usd: float # cost per run in USDdef summarize(records: list[RunRecord]) -> dict:n = len(records)pass_rate = sum(r.passed for r in records) / ncompletion = mean(r.cases_passed / r.cases_total for r in records)durations = [r.duration_hours for r in records]return {"pass_rate": round(pass_rate * 100, 1),"avg_completion": round(completion * 100, 1),"avg_duration_h": round(mean(durations), 1),"median_duration_h": round(median(durations), 1),"avg_cost_usd": round(mean(r.cost_usd for r in records), 2),}
Three points are worth noting. First, pass rate and completion are calculated separately so that neither substitutes for the other. Second, duration reports both the average and the median, making it easier to detect when a few very long tasks pull up the mean. Third, costs are converted to the same currency, and the exchange-rate source and date should be recorded.
5. Common Pitfalls When Interpreting Results
- Looking only at a single ranking. Different task sets have different difficulty distributions, and rankings may change with task selection.
- Ignoring sample size. When the number of tasks is small, individual task results can cause large fluctuations in pass rate. Reports should include the number of tasks and explain the uncertainty range.
- Treating cost as an absolute comparison. Cost per run is affected by billing policies, cache hits, and retry counts. Comparisons should be based on controlled experiments under the same conditions.
- Ignoring the runtime environment. Toolsets, prompt templates, and timeout settings can significantly affect results and need to be fully recorded.
- Equating benchmark results directly with production outcomes. Benchmark tasks are abstractions of real work, and teams still need to run small-scale pilots on their own codebases.
6. Decision Framework for Team Selection
Selection can proceed through the following steps:
- Define scenarios: List the three to five most common types of development tasks for the team, such as page development, API integration, and version upgrades.
- Build an internal task set: Sample representative tasks from historical requirements and write acceptance tests for each task.
- Set weights: Assign weights to the four metrics based on delivery pressure, budget, and quality requirements.
- Run a small-scale pilot: Select two or three candidate solutions, run them under the same conditions, and record actual data.
- Review and iterate: Analyze failure causes, distinguish between insufficient model capability, unclear task descriptions, and environment configuration issues, and then decide whether to adjust the approach.
7. Practical Recommendations
- Put acceptance tests under version control to ensure that each evaluation can be reproduced.
- Run agent-generated code through the same review process as human-written code, including static checks, unit tests, and security scans.
- Set budget limits for high-cost tasks and terminate automatically after timeouts to avoid wasted resources.
- Rerun evaluations periodically because model versions, billing rules, and tool capabilities change over time.
8. Conclusion
The key to evaluating AI coding capability is to transform “can it write code” into “can it deliver engineering results that pass acceptance validation.” Pass rate, completion, duration, and cost each reflect different aspects. Only by interpreting them with team-specific weights can teams obtain a reliable basis for selection.
