AI Coding Evaluation for Mobile App Development: Methods and Practices
This article describes a methodology for evaluating AI coding capabilities on real mobile app engineering tasks. It covers task categories, core metrics, environment consistency, a reference evaluation flow, result interpretation, constraint-based model selection, common pitfalls, and suggestions for building an evaluation system.
1. Why Evaluate Programming Models with Real Engineering Tasks
Code-generation models often look impressive in demos, but once they enter a team, problems tend to appear in engineering details. A function that runs does not mean an app can build, pass tests, and be delivered to users. This is especially true for mobile app development, which involves UI layout, lifecycle management, permission requests, local storage, network requests, dependency versions, and compatibility differences across OS versions.
Therefore, evaluation should start from verifiable engineering tasks rather than isolated algorithm problems. An effective evaluation needs to answer three questions: Can the model complete the task independently? What is the quality of the result? Is the cost acceptable?
2. Break Evaluation Targets into Measurable Task Types
Common requirements in mobile app development can be roughly divided into four categories.
- App creation: Build a project skeleton based on a requirements description, including directory structure, build configuration, basic pages, and navigation.
- Feature development: Add features to an existing project, such as list pagination, form validation, or offline caching.
- Code migration: Migrate old APIs or syntax to a new version while requiring existing behavior to remain unchanged.
- App conversion: Rewrite an app from one technology stack to another, for example, migrating native UI to a cross-platform framework.
Each task type should have its own requirements description, initial code, acceptance tests, and scoring rules. The requirements description must specify inputs, outputs, and boundary conditions, and must not rely on verbal additions from the evaluator; otherwise, different people will get different results when executing the evaluation.
3. Core Metric System
A single metric can hardly reflect a model’s overall capability. At least the following four metrics should be collected.
3.1 Pass Rate
Pass rate measures whether a task fully meets the requirements. It is calculated as:
Pass rate = passed tasks / total tasks
A task should be considered passed only when all acceptance tests pass and the build has no errors. If any item fails, the task is recorded as failed.
3.2 Average Completion
Completion measures the degree of partial completion. The approach is to break acceptance items into finer items, calculate the proportion of passed acceptance items for each task, and then average the results:
Average completion = Σ(passed acceptance items per task / total acceptance items per task) / total tasks
Completion helps distinguish between “almost successful” and “little progress,” and is useful for diagnosing weak areas in a model.
3.3 Average Elapsed Time
Elapsed time refers to the wall-clock time from submitting the requirement to producing the final project. It includes queuing, retries, and tool-call waiting, not just model inference time. Reports should also state concurrency and network environment; otherwise, the numbers are not comparable.
3.4 Cost per Run
Cost includes model call fees, sandbox resource fees, and the amortized cost of manual review. If only call fees are counted, the result will be underestimated. Costs should use the same currency and the same exchange-rate date, and this should be noted in the report.
The four metrics often involve trade-offs. A solution with a high pass rate may not be the most time-efficient, and a solution with short elapsed time may have lower completion. When selecting a model, teams should first prioritize based on team constraints rather than looking at only one column of numbers.
4. Ensure Consistency of the Evaluation Environment
Whether evaluation results are credible depends heavily on whether the environment is consistent.
- Fix the base image and dependency versions to avoid different results for the same task at different times due to dependency updates.
- Use a fresh sandbox for each run, and prohibit reuse of files and caches left by previous runs.
- Standardize the agent’s tool permissions, such as whether network access is allowed and whether build commands can be executed.
- Set a timeout limit; record timeouts as failures and log the reason.
- If the API supports fixed random seeds or parameters that reduce randomness, set them consistently. If not, reduce fluctuation by averaging multiple runs.
5. Evaluation Flow Example
Below is a simplified evaluation flow to illustrate the relationship between stages.
Input: task set T, model list M, runs per task k, timeout limit LOutput: metric records for each modelfor m in M:for t in T:for i in 1..k:env = create_isolated_sandbox(t.base_image, t.dependency_version)start = current_time()result = run_agent(m, t.requirement_description, env, timeout=L)cost = read_call_billing(result)elapsed = current_time() - startchecks = run_acceptance_tests(env.project_dir, t.acceptance_cases)passed = build_success and all(checks)ratio = passed_check_count / len(checks)record(m, t, i, passed, ratio, elapsed, cost)destroy(env)Summarize:pass_rate = passed_runs / total_runsavg_completion = average(ratio)avg_elapsed = average(elapsed)cost_per_run = total_cost / total_runs
The example code has three key points: every run is isolated, every result is recorded, and raw records are kept before aggregation. Raw records make later review easier and allow metrics to be recalculated if disputes arise.
6. Interpret Evaluation Results Correctly
After receiving a set of numbers, first determine whether they are reliable.
- Sample size: When the number of tasks is too small, pass rates fluctuate greatly. In a set with only three to five tasks, a few result changes can alter the ranking.
- Number of runs: The same model may produce different results across multiple runs on the same task. Report the average and provide the range of variation.
- Task distribution: If tasks are concentrated in one category, the results represent only performance on that category and should not be extrapolated to all development scenarios.
- Agent configuration: The same model paired with different agent frameworks may perform very differently. Comparisons should ensure consistent configuration or list configurations separately.
- Time point: Model versions and billing rules change, so reports should note the evaluation date.
When comparing results, also distinguish between “capability differences” and “engineering condition differences.” If two sets of results use different runtime environments, timeout settings, or tool permissions, the conclusions cannot be directly applied.
7. Select Models Based on Constraints
Different teams assign different weights to metrics. The following approaches can be used as references.
- Quality first: When deliverables go directly to users, prioritize pass rate and completion, and accept longer elapsed time and higher cost. These scenarios usually still require manual review of critical modules.
- Budget-sensitive: When task volume is large and frequent iteration is needed, cost is the main constraint. Calculate cost per successful task, that is, total cost divided by the number of passed tasks, and compare it with other solutions.
- Response-time-sensitive: If the development workflow requires fast feedback, elapsed time should be a threshold condition. For example, set an upper limit on average elapsed time per task and directly exclude solutions that exceed it.
Another approach is layered use. Assign simple, repetitive tasks to solutions with shorter elapsed time and lower cost, assign complex tasks to solutions with higher pass rates, and retain a manual review step.
8. Common Pitfalls
- Treating completion as pass rate. High completion does not mean the deliverable is usable; if critical acceptance items fail, rework is still required.
- Ignoring manual review cost. If generated code requires extensive modification, the apparent low cost may be offset by subsequent work hours.
- Looking only at averages. Two solutions may have similar average elapsed time but very different maximum elapsed time, and therefore different timeout risks.
- Replacing real tasks with demo samples. Demo samples are usually selected and cannot reflect the complexity of daily requirements.
9. Implementation Suggestions
Teams can establish their own evaluation system in the following order.
