Evaluating AI Coding for Mobile Apps: Pass Rate, Completion, and Cost Trade-offs
This article explains why AI coding evaluations for mobile apps should focus on engineering task completion rather than isolated code generation. It examines pass rate versus completion, runtime and cost accounting, system-level evaluation factors, and practical methods for building credible mobile development evaluation sets and workflows.
1. From Code Generation to Engineering Task Completion
Evaluations of AI coding capability are shifting from whether a system can write a correct piece of code to whether it can complete an acceptable change in a real project. The two goals may look similar, but they test noticeably different capabilities.
For isolated code generation, requirements are usually clear, the input scope is limited, and the output can often be checked with unit tests. In mobile app development, however, a single feature may involve UI, business state, data persistence, network interaction, permission handling, and build configuration. An AI coding system must not only understand the requirement but also read existing code, identify architectural constraints, plan a modification path, and fix issues through multiple rounds of validation.
The core of long-horizon engineering evaluation is therefore not code generation speed. It is whether the system can form a stable execution loop:
Understand requirements → locate relevant code → make a modification plan → implement changes → run validation → fix based on feedback → submit an acceptable result.
If any link in this process fails, the final task may fail. When interpreting evaluation results, attention should shift from a single ranking to engineering delivery quality. In particular, it is important to distinguish what pass rate, completion, elapsed time, and cost each measure.
2. Why Pass Rate and Completion Must Be Evaluated Separately
1. Pass Rate Measures Acceptance Results
Pass rate usually refers to the proportion of tasks judged successful under given evaluation rules. It is closer to “whether the delivery meets acceptance conditions” than to “how much code has been written.”
For example, an offline editing feature may include the following requirements:
- Users can create and modify content;
- Edits can be saved locally;
- Data can be restored after the app restarts;
- Unsubmitted content is not lost during network failures;
- Related interactions do not break existing functionality.
Even if the first four items are implemented, a serious regression in the last item may still cause the entire task to fail. For changes involving data consistency, permission boundaries, or critical business flows, this strict acceptance is reasonable.
However, pass rate is also affected by task difficulty, acceptance criteria, retry strategy, and execution environment. Comparing numbers without these conditions can easily mistake differences in evaluation configuration for differences in system capability.
2. Completion Measures Partial Progress
Average completion is often used to describe the degree to which task requirements have been met, but there is no unified calculation method. It may come from the proportion of tests passed, scoring of requirement sub-items, or weighted results under review rules.
Before interpreting completion, therefore, three questions must be clarified: What is the scoring basis? How are the weights determined? Are there veto conditions for critical requirements?
High completion and low pass rate can occur at the same time. This usually means the system completed many sub-goals but did not stably solve the last few issues that determine delivery success. Common causes include:
- The main flow runs, but exception paths are insufficiently handled;
- Code changes are mostly correct, but integration validation is incomplete;
- The new feature matches the description but breaks existing interface constraints;
- The UI is implemented, but state restoration or data consistency has defects.
High completion does not mean close to production. A small permission defect may matter more than several unfinished UI details. Teams cannot directly convert completion into remaining manual effort.
3. What High Completion and Long Runtime Reveal About Engineering Trade-offs
Long execution may mean the system performed more code retrieval, builds, tests, and fixes. It may also mean it repeatedly tried the wrong direction. Total runtime alone cannot show whether that time produced useful value.
One case is “effective depth”: the system gradually discovers boundary issues, continuously reduces defects through validation, and finally outputs a more reliable change.
Another case is an “inefficient loop”: the system repeats similar operations, expands the modification scope, but produces no new valid evidence and may even introduce additional regressions.
Distinguishing the two requires keeping an execution trace, not just recording start and end times. Total runtime should be broken down at least into:
- Code understanding time: reading requirements, searching files, analyzing dependencies;
- Modification implementation time: writing code, adjusting configuration, updating tests;
- Validation time: building, running tests, executing static checks;
- Environment waiting time: resource queuing, dependency retrieval, test device preparation;
- Failure recovery time: locating errors, rolling back, and retrying.
This breakdown helps teams locate the real bottleneck. If most time is consumed by environment preparation, optimizing the execution environment may be more effective than changing the model. If time is concentrated on repeatedly fixing the same type of error, the task description, feedback quality, or termination policy may need improvement.
Average runtime is also insufficient to represent actual experience. For teams with clear delivery deadlines, median runtime, higher-percentile runtime, and timeout ratio should also be tracked to prevent a few long-tail tasks from delaying the entire development process.
4. Cost Is More Than the Price of a Single Run
Single-run cost is an important metric, but it is not a complete economic metric. A low-cost task that requires extensive manual remediation may not have a lower final delivery cost. A high-cost task may still be valuable if it significantly reduces rework. Whether it is worthwhile must be judged within a specific workflow.
The following framework can be used:
Actual delivery cost = automated execution cost + execution environment cost + human review cost + rework cost.
Automated execution cost may include model calls and related computing resources. Environment cost may involve test devices, storage, and isolated runtime resources. Human cost includes reading changes, confirming requirements, handling residual defects, and performing final acceptance.
For batch tasks, teams can also calculate:
Average cost per accepted task = total investment for the batch ÷ number of accepted tasks.
This metric should be calculated using actual aggregated data from the same batch of tasks. If no tasks pass, the fact should be reported directly rather than presenting a meaningless average.
Mechanical estimates of retry benefits should also be avoided. Failures across multiple executions of the same task are often correlated. If the system consistently misunderstands requirements, increasing retries may simply consume budget repeatedly. Whether retries are effective should be verified through additional passed tasks, changes in error types, and the human takeover ratio.
5. The Evaluation Target Is the Entire Execution System
The result of an engineering task depends not only on the model itself but also on the agent workflow, tool permissions, validation feedback, and runtime environment.
The same model may produce noticeably different results when given a clearer code index, more reliable test feedback, and a more reasonable execution budget. Conversely, if the environment is unstable and error information is missing, even a model that can propose correct changes may fail to complete final acceptance.
A rigorous evaluation record should therefore include:
- Model and agent configuration;
- Task description and information allowed to be accessed;
- Available files, execution environment, and tool scope;
- Time, call count, and budget limits;
- Whether retries, execution recovery, and human intervention are allowed;
- Test and scoring rules;
- Handling of failures, timeouts, and environment exceptions.
If these conditions differ, comparisons must state the differences clearly. Overall system results cannot be directly attributed to a single model’s capability.
6. Building a More Credible Mobile Development Evaluation Set
1. Divide Tasks by Real Work Types
Internal evaluation should cover daily development work rather than selecting only problems that are easy to score automatically. The following categories can be considered:
- App skeleton and basic page creation;
- Extension of existing features;
- Defect localization and repair;
- Data structure and interface adaptation;
- Architecture adjustment and code migration;
- Exception handling and compatibility fixes;
- Test supplementation and regression governance.
Tasks should also be differentiated by complexity. For example, a single-page adjustment and a cross-module migration require different budgets and acceptance methods. They should not be mixed directly and then reported only as a single total score.
2. Define Acceptance Criteria Before Execution
Requirements must be verifiable. “Optimize the experience” is difficult to score objectively, while “show a retryable state when loading fails, and repeated clicks do not create duplicate submissions” is easier to turn into clear validation steps.
Acceptance conditions should cover normal paths, exception paths, and inviolable constraints. For critical requirements such as security, permissions, and data integrity, necessary pass conditions can be set so that defects are not hidden by scores on other sub-items.
3. Prevent Local Patches Aimed at Visible Tests
If the system only fixes code around known tests, tests may pass while real behavior remains incorrect. Independent acceptance cases, boundary inputs, and manual spot checks can be used to verify whether the implementation truly meets the requirements.
At the same time, failed tasks and their reasons should be retained. Showing only successful cases systematically underestimates implementation risk.
7. Turning Evaluation Results into an Actionable Development Workflow
1. Establish a Stable Baseline Before Changes
Before execution, confirm that the project can build, existing tests can run, and original failures are recorded. Otherwise, newly appearing issues may be unrelated to the current change, and the system may waste a large budget fixing historical environment problems.
2. Split Tasks Along Independently Verifiable Boundaries
Reasonable splitting is not simply cutting requirements into shorter text. It means giving each subtask clear inputs, outputs, and checks.
For example, a data synchronization requirement can be split into data structure adjustment, local storage behavior, synchronization state handling, conflict strategy, and UI feedback. Each step should state its dependencies to avoid incompatible local implementations.
3. Build Layered Validation
Validation should start with cheap, fast checks and gradually move to checks that are closer to the real runtime environment:
Formatting and static checks → local tests → module tests → integration tests → key user flow validation.
When an earlier layer finds an obvious issue, it should be fixed before moving to later stages, reducing wasted execution cost. Passing an earlier layer cannot replace later validation, and “build success” in particular cannot be equated with “business correctness.”
4. Set Stop and Escalation Rules
Agents should not loop indefinitely. Based on task risk, teams can predefine maximum runtime, budget caps, consecutive no-progress counts, and allowed modification scope. These are internal governance rules and should be adjusted through pilots rather than treated as universal parameters.
When the system repeatedly encounters the same error, needs to modify sensitive modules, lacks necessary permissions, or cannot determine acceptance requirements, execution should pause and be handed over to a human. Handoff materials should include at least modified files, the current issue, checks performed, and suggested next steps.
8. Focus on Deliverability, Not Isolated Scores
The value of AI coding for mobile apps lies in whether it can continuously produce engineering outcomes that are reviewable, verifiable, and maintainable. Pass rate shows whether tasks are completed. Completion describes partial progress. Runtime affects delivery cadence. Cost determines the economics of scaling. Human takeover rate and rework volume reveal the real boundary of automation.
For development teams, a safer path is to start with lower-risk tasks that have clear acceptance criteria, continuously record successes and failures, and then gradually expand to complex scenarios. Selection should not pursue the highest score divorced from context. It should seek an execution approach that performs stably under the team’s own task distribution, budget constraints, and quality requirements.
Only by evaluating model capability, execution environment, test system, and human governance together can teams determine whether AI coding is accelerating delivery or merely accelerating code generation.
