Evaluating AI-Assisted Mobile App Development: Completion Rate, Time, and Cost Trade-offs
Evaluation of AI coding for mobile apps is shifting from whether a model can complete a code snippet to whether an agent can deliver an engineering task. This article explains why end-to-end delivery matters, defines four core metrics—task pass rate, average completion, execution time, and running cost—and discusses how to interpret the trade-offs among them. It also outlines the variables a credible evaluation must control, including task inputs, runtime environment, budgets, stopping conditions, model-versus-agent workflow separation, and statistical uncertainty. Finally, it gives development teams a practical method for building reproducible evaluation sets and a delivery loop with risk-based automation and human takeover.
1. From Code Generation to Engineering Delivery: What Is Being Evaluated?
The evaluation of AI coding capability is moving from “can it complete a piece of code?” to “can it complete an engineering task?” In mobile app development, this shift is especially important. App creation, feature expansion, code migration, and conversion of existing apps are not tasks that can be completed by generating a few functions.
A seemingly simple feature requirement may involve UI state, lifecycle, permission requests, data persistence, network exceptions, and build configuration at the same time. Even if the generated code is locally correct, the result may still fail to become a deliverable because of missing resources, dependency conflicts, or incomplete state restoration.
Long-horizon engineering evaluation therefore examines a complete chain:
Requirement understanding → project exploration → solution planning → code modification → build verification → test repair → result delivery.
Any link in this chain can become a bottleneck. A model that is good at explaining requirements is not necessarily able to correctly modify a complex project. Being able to complete a build does not mean the feature meets acceptance requirements. When reading an evaluation, the first thing to clarify is this: the result usually reflects the combined performance of the model, the agent execution strategy, the tool environment, and the resource budget—not the absolute capability of a model outside a given environment.
2. Four Core Metrics Answer Different Questions
2.1 Task Pass Rate: Does the Final Result Meet Acceptance Requirements?
Task pass rate usually represents the number of tasks that meet the specified acceptance conditions as a proportion of the number of valid evaluation tasks:
Task pass rate = tasks passing acceptance ÷ total valid evaluation tasks.
It is closest to the “can it be delivered?” question that development teams care about, but its interpretation depends heavily on the acceptance rules. For example, a feature may require both an operable UI and offline availability with exception recovery. If one critical condition is not met, the entire task may be judged as failed.
Strict full acceptance is useful for measuring delivery reliability, but it can also obscure partial progress. Therefore, pass rate is suitable for answering “how many tasks ultimately meet the bar,” but not for explaining on its own “how far the failed tasks were from completion.”
2.2 Average Completion: How Much of the Requirements Was Covered?
Average completion is usually calculated based on the completion of requirement items, checkpoints, or test items. It can distinguish between “almost nothing was completed” and “only a small number of boundary conditions are missing.”
However, completion must be understood together with the scoring method. Passing a large number of low-difficulty checkpoints does not necessarily mean the core business path works. If a task contains multiple UI details and one critical data consistency requirement, a simple average may allow UI completion to mask core defects.
A more reasonable approach is to disclose all of the following:
- Whether the core business path passes;
- Whether critical blocking issues exist;
- The coverage ratio of general feature items;
- Whether non-functional requirements have been verified.
Therefore, “high completion with a limited pass rate” is not contradictory. It often means that most tasks have made substantive progress, but final integration verification, boundary handling, or stability requirements are still unmet.
2.3 Execution Time: More Than the Time to Generate Code
Long-horizon task time may include requirement analysis, project search, dependency preparation, build, test, failed retries, and result packaging. Different evaluations may also use different statistical scopes, so the definition must be confirmed first.
It is especially important to distinguish wall-clock time from cumulative execution time. The former is the actual elapsed time from start to finish, while the latter may sum the runtime of multiple parallel work units. The two cannot be directly substituted for each other.
In addition, reporting only the average is not enough. A small number of repeatedly failing tasks may significantly raise the average runtime. For teams with clear delivery deadlines, median time, high-percentile time, and timeout rate are often more informative than the average.
2.4 Running Cost: Not Equal to Full Delivery Cost
The cost of a single run only reflects direct spending within a certain statistical scope. If compute resources, build and test, failed retries, and human review are not included, it cannot be directly treated as project cost.
A cost expression closer to actual decision-making is:
Total delivery cost = agent execution cost + compute and testing cost + human review cost + rework cost.
Low-cost execution is not necessarily truly cheap if it frequently produces results that require deep repair. Higher execution cost may also have practical value if it significantly reduces manual debugging. Cost evaluation must be combined with delivery quality rather than comparing only the bill for a single run.
3. High Completion and Long Runtime: What Engineering Trade-off Do They Reflect?
High completion accompanied by long runtime may come from more thorough engineering exploration, or it may come from inefficient retries. A conclusion cannot be drawn from these two metrics alone.
3.1 Effective Exploration: Using More Verification to Reduce Uncertainty
In complex projects, understanding module boundaries, locating relevant code, and adding tests before making modifications is usually more robust than directly generating broad changes. These activities increase upfront time but may reduce later rework risk.
If the extra time is used to discover hidden dependencies, verify critical paths, and resolve real defects, the longer execution time can be regarded as an investment in quality.
3.2 Ineffective Loops: Repeated Operations Without New Information
Another case is when an agent continuously modifies similar code, repeatedly triggers the same build error, or switches solutions without new evidence. Such runs consume time but do not necessarily improve the probability of delivery.
Distinguishing the two requires process evidence, not just final duration. A practical criterion is whether each modification corresponds to an explicit hypothesis, whether verification results update the judgment, and whether subsequent actions change accordingly.
3.3 Diminishing Returns: The Last Part May Be the Hardest
The first half of an engineering task often produces visible progress, such as creating pages, connecting data, and implementing basic interactions. The second half may focus on exception recovery, concurrency behavior, device differences, and compatibility issues.
As a result, completion does not grow linearly with time. Teams should record milestone results and observe whether additional investment continues to produce effective gains. After reaching a time or cost limit, moving to human handling may sometimes be more reasonable than continuing automatic retries.
4. What Variables Must a Credible Evaluation Control?
4.1 Unify Task Inputs and Runtime Environment
Different solutions should use the same initial project, requirement description, dependency snapshot, and runtime resources wherever possible. Otherwise, whether environment preparation goes smoothly may affect results and undermine the basis for capability comparison.
Network access, cache availability, and permissions for external materials should also be clearly recorded. If one run can obtain complete dependencies while another is limited by network access, the two failures cannot simply be attributed to capability differences.
4.2 Define Budgets and Stopping Conditions
Evaluation should predefine the maximum runtime, allowed retry range, and human intervention rules. If one solution can repair indefinitely while another stops at a timeout, the final results are not equally comparable.
Timeouts should also be recorded as formal results rather than arbitrarily excluded. They reflect the actual usability of a solution under given resource constraints.
4.3 Observe the Model and Agent Workflow Separately
Engineering performance depends not only on the model but also on context organization, file editing methods, tool calls, and test feedback mechanisms.
A reasonable evaluation can be divided into two layers: one layer unifies the execution framework as much as possible to observe model differences; the other evaluates the complete system to measure the overall effect of a real workflow. The two answer different questions and should not be conflated.
4.4 Disclose Statistical Uncertainty
When the number of tasks is limited, the success or failure of a few tasks may change the ranking. Repeated runs of the same task set cannot be treated as equivalent to adding the same number of independent tasks, because those results are often correlated.
Evaluation reports should state the number of tasks, number of repetitions, aggregation method, and failure distribution. Where conditions allow, confidence intervals can be reported and results can be grouped by task type to avoid letting an overall average hide local weaknesses.
5. How Development Teams Can Design Their Own Evaluation Sets
Step 1: Choose Real and Reproducible Tasks
Do not test AI coding only with demonstration requirements. More valuable tasks should come from actual work, including modification of existing pages, bug fixes, data structure adjustments, test supplementation, and compatibility maintenance.
Each task should preserve the initial project state and be reproducible in a controlled environment. In this way, after a failure, it is possible to determine whether the problem came from code modification, environment configuration, or the task itself.
Step 2: Define Acceptance Before Execution
Acceptance conditions should cover both functional and engineering constraints. For example, a data entry feature should not only save content but also handle empty input, duplicate submission, abnormal exit, and state restoration after re-entry.
Critical requirements should have thresholds that cannot be offset by averaging. Issues involving sensitive data, permission boundaries, or data integrity cannot be considered deliverable simply because other check items perform well.
Step 3: Preserve Complete Process Evidence
At a minimum, record the following information:
- Modified files and their diffs;
- Build and test results;
- Time to first valid run;
- Failure types and retry counts;
- List of unresolved issues;
- Location and time of human intervention.
These records can turn “it failed” into an actionable diagnosis. For example, misunderstanding of requirements requires improving the task description; environment failures require fixing the execution infrastructure; repeated regressions require strengthening test constraints.
Step 4: Configure Automation Permissions by Risk
Low-risk, easy-to-roll-back, and well-tested modifications can increase the proportion of automated execution. When identity verification, sensitive information, payment logic, or destructive data operations are involved, human review should be added.
The degree of automation should be determined by task risk and verification capability, not by a particular evaluation ranking. Even a system with strong overall performance should not bypass necessary security and release processes.
6. Beyond Chasing Rankings: Building a Delivery Loop
The core of AI-assisted mobile app development is not how much code is generated at once, but how to continuously produce verifiable results under clear constraints.
In practice, a layered workflow can be used: first let the agent complete requirement decomposition and low-risk implementation, then filter results through build and test; when high-risk modifications, repeated failures, or diminishing returns appear, hand them over to developers for review and takeover. At delivery, submit code, verification evidence, and remaining risks together to avoid mistaking “it looks done” for “it is ready to release.”
Ultimately, task pass rate measures whether results meet the bar, completion explains the degree of progress, execution time reflects delivery pace, and cost determines the feasibility of scaling. Only by placing these four metrics back into unified tasks, budgets, and acceptance conditions can meaningful comparison be achieved.
For development teams, what is most worth building is not a static ranking table, but an engineering loop that supports reproducible evaluation, traceable execution, diagnosable failures, and human takeover. Such a mechanism can turn AI coding from a demonstration capability into stable productivity.
