AI Coding Evaluation for Mobile Apps: Balancing Quality and Cost

    This article examines how to evaluate AI coding systems for mobile app engineering tasks by looking beyond code generation to complete delivery. It explains why pass rate and completion rate should be observed together, how to break down long runtimes, how to account for full delivery costs, why the evaluation target is an execution system rather than a model alone, how to design realistic task sets, and how teams can turn evaluation results into risk-tiered workflows.

    1. From “Writing Code” to “Completing Engineering Tasks”

    The evaluation of AI coding capability is shifting from local code generation to complete engineering task delivery. For mobile app development, this change is especially important: a piece of code that looks correct does not mean the app can build or launch, let alone that the feature meets business requirements.

    Real development tasks usually span multiple stages, including understanding requirements, locating existing implementations, modifying UI and business logic, handling dependencies, running tests, and continuing to fix issues based on error feedback. Failure at any one of these stages can cause the final delivery to fail.

    Therefore, the evaluation focus should not be merely “how much code was generated,” but should shift to the following questions:

    • Does the system accurately understand the requirements and their boundary conditions?
    • Does it preserve the existing project structure and interface constraints?
    • Does it complete builds, tests, and necessary runtime verification?
    • Can it converge within a limited time and budget?
    • Is the delivered result easy for developers to review, maintain, and continue iterating on?

    From this perspective, AI coding evaluation is not simply measuring the amount of model knowledge. It is testing an engineering system composed of the model, execution environment, tool-calling mechanisms, and feedback strategies.

    2. Why Pass Rate and Completion Rate Must Be Observed Together

    1. Pass rate reflects complete delivery capability

    Task pass rate can usually be expressed as:

    Task pass rate = number of tasks meeting all necessary acceptance conditions ÷ total number of valid evaluation tasks.

    It answers a direct question: how many tasks can be accepted according to pre-agreed criteria?

    This metric is suitable for measuring automated delivery capability, but only if the acceptance conditions are sufficiently clear. For example, a data entry feature may require input validation, state persistence, error prompts, and data restoration after returning. Even if the UI is complete, the entire task cannot be judged as passed if the necessary data restoration capability is missing.

    Note that pass rate depends on task difficulty, acceptance strictness, number of attempts, and execution budget. Different evaluation results are comparable only when these conditions are broadly consistent.

    2. Completion rate reflects partial progress but does not mean the result is ready for release

    Average completion rate usually describes the proportion of acceptance items met within a task, and the specific calculation method depends on the evaluation design.

    If a task contains multiple acceptance items, equal weighting can be used, or weights can be set according to business importance. Whichever method is used, the rules should be published in advance to avoid changing the criteria after the evaluation is complete.

    High completion rate and low pass rate are not contradictory. A common reason is that most routine steps have been completed, but key acceptance items still fail. For example, the main flow may run, but data is lost in an exception state; or the feature behaves correctly but introduces a regression in an existing module.

    Therefore, completion rate is better suited to helping determine “how far the result is from delivery,” rather than replacing pass rate. For requirements involving data integrity, permission control, and sensitive information protection, hard gates should be set that cannot be offset by other scores.

    3. Only by combining both types of metrics can engineering bottlenecks be identified

    If the pass rate is low but the completion rate is high, teams need further analysis: are failures concentrated in a small number of critical steps? Can those steps be completed manually at low cost? Or are seemingly local issues actually requiring a refactor of the entire implementation?

    The former may mean the system has good value as development assistance; the latter indicates that the high completion rate overestimates actual progress. Only by combining failure reasons and manual repair records can these two cases be distinguished.

    3. What Long Runtimes Actually Buy

    Increased time on long-horizon tasks sometimes comes from more thorough exploration and validation, and sometimes from inefficient retries, repeated changes in direction, or environment issues. Runtime length cannot be directly interpreted as “deeper thinking.”

    During evaluation, total time should be broken down into:

    • Requirement analysis and solution formation time: understanding constraints, inspecting project structure, and formulating a modification plan.
    • Code modification time: generating implementations, adjusting interfaces, and adding tests.
    • Build and validation time: compiling, launching the app, and running automated checks.
    • Waiting time: resource queuing, dependency retrieval, and external service responses.
    • Rework time: fixing errors, reverting changes, and repeated attempts.

    This breakdown has direct optimization value. If most time comes from builds, caching and environment preparation should be checked. If most time comes from repeatedly fixing the same error, feedback extraction and stopping strategies should be improved. If a large amount of time is spent on ineffective exploration, requirement constraints, task decomposition, and stage checks should be strengthened.

    It is also important to distinguish between “single-task duration” and “full-suite evaluation duration.” Parallel execution can shorten the calendar time of a full evaluation, but it does not necessarily reduce resource consumption. Without disclosing concurrency and statistical scope, time metrics can easily be misleading.

    4. Cost Evaluation Should Cover the Full Delivery Chain

    The cost of a single execution is only part of the cost. What enterprises really care about is how many resources are required in total to obtain an acceptable result.

    The following accounting framework can be used:

    Full delivery cost = AI coding execution cost + compute and storage cost + validation cost + human review and repair cost + failed retry cost.

    For tasks that are not yet complete, the cost already consumed without producing a qualified delivery should also be recorded. Do not count only successful samples.

    Under the same task set and consistent budget constraints, the following can be further observed:

    Unit successful delivery cost = total cost of all valid attempts ÷ number of successfully delivered tasks.

    If no task succeeds, this metric should not be presented directly as an ordinary number. It should be clearly stated that no valid estimate of successful delivery cost can be formed.

    Low execution cost does not necessarily mean low total cost. If generated results are difficult to understand and the scope of changes gets out of control, developers may need to spend a great deal of time cleaning up. Conversely, a solution with higher execution cost may demonstrate overall value only if it reduces manual work or lowers rework risk.

    Therefore, cost analysis must be placed in the same evaluation table as quality and time, rather than compared independently.

    5. The Evaluation Target Is a Complete Execution System, Not Just a Model

    AI coding results are affected by multiple factors, including model capability, context organization, file retrieval mechanisms, tool permissions, error feedback, and retry strategies.

    For example, when facing the same build failure, one execution workflow may pass only the last line of the error to the model, while another may extract the relevant file locations, dependency conflicts, and earlier errors. The repair results can differ significantly.

    To improve comparability, the evaluation should at least fix and record:

    1. The initial code version, dependency state, and build environment.
    2. The available tools, network conditions, and file access permissions.
    3. The time, cost, and retry limits for each task.
    4. The requirement text, supplementary instructions, and human intervention rules.
    5. Acceptance scripts, runtime logs, and final modification records.

    If these conditions differ, the conclusion should be limited to “system performance under a specific configuration” and should not be simply attributed to the model itself.

    6. How to Design Task Sets That Reflect Real Development

    1. Cover different types of engineering tasks

    Mobile app evaluation should not include only creating simple pages from scratch. It should also cover feature extensions in existing projects, bug fixes, structural adjustments, dependency migrations, and behavior-preserving refactoring.

    Different tasks test different capabilities. New feature development relies more on requirement understanding and implementation capability; bug fixing relies more on localization and validation; migration tasks place more emphasis on compatibility constraints and regression control.

    2. Clearly define necessary conditions and optional goals

    Each task should specify input conditions, expected behavior, allowed modification scope, and constraints that must not be broken. Optional goals such as visual optimization and code style adjustments should not be conflated with data correctness and basic feature availability.

    At the same time, acceptance should not be overly tied to a single implementation path. As long as the result satisfies the constraints, reasonable technical choices should be allowed, avoiding the mistake of treating “different from the reference implementation” as failure.

    3. Validate both new features and existing behavior

    Validating only new features can easily miss regression issues. A reasonable acceptance hierarchy should include build checks, local logic tests, cross-module flow tests, and necessary real runtime checks.

    Among these, a successful build is only a basic condition. Whether the app can correctly restore state, handle failed input, release resources, and maintain core behavior under different runtime conditions should be verified according to the characteristics of the task.

    4. Preserve reviewable evidence

    Each execution should retain the task description, code diff, test results, key logs, resource consumption, and human intervention records. For failed tasks, the final state should also be saved so that it can be determined whether the failure is due to requirement misunderstanding, implementation defects, environment failure, or insufficient validation.

    7. Turning Evaluation Results into Practical Workflows

    Teams do not need to assign all tasks to the same execution strategy. A more reasonable approach is to handle tasks in tiers based on risk, complexity, and verifiability.

    For tasks with clear boundaries, small impact scope, and easy automated validation, a lower-budget execution workflow can be used, followed by human review after checks pass.

    For tasks involving multi-module interaction, data structure changes, or complex state management, a plan should be formed first and then implemented in stages. Each stage needs clear artifacts and acceptance points to avoid accumulating a large number of unverified changes at once.

    For high-risk tasks, AI coding is better suited as assistance for analysis and implementation. Changes involving permissions, sensitive data, and irreversible operations should retain explicit human confirmation steps.

    An executable workflow can be:

    Requirement clarification → risk classification → task decomposition → limited execution → automated validation → human review → merge and continuous observation.

    Among these, “limited execution” should include time and cost boundaries. When multiple consecutive rounds of modification do not improve validation results, unrelated changes begin to expand, or the preset budget is reached, automatic retries should stop and the task should be handed over for human analysis.

    8. Make Decisions Based on Deliverability, Not a Single Ranking

    The most valuable output of AI coding evaluation is not an isolated score, but a clear statement of what results the system can deliver under what tasks, constraints, and budgets.

    For developers, the focus should be on whether results are verifiable, whether changes are understandable, and whether failures are easy to take over. For enterprises, permission boundaries, audit records, budget control, and long-term maintenance costs must also be considered.

    Pass rate measures complete delivery, completion rate helps locate gaps, duration reveals execution efficiency, and cost reflects resource investment. Only by combining these metrics with task risk, failure types, and the burden of manual repair can teams determine whether AI coding truly improves the R&D process.

    The ultimate goal is not to have the system continuously generate more code, but to stably deliver engineering outcomes that meet requirements, are easy to maintain, and can be traced within controllable time and budget.