Interpreting Metrics and Selection Practices for AI Coding in Mobile App Development

    AI coding is evolving from simple code completion to executing long-horizon tasks such as cross-file modifications, builds, and test runs. For mobile app development, evaluating these capabilities requires a shift from leaderboard rankings to real-world engineering delivery. This article explores how to interpret key metrics like pass rate, completion rate, execution time, and total cost. It also provides practical guidance on building trustworthy internal evaluations, transforming failure attribution into actionable improvements, and adopting tiered task scheduling to deliver reliable, verifiable software changes at a controlled cost.

    I. Shifting from Leaderboard Rankings to Real-World Engineering Delivery

    AI coding is transitioning from localized code completion to handling long-horizon tasks that involve cross-file modifications, executing builds, running tests, and continuously fixing issues. This shift is particularly pronounced in mobile app development: creating apps, adding business features, migrating legacy code, and restructuring applications cannot be accomplished by simply generating a correct snippet of code.

    A single feature might simultaneously involve UI state, data persistence, permission handling, background tasks, and exception recovery. Even if most of these components are implemented, the entire task may fail acceptance if issues exist in the critical path. Therefore, evaluating AI coding capabilities should not merely ask “Is the generated code reasonable?” but rather “Can it deliver verifiable results under given constraints?”

    To understand such evaluations, it is essential to distinguish three distinct objects:

    • Model capabilities: The ability to understand requirements, reason about constraints, generate code, and analyze errors.
    • Agent execution capabilities: The ability to organize steps, read files, call tools, retain context, and determine stopping conditions.
    • Engineering environment conditions: Build configurations, dependency availability, testing facilities, compute resources, and network limitations.

    The final outcome is the result of the interaction among all three, rather than a single attribute of the model itself.

    II. Pass Rate vs. Completion Rate: Answering Different Questions

    1. Pass Rate Reflects End-to-End Delivery Capability

    Pass rate typically represents the proportion of tasks that meet all necessary acceptance conditions. For evaluations emphasizing end-to-end delivery, a task is judged as failed if required tests are still failing, even if the majority of the development is complete.

    This strict criterion has practical significance. In real-world projects, a “page displays but submission fails” scenario is usually not considered a delivered feature; “basic flow works but data is lost after an upgrade” cannot be accepted simply because a large amount of work was completed.

    However, pass rate has its limitations: it compresses information about partial progress. Two failed tasks might differ vastly—one might only need a boundary condition fix, while the other might have never successfully built. Relying solely on the final pass rate makes it difficult to assess the amount of subsequent human handover required.

    2. Average Completion Rate Reflects Phased Progress

    Completion rate is usually calculated based on subtasks, checklist items, or test results, describing how far the system has progressed. However, different evaluations define “completion” differently: some score by the number of tests passed, others by functional checklist items, and some may include structural requirements.

    Therefore, completion rate must be read alongside the scoring rules. For example, a task might include several routine checklist items and one critical data consistency requirement. If all routine checks pass but the core consistency requirement fails, the completion rate might still be high, yet the delivery risk cannot be ignored. This indicates that the quantity proportion of checklist items does not equal business importance proportion.

    Internal enterprise evaluations should adopt a “mandatory gates plus bonus items” approach: critical acceptance conditions must be met entirely, while other checklist items describe quality and progress. A large number of minor scores should not offset severe defects.

    3. What High Completion Rate with Low Pass Rate Usually Means

    Such results often indicate that the system excels at building the main implementation but gets blocked during the finishing stages. Common obstacles include:

    • Insufficient understanding of implicit conditions in requirements;
    • Cross-module modifications not synchronized;
    • Timing issues in asynchronous operations;
    • Incorrect state recovery after lifecycle transitions;
    • Repeatedly modifying local code after test failures without locating the root cause.

    However, one cannot determine the failure cause solely based on two aggregate metrics. To confirm these judgments, a review combining execution traces, failed tests, and final code is necessary.

    III. What Does Long Execution Time Actually Buy?

    Longer execution times can represent either thorough verification or inefficient exploration. These two scenarios must be clearly distinguished.

    Effective investments typically include: reading relevant modules, mapping call relationships, executing targeted tests, checking the impact of modifications, and fixing regressions. Although these activities increase time consumption, they may reduce the difficulty of human handover.

    Ineffective consumption may manifest as: repeatedly reading the same batch of files, toggling between conflicting implementations, repeatedly executing the same operation without changing inputs, or continuously waiting for unrecoverable external conditions.

    Therefore, teams should not directly interpret long execution time as “deep thinking,” nor should they equate short execution time with “higher efficiency.” A more reasonable approach is to break down the time sources:

    • Requirement analysis and code location time;
    • Actual modification time;
    • Build and test waiting time;
    • Failure diagnosis and rework time;
    • Retry time with no effective progress.

    Additionally, distinguish between wall-clock time and cumulative resource time. When tasks are executed in parallel, the two can differ significantly; without specifying the statistical scope, time data is difficult to compare horizontally.

    IV. Low API Costs Do Not Mean Low Delivery Costs

    The cost of AI coding includes at least direct execution fees, environment costs, manual review costs, and rework costs after failures. Comparing only the book expenditure of a single run easily underestimates the true cost borne by the engineering team.

    Cost accounting can be organized using the following relationship:

    Total Engineering Cost = AI Coding Execution Fees + Runtime Environment Costs + Manual Review and Repair Costs + Attributable Rework Costs.

    Among these, manual costs often dictate the final economics. A solution with low execution fees but uncontrolled modification scopes might require developers to invest significant time understanding diffs, troubleshooting hidden risks, and reorganizing code. Conversely, a solution with higher execution fees, but clear change boundaries, comprehensive tests, and easy review, may not necessarily result in a higher total cost.

    It is recommended to track at least the following metrics simultaneously:

    Metric Primary Question Answered
    Cost per attempt How many resources are needed to initiate a task
    Acceptance pass rate How many tasks can be fully delivered
    Cost per successful delivery How much total investment is needed to achieve one verifiable result
    Human handover time How much manual effort is required after automated execution ends
    Time to first verifiable result How long it takes to get a genuinely usable result

    Special attention should be paid to the fact that “average cost per attempt divided by pass rate” can only serve as a rough estimate under conditions like consistent statistical scopes and representative samples. It cannot replace actual cost accounting that includes failed attempts and manual handovers.

    V. How to Build a Trustworthy Internal Evaluation

    1. Sample from Real Task Distributions

    Evaluation tasks should cover the team’s daily work, rather than only selecting minor changes that are easy to verify automatically. Mobile app scenarios can include:

    • New pages and their complete interaction flows;
    • Feature extensions for existing modules;
    • Persistence structure adjustments and historical data compatibility;
    • Exception handling for permission denial, network interruption, etc.;
    • Legacy code migration and common module refactoring;
    • Build failure or runtime defect localization.

    Different categories should be reported separately to prevent an excess of simple tasks from masking deficiencies in high-risk tasks.

    2. Fix Acceptance Criteria Before Execution

    Before task initiation, input conditions, necessary behaviors, restricted modification areas, and test requirements should be explicitly defined. Otherwise, evaluators might unintentionally relax or tighten standards after seeing the results.

    For UI-related tasks, do not just verify if the page can open; check interaction states, error prompts, and behavior across different window sizes. For data-related tasks, focus on consistency, repeated operations, and failure recovery.

    3. Unify Runtime Conditions and Retain Traces

    Comparable results require comparable environments. Code baselines, dependency versions, resource quotas, timeout limits, and external access conditions should be as consistent as possible.

    Execution records should at least retain task descriptions, code diffs, tool calls, test outputs, cost statistics, and stop reasons. Records are not only for reproduction but also for distinguishing between “insufficient capability” and “environment failure,” preventing infrastructure anomalies from being directly attributed to the AI coding system.

    4. Report Distributions and Uncertainties

    Small-scale task sets are easily affected by task difficulty variations and execution randomness. In addition to averages, provide the median execution time, tail performance of slower tasks, failure types, and necessary uncertainty notes.

    If the pass rate comes from multiple runs, weighted statistics, or grouped aggregations, it should be clearly labeled. Readers should not be misled into thinking it is simply the proportion of tasks passed in a single run.

    VI. Turning Failure Attribution into Actionable Improvements

    Failure analysis should translate into actionable engineering steps, rather than generalizing it as “the model is not strong enough.”

    • Requirement understanding failure: Supplement with verifiable behavioral descriptions, input/output examples, and boundary conditions.
    • Code location failure: Improve module descriptions, entry point navigation, and dependency documentation to reduce aimless searching.
    • Implementation scheme failure: Require a brief plan before modification, clarifying involved modules, state changes, and compatibility impacts.
    • Insufficient verification: Bind task completion conditions to builds, relevant tests, and regression checks. Avoid substituting natural language claims for execution evidence.
    • Execution out of control: Set budget caps, no-progress detection, and manual escalation conditions. Upon consecutive failures, re-locate the problem first instead of endlessly appending retries.

    This attribution framework can transform leaderboard-style evaluations into R&D process optimizations: whether a problem requires stronger reasoning capabilities or merely improvements in task descriptions, testing facilities, or stopping strategies must be judged separately.

    VII. Adopt Tiered Scheduling Instead of Uniform Configuration

    Different tasks have different requirements for correctness, timeliness, and cost, making it unwise to use the same execution strategy for all.

    For tasks with small impact scopes that are easy to verify and roll back, tighter time and cost budgets can be applied, prioritizing fast feedback. For feature development involving multiple modules, increase plan reviews, phased testing, and change scope checks. For high-risk tasks such as account permissions, sensitive data processing, and irreversible data changes, retain manual design confirmation and final approval.

    A general workflow is:

    Task Classification → Clarify Acceptance Criteria → Set Execution Budget → Auto Modify & Verify → Check Risks and Results → Pass, Escalate, or Rollback.

    Escalation conditions should be as objective as possible, such as persistent critical test failures, obvious expansion of modification scopes, involvement of sensitive logic, or budget exhaustion. Whether to continue investing should not rely solely on the system’s confidence in its own results, but on checkable evidence.

    VIII. Measuring the Value of AI Coding with Verifiable Results

    The value of long-horizon development evaluation for mobile apps lies not in establishing a permanently valid ranking for all scenarios, but in revealing capability boundaries under different engineering constraints.

    Pass rate measures complete delivery, completion rate describes phased progress, execution time reflects the process, and cost represents resource investment. Only by combining these metrics with manual handover, defect risk, and maintainability can a reliable judgment be formed.

    For development teams, a reasonable path is to first establish a small, real, clearly standardized, and reproducible internal task set, and then implement tiered scheduling based on business risk. The ultimate goal is not to make the AI coding system generate more code, but to deliver more reliable, easily verifiable, and maintainable software changes at a controlled cost.