Evaluating AI Code Generation Models for Mobile App Development: Metrics and Engineering Trade-offs

    This article examines how to evaluate AI programming models for real-world mobile application development tasks. It introduces four core metrics—pass rate, average completion rate, average execution time, and per-run cost—and analyzes the engineering trade-offs between them. The article also provides practical guidance on designing an evaluation workflow, including task set design, environment standardization, automated acceptance testing, and cost tracking, followed by actionable recommendations for model selection based on scenario-specific priorities.

    Why Evaluation Should Target Real Engineering Tasks

    Early code generation benchmarks typically focused on single functions or algorithmic problems with well-defined inputs and outputs that could be verified using a few test cases. Such tasks differ significantly from actual software development. Real mobile application projects usually span multiple stages—project scaffolding, UI and state management, local storage, network requests, permission handling, and version migration—requiring cross-file consistency, successful compilation, runtime stability, and compatibility compliance.

    Evaluation based on real engineering tasks therefore provides a more accurate reflection of a model’s practical value. The core question shifts from “can it generate a snippet of correct code” to “can it deliver a runnable result within a complete project.” Teams evaluating AI-assisted development tools should prioritize results from this type of benchmark over isolated demonstrations.

    Four Core Metrics and What They Mean

    1. Pass Rate

    Pass rate refers to the proportion of tasks in which the final output satisfies all acceptance criteria, which typically include successful compilation, passing functional test cases, and the absence of critical runtime errors. Pass rate is the strictest metric—it only credits fully correct outcomes.

    Its main limitation is the binary nature of the judgment: it cannot distinguish between “nearly finished” and “completely off target.” In long-horizon tasks, a single minor defect can cause an entire task to be marked as failed.

    2. Average Completion Rate

    Average completion rate measures the proportion of sub-goals that were correctly implemented within a task, scored item by item. It reflects a model’s ability to deliver partial results. For scenarios where human developers will take over to finish the remaining work, completion rate is often more informative than pass rate.

    The risk of this metric is that the average can obscure the underlying distribution. A model that fully succeeds on a few tasks and performs moderately on most others may have the same average as one that nearly completes most tasks but occasionally fails entirely—yet the user experience would be very different. Distributions should always be examined alongside the average.

    3. Average Execution Time

    Average execution time measures the total duration from when the model receives a task to when it produces the final output. This includes inference time, tool invocation time, and multi-turn interaction time. Execution time directly affects developer waiting costs and determines whether the model can be integrated into CI/CD or batch processing pipelines.

    Execution time is influenced by task complexity, the agent’s execution strategy, and the number of retries. The same model can vary by several times in execution time under different scheduling strategies, so the runtime environment and configuration must be specified when making comparisons.

    4. Cost per Run

    Cost per run is the expense incurred for a single complete benchmark task, generally determined by the computational volume of inference and the number of API invocations. This metric directly relates to team budgets. For scenarios requiring extensive automated invocation, small differences in per-run cost can be significantly amplified at scale.

    It is important to note that cost data is only comparable when the pricing basis is consistent—including how input and output tokens are billed, whether caching is included, and whether retries are counted. Cost figures without a clearly defined pricing basis should not be compared directly across models.

    Engineering Trade-offs Between the Metrics

    The four metrics do not necessarily move in the same direction. Typical combinations observed in public evaluations include:

    • High completion with low pass rate: The model gets most steps right but tends to leave defects in the final stage. These results are suitable for workflows where human review is a built-in step.
    • Moderate completion with low latency and low cost: The model responds quickly and inexpensively, but its planning capability for long-horizon tasks is limited. This profile is better suited for breaking work into smaller tasks and processing them in batch.
    • Pass rate positively correlated with execution time: The model invests more inference cycles and self-checking steps during the task, resulting in more stable outcomes—at the cost of increased waiting time.

    Evaluations do not yield a single “best model” in absolute terms; they reveal which model fits a given scenario. Teams should first identify the metrics they care about most, then assign weights accordingly.

    Designing an Evaluation Workflow

    If a team needs to reproduce a similar evaluation internally, the following workflow can serve as a reference:

    1. Task set design: Cover typical stages including project initialization, UI development, data persistence, network integration, and version migration. Each task should be decomposed into individually verifiable sub-goals.
    2. Standardized runtime environment: Fix dependency versions, build tools, and emulator configuration to ensure all models are evaluated under identical conditions.
    3. Automated acceptance testing: Compilation checks, unit tests, UI tests, and static analysis should be automated as much as possible to reduce subjective bias in manual assessment.
    4. Multiple runs with averaged results: Repeat the same task multiple times and record the distribution of pass/fail outcomes to avoid drawing conclusions from a single run.
    5. Track cost and time concurrently: Record input/output sizes, waiting times, and retry counts for each invocation to facilitate subsequent cost accounting.

    Below is a simplified evaluation record structure for aggregating metrics across tasks:

    1. from dataclasses import dataclass
    2. from statistics import mean
    3. @dataclass
    4. class RunResult:
    5. task_id: str
    6. passed: bool # whether all acceptance criteria are met
    7. completion: float # completion rate from 0 to 1
    8. elapsed_hours: float # execution time in hours
    9. cost_usd: float # cost of this run
    10. def summarize(results: list[RunResult]) -> dict:
    11. total = len(results)
    12. return {
    13. "pass_rate": sum(r.passed for r in results) / total,
    14. "avg_completion": mean(r.completion for r in results),
    15. "avg_elapsed_hours": mean(r.elapsed_hours for r in results),
    16. "total_cost_usd": sum(r.cost_usd for r in results),
    17. }

    This structure is only illustrative. Teams can extend the fields to fit their own task sets and metric systems, such as adding failure cause classification and the number of manually modified lines.

    Recommendations for Model Selection

    Selection decisions can be approached in the following order:

    • Consider the tolerance for errors in the target scenario first. If generated code goes directly into production, pass rate should be prioritized. If outputs go through human review before use, completion rate and modification cost are more relevant.
    • Consider scale and frequency next. High-frequency, batch-oriented automated tasks are more sensitive to cost and execution time. Minor differences in per-run cost can become significant in monthly billing.
    • Finally, consider how the team collaborates. If developers prefer to iterate and modify code interactively, response speed matters more than one-shot complete delivery.

    In addition, any evaluation conclusion has a limited shelf life. Model versions, agent strategies, and pricing rules all change over time. Teams should re-run evaluations periodically and maintain documentation of the evaluation methodology so that results from different time periods remain comparable.

    Conclusion

    Evaluation based on real engineering tasks provides more reliable reference data than single-point demonstrations. The four metrics—pass rate, average completion rate, average execution time, and per-run cost—each capture a distinct aspect of performance and involve clear trade-offs between one another. Teams should begin by defining their business priorities, then conduct evaluations under a consistent methodology to select the approach that best fits their specific scenario.