Mobile App Coding Benchmark Reveals Model Capabilities and Engineering Costs

    As AI coding tools move into real development workflows, benchmarks for mobile app development are shifting from code snippet tests to complete engineering delivery. This article explains the metrics in a recent benchmark update, including pass rate, average completion rate, runtime, and cost per run. It also shows how teams can weigh model capability against cost.

    As AI coding tools move into real development workflows, judging a model only by whether its answers are “correct” no longer reflects its engineering value. For mobile app development, a model must do more than generate code. It needs to understand project structure, complete changes across multiple files, handle dependencies, run build pipelines, locate failures, and keep its context consistent through long chains of tasks. As a result, benchmarks built on real development tasks are moving from “code snippet evaluation” to “complete engineering delivery evaluation.”

    A recent public benchmark for mobile app development has updated its results and added data for three types of AI coding models. The tests cover app creation, feature implementation, code migration, UI adjustment, project conversion, and bug fixing. The results show that no single model leads on every dimension. Instead, there are clear trade-offs between capability, speed, and cost.

    1. From Isolated Answers to Complete Engineering Tasks

    Traditional code evaluations usually rely on function completion, algorithm problems, or short code generation. These tests measure syntax skills and local logical reasoning, but they remain far from real project development.

    Real mobile app tasks typically involve the following steps:

    1. Reading the existing project directory and configuration files;
    2. Understanding how the business requirements relate to the existing code;
    3. Modifying multiple source files;
    4. Adjusting resources, layouts, permissions, or build configuration;
    5. Running compilation, tests, and static checks;
    6. Fixing problems based on error logs;
    7. Verifying that the final functionality meets the requirements.

    A model’s final score therefore depends not only on the quality of its code, but also on whether it can work reliably over a long execution chain. A model may produce correct local code, but if it cannot handle dependency conflicts, build errors, or changes that span multiple files, it may still fail to complete the task.

    Accordingly, the pass rate in this kind of benchmark is closer to an engineering delivery metric, while the average completion rate shows how far a model’s output is from a usable final state.

    2. What the Metrics Measure

    The benchmark reports four types of data: pass rate, average completion rate, average runtime, and cost per run. Understanding how these metrics relate to each other is necessary for reading the leaderboard correctly.

    1. Pass Rate: Whether the Task Is Fully Closed

    Pass rate is usually the share of all test tasks that a model completes to the acceptance standard. It measures whether the result is complete, not how much work was done along the way.

    For example, a task may require three features: displaying data, page navigation, and error handling. If a model completes only two of them, the task may not count as passed, even if the code quality is high. Pass rate is therefore better suited to measuring whether a model can independently finish complex tasks.

    One model reached a pass rate of 24.0% across 30 long-horizon tasks, meaning it fully completed about one quarter of them. Another model reached 17.3%. That is lower, but it still shows the model can produce useful output on some complex engineering tasks.

    Pass rate alone does not capture a model’s full capability. The number of tasks, the mix of task types, and the strictness of the acceptance criteria can all affect the result.

    2. Average Completion Rate: How Far the Work Got

    Average completion rate measures how much of the work a model finishes on tasks it does not fully pass. It adds information that pass rate cannot show.

    One model reached an average completion rate of 82.3%. Even when some tasks missed the final standard, this model may have completed most of the requirements, such as:

    • Main pages were generated;
    • Core interaction logic was largely implemented;
    • Data structures and API calls were connected;
    • Only edge cases, compatibility, or build configuration remained problematic.

    Another model reached an average completion rate of 71.2%. It also has strong code generation ability, but it may be more likely to stop during complex wrap-up, error fixing, or multi-step coordination.

    For development teams, a model with a high completion rate may not fully replace human developers. However, it can significantly reduce the time spent building basic project structure from a blank project. Whether it creates real value still depends on how long engineers need to fix what remains.

    3. Average Runtime: Execution Efficiency

    Average runtime is the time a model takes from starting a task until it finishes or stops. A long runtime usually means the model went through more rounds of analysis, modification, building, and debugging.

    In the test, the high-completion model averaged 27.4 hours, while the other model averaged 12.0 hours. Several factors may explain the gap:

    • Whether it repeatedly runs builds and tests;
    • Whether it actively checks error logs;
    • Whether it uses multiple rounds of code repair;
    • Whether it keeps trying after a failure;
    • Whether it performs more thorough result verification.

    A longer runtime does not necessarily mean lower efficiency. For complex tasks, more verification and repair may explain higher completion. In interactive development, rapid prototyping, or urgent fixes, however, long execution cycles can hurt the user experience.

    3. Trade-offs Between High Completion and High Cost

    The high-capability model combined with an agent workflow cost about 317.4 US dollars per run. The other model cost about 9.9 US dollars per run. The cost gap is very large.

    This shows that model evaluation should not focus only on which model has the higher pass rate. It also needs to consider the practical output relative to cost.

    If a company runs many tasks every day, even a modest per-run cost difference adds up to significant spending over time. For example, a team that runs 20 complex tasks per day over 22 working days in a month makes 440 calls per month. At that scale, cost per run, the number of failed retries, and human review time all directly affect total development spending.

    A more reasonable evaluation uses composite metrics, such as:

    Cost per passed task = Total run cost ÷ Number of passed tasks

    Cost per effective output = Total run cost ÷ Completion-weighted task output

    Suppose model A has a higher pass rate but costs far more per run than model B. Model A only shows greater overall value if it significantly reduces human repair time. Conversely, if model B has a lower final pass rate but quickly generates most of the basic code, it may be better suited to low-risk, batch, or prototype tasks.

    4. Why Long-Horizon Tasks Reveal Model Differences

    Short code generation mainly tests local reasoning. Long-horizon tasks test sustained planning and state management. These usually involve the following areas.

    1. Project Understanding

    A model needs to identify entry files, module boundaries, build configuration, resource directories, and test structure. If it misreads the project layout, later changes can easily cause path errors, naming conflicts, or missing dependencies.

    2. Task Decomposition

    Complex requirements rarely get completed in a single pass. They usually need to be split into subtasks such as UI, data, state, navigation, testing, and build. How well the work is decomposed directly affects execution efficiency.

    3. Error Recovery

    In real engineering work, a first change rarely passes the build. The model needs to read logs, determine whether a problem is related to syntax, dependencies, types, permissions, or runtime logic, and then apply a targeted fix.

    4. Context Retention

    During a long run, the model needs to remember which files it has changed, which problems it has already fixed, and which goals remain open. When context management is unstable, the model may repeat changes, overwrite completed work, or introduce regressions.

    5. Verification Awareness

    Some models generate code but do not check whether the functionality is complete. An agent suitable for engineering use should run builds, tests, or basic behavior checks before finishing. It should not treat “code written” as “task complete.”

    5. How to Choose Models for Different Scenarios

    Development teams should not simply chase first place on the leaderboard. They should match models to task characteristics and use them in tiers.

    Scenarios Suited to High-Capability Models

    • Refactoring core business modules;
    • Migrating code across multiple modules;
    • Building complex UI and interactions;
    • Systematically repairing legacy projects;
    • Tasks with high requirements for stability and delivery quality;
    • Work that requires the model to carry out a large share of verification on its own.

    These tasks cost more, but if they reduce manual troubleshooting and rework, the overall investment can still be justified.

    Scenarios Suited to Lower-Cost Models

    • Adjusting page styles;
    • Converting simple data structures;
    • Adding test code;
    • Generating documentation and comments;
    • Organizing repetitive code;
    • Building early prototypes and proofs of concept.

    These tasks usually have clear boundaries and controllable risk, so a low-cost, fast-response model is a good fit.

    A company can build a task routing process like this:

    1. Use a lightweight model to assess task complexity;
    2. Execute low-risk tasks directly;
    3. Send tasks that involve multiple files, core logic, or build configuration to a high-capability model;
    4. Automatically retry or escalate tasks that fail on the first attempt;
    5. Have a human review the critical code.

    This approach costs less than using a high-cost model for every task, and it is more reliable than relying entirely on a low-cost model.

    6. Hidden Costs to Consider

    Public benchmarks provide useful reference points, but enterprise deployment involves additional factors.

    The first is human review cost. Code that builds successfully may still have weak exception handling, improper permission use, missing data validation, or poor maintainability.

    The second is environment adaptation cost. Build tools, dependency versions, test frameworks, and code standards vary between projects. A model’s performance in a standard environment does not necessarily carry over to an internal enterprise project.

    The third is retry cost. After a failed task, the model may need to reread the project and rerun the build. The actual cost can be much higher than the average reported in a leaderboard.

    The last is security and compliance cost. When code involves user data, authentication, payment flows, or sensitive business logic, it must go through strict review. Higher automation does not justify lower security standards.

    7. Building an Internal Evaluation System

    Before adopting AI coding tools, companies should build a small test set from their own projects. It should cover at least the following task types:

    • New feature development;
    • Refactoring legacy code;
    • Bug fixing;
    • Test completion;
    • Version migration;
    • Build troubleshooting;
    • Multi-module integration.

    Each task should have acceptance criteria defined in advance. These include whether the build succeeds, whether tests pass, whether the feature behaves as expected, whether the code meets standards, and how much time manual fixes take.

    Beyond pass rate, teams can also track: