Mobile App Agent Development Benchmark: A Combined Evaluation of Pass Rate, Completion, Latency, and Cost

    A recent public benchmark for mobile app development agents shows how pass rate, completion, latency, and cost interact in real engineering tasks. This article summarizes the disclosed results, explains how to interpret benchmark scores, introduces cost per effective task, and offers practical guidance for model selection, tiered scheduling, and workflow design.

    1. Background: Why a Development Benchmark for Real Engineering Tasks Is Needed

    As large language models move from “code completion” toward “task-level agents,” evaluation methods must evolve as well. Early evaluations mostly focused on function-level code generation, algorithm problems, or snippet completion. A model only needed to produce a correct snippet within a limited context to score well. Real enterprise mobile app development, however, is a long-horizon, multi-stage process: understanding requirements, planning modules, writing UI and business logic, debugging build errors, adapting to different OS versions, and even migrating legacy projects to new frameworks. If any stage fails, the task fails.

    Therefore, over the past year or two, the industry has seen programming benchmarks centered on “end-to-end tasks.” The core idea is to give an agent a complete development goal—for example, “create an app with login, list, and detail pages” or “migrate a project to a new architecture”—let it execute autonomously in a real repository environment, and then determine whether the task is truly complete through automated tests and manual review. This evaluation approach is closer to production reality and has become one of the recognized yardsticks for measuring an agent’s engineering capability.

    2. Key Information and the Broader Landscape in the Public Data

    According to public information, the benchmark platform recently disclosed, for the first time, the results of several mainstream code agent models on this benchmark. From the combined performance of these models, several key observations can be drawn:

    1. The pass-rate gap among leading models is converging, but tiers remain clear.

    Public data shows that the strongest model achieved a pass rate close to 30% on about 30 long-horizon tasks, while another differently positioned model achieved about 17%. It is important to note that “pass rate” measures the proportion of tasks that are fully solved. Even if completion is high—for example, more than 80% of code functionality has been implemented—the task is still counted as a failure if final acceptance is not passed. This metric deliberately emphasizes “usability” rather than “half-finished work,” reflecting a delivery-oriented engineering value.

    2. Completion and pass rate are two complementary measures.

    One model had a pass rate of about 24%, but its average completion was above 82%. This means the model can push most tasks to a near-complete state, with weaknesses concentrated in final integration, builds, or edge cases. For development teams, the practical value of such a model is that it can handle most repetitive work, while human engineers only need to focus on final quality control. Conversely, if a model has an acceptable pass rate but low completion, it suggests a structural weakness in planning, and intermediate stages are likely to break down.

    3. Latency and cost have become new decision dimensions.

    The most engineering-significant comparison in the data comes from two models with very different positioning:

    • One model took an average of about 27 hours per benchmark run, with a cost of approximately RMB 2,000+ per run;
    • The other took only about 12 hours on average, with a cost of under RMB 100 per run.

    The pass-rate difference between them was about 6.7 percentage points, but the cost difference was more than 30x. This clearly reveals a fact: in the agent era, model capability is no longer the only selection variable. “The marginal cost of each real task solved by an agent” is becoming a core consideration for enterprise deployment.

    3. How to Interpret Benchmark Results Correctly: Four Methodological Caveats

    Benchmark data is authoritative, but it still requires careful interpretation. The following is a framework that developers and decision-makers should establish.

    1. Pass rate is a statistical measure, not a deterministic commitment.

    Long-horizon tasks have high variance: small differences in an agent’s exploration paths, tool calls, or retry strategies can change the final result. A score from 30 tasks in a single run can only provide a rough estimate of the capability range. A more robust approach is to focus on consistency across multiple runs—for example, a combination of pass rate and completion—or to require the evaluator to disclose variance across multiple samples.

    2. The impact of the agent framework and scaffold cannot be ignored.

    The same base model, paired with different agent frameworks—the orchestration layer responsible for planning, tool calls, and memory management—may produce very different results. In the public data, all participating models were connected through specific agent implementations. Therefore, a benchmark score is actually the joint performance of “model + agent framework + prompts and toolchain.” When teams deploy models themselves, if the scaffold design differs, actual results may be expected to deviate from the leaderboard.

    3. Task distribution determines how well scores transfer.

    This benchmark focuses on mobile app development scenarios and covers four task types: app creation, feature development, code migration, and app conversion. If a team’s workload is mainly server-side development, data pipelines, or legacy system refactoring, the transfer value of leaderboard scores will be reduced. The right approach to model selection is to first use the benchmark to shortlist candidates, and then run small-scale retests with the team’s own real task set.

    4. Cost accounting must use a consistent basis.

    Single-run costs on leaderboards usually include only model inference fees, not tool-call overhead in the agent loop, manual review costs, or hidden losses from failed retries. Enterprises should incorporate total cost of ownership (TCO) into their evaluation. The formula can be simplified as:

    1. Cost per effective task = Cost per run / Task pass rate

    Using the public data for a rough estimate, the “effective cost” of the high-capability model is approximately 317.4 / 0.24 ≈ USD 1,322; the low-cost model is approximately 9.9 / 0.173 ≈ USD 57. Even after accounting for pass-rate differences, the lightweight model still has an order-of-magnitude advantage in cost per effective task—this is the data basis for an “on-demand tiering” strategy.

    4. Practical Implications for Development Teams

    1. Schedule by task complexity instead of relying on a single model for everything.

    Based on the analysis above, a pragmatic strategy is to establish two-tier (or multi-tier) scheduling: long-horizon, complex tasks such as feature development and code migration are handled by high-capability agents; high-frequency, low-difficulty tasks such as unit test generation, boilerplate code, documentation, and simple fixes are handled in batches by low-cost, fast models. This architecture can be implemented at the inference gateway or orchestration layer and is model-agnostic. It is a widely recognized best practice in the industry today.

    2. Build “human finishing” into the process design.

    An average completion rate above 82% means agent output already has high engineering value, but last-mile build fixes, boundary conditions, and acceptance alignment still require human engineers. Teams should treat agent output as a “high-quality first draft” and pair it with code review, automated testing, and build pipelines to form a closed loop of “generate—build—verify—fix,” rather than pursuing fully automated delivery.

    3. Build your own evaluation set and run continuous regression tests.

    Leaderboards are a starting point, not the endpoint. Teams should select 20–50 typical tasks from real requirements—covering their own tech stack, code style, and acceptance criteria—and retest candidate models quarterly. They should track trends in four metrics: pass rate, completion, time, and cost. Agent capabilities iterate very quickly, and today’s high-scoring model may be overtaken within months. A self-built evaluation set is the only reliable way to keep decisions timely.

    4. Pay attention to supporting infrastructure and engineering platforms.

    Regardless of how model capabilities evolve, deployment depends on a stable engineering foundation: version control, continuous integration, containerized build environments, and logging, monitoring, and alerting systems. For enterprises that want to build such capabilities quickly, mature cloud infrastructure and service categories—including code hosting, container platforms, pipeline services, and observability tools—can reduce environment setup costs. General cloud product categories such as those offered by Baidu AI Cloud can serve as one reference option when building an agent runtime environment.

    5. Conclusion

    The greatest value of this round of public benchmark data is not a specific score. Rather, it establishes a four-dimensional evaluation paradigm—long-horizon task pass rate, completion, latency, and cost—and brings “cost per effective task” formally into selection discussions. For development teams, the rational path is to use public benchmarks to build a candidate pool, use a self-built task set for final validation, and then convert agent capability into measurable, sustainable productivity through tiered scheduling and closed-loop processes.