A New Benchmark Leaderboard for Mobile AI Development: Pass Rate, Completion, and the Cost Trade-off

    A new long-horizon coding benchmark for mobile app development measures AI agents on pass rate, completion rate, runtime, and cost. This article explains what the leaderboard data reveals about the capability–cost spectrum, why high completion with low pass rate is a meaningful engineering signal, and how teams can adopt practical strategies for model selection, task routing, and cost control.

    1. A Shift in Evaluation Paradigm: From “Answering Questions” to “Delivering Work”

    1.1 The Limits of Traditional Benchmarks

    Historically, AI code generation has been evaluated much like a written exam: give the model a function signature and docstring, have it fill in the function body, then validate correctness against a pre-written suite of unit tests. These “HumanEval-style” static benchmarks are objective and cheap to run, but they have three notable blind spots:

    • Task granularity is too small: In real development, a single feature often involves multi-file changes, interface design, and compatibility handling. Completing one function cannot cover this collaborative work.
    • Lack of context understanding: Real repositories carry many implicit conventions—architecture layers, naming standards, existing abstractions. Code generated without repository context often cannot be merged directly.
    • The engineering loop is ignored: Delivery is not just “writing code”—it includes compiling, running tests, fixing defects, and iterating.

    1.2 Why Long-Horizon Task Benchmarks Matter

    The mobile platform coding benchmark behind this leaderboard (best understood as a long-horizon evaluation for app development scenarios) attempts to close these gaps. Its core design: given a near-realistic task description—such as “create a new app module with a given feature,” “add a capability to an existing app,” “migrate legacy code to a new API system,” or “convert one UI paradigm to another”—an AI agent autonomously plans, writes, executes, and fixes code in a real engineering environment. An automated pipeline then determines whether the task was truly completed.

    The metrics go beyond pass rate alone and include at least four dimensions:

    Metric Meaning Engineering Significance
    Pass rate Share of tasks fully achieving the goal Measures the reliability ceiling
    Average completion Share of sub-goals achieved per task Measures the value of “partially usable” output
    Average runtime Time per benchmark run Measures interaction and waiting costs
    Running cost Compute cost per benchmark run Measures the barrier to scale

    The multi-dimensional design itself sends an important message: there is no single “strongest model”—only models that match a given scenario.

    2. Reading the Leaderboard: A Typical “Capability–Cost” Spectrum

    2.1 High-End Models: High Completion, Long Runtime, High Cost

    The top-performing long-horizon models show a distinctive profile: the highest pass rates and average completion above 80%, but with average task runtimes measured in days and per-run costs in the hundreds of dollars. At first glance this seems poor value, but the engineering logic becomes clear on closer inspection:

    • What 80%+ completion actually means: Even when a task does not fully pass, the AI has completed most sub-goals. For developers, “reviewing and fixing an 80%-complete draft” is far more efficient than “implementing from scratch”—the marginal cost of human intervention drops sharply.
    • Where the long runtime comes from: Long-horizon tasks require repeated cycles of planning, execution, observing failures, and backtracking to fix. Longer runtimes often mean the model invests more in “self-repair” attempts on hard tasks—both a sign of capability and a current efficiency bottleneck.
    • The nature of high cost: Under the agent paradigm, models repeatedly call tools, read context, execute, and retry, and token consumption grows exponentially. The cost problem is fundamentally one of long-context management and runaway tool-call counts.

    Key insight: These models suit “task-based” workflows—describe the full requirement clearly, let the AI work autonomously for an extended period, and review at milestones. What they replace is not “typing” but “implementation.”

    2.2 Lightweight Models: Fast and Cheap, Completion Is “Good Enough”

    At the other end of the leaderboard, lightweight models achieve pass rates of roughly 10–20% and average completion around 70%, but with runtimes a fraction of high-end models and per-run costs in the single-digit dollars. These numbers point to a very different set of use cases:

    • Interactive assistance: Real-time completion, explanation, and small snippet generation during coding require millisecond-to-second responses—where lightweight models offer excellent value.
    • High-frequency, low-difficulty tasks: Scaffold code, simple UI layouts, and routine data processing do not demand deep reasoning; lightweight models handle them without over-thinking and slowing the workflow.
    • Budget-sensitive teams: For startups and individual developers, scaling high-end model usage is unaffordable. “Good enough and cheap” is the rational first principle.

    2.3 Lessons from the Middle of the Spectrum

    Several models with balanced overall capabilities sit in the middle of the leaderboard. Their shared lesson: each additional percentage point of pass rate typically comes with a non-linear cost curve. When selecting models, teams should first map the difficulty distribution of their own tasks, then decide where to spend budget—rather than blindly chasing the top of the leaderboard.

    3. High Completion with Low Pass Rate: An Important Engineering Signal

    The most thought-provoking phenomenon in the leaderboard is that some models show “high average completion but low pass rate.” This seemingly contradictory data actually describes the real boundary of today’s agent coding capabilities:

    1. Long-tail sub-goals are hard to crack: In typical long-horizon tasks, 80% of sub-goals (UI construction, logic implementation, data binding) are relatively patterned and can be completed reliably. The remaining 20% often involve platform-specific compatibility handling, edge-case state conditions, or changes requiring global consistency—precisely the current weakness of these models.
    2. The value of a “nearly there” failure: A failed task at 82% completion means something very different from one at 30%. The former is a final step away from done; human effort to finish it is far lower.
    3. Pass rate is a double-edged metric: Counting only “all-or-nothing” passes systematically underestimates the practical value of high-completion models. Internal evaluations should incorporate completion rate and human rework hours—metrics closer to production reality.

    The practical recommendation: manage AI like “a capable junior engineer who occasionally gets stuck”—assign complete tasks, require runnable intermediate outputs, and set up human review checkpoints, rather than treating it as “an exam machine that must get everything right the first time.”

    4. Practical Adoption Guidance for Different Teams

    4.1 Evaluate First: Build Your Own Task Sample Set

    A general-purpose leaderboard’s weighting may not match your business. Teams should extract 20–30 real historical tasks (covering creation, feature addition, migration, and refactoring), run candidate models against them with the same acceptance criteria, and observe how the four metrics behave on your task distribution. A simplified scoring approach:

    1. Task value = Completion × Human-effort-saved coefficient − (Runtime × time cost + Running cost × call frequency)

    4.2 Layered Usage: Route Models by Task Difficulty

    • Real-time coding assistance, simple generation → lightweight fast models;
    • Medium-complexity standalone features → balanced models;
    • Cross-module migrations, complex features built from scratch → long-horizon high-end models + human review.

    4.3 Controlling Cost and Risk in Long-Horizon Tasks

    • Cap retries: An agent’s “self-repair loop” is the main source of wasted spend. Set maximum rounds and budget circuit breakers.
    • Shrink context: Load only relevant files on demand rather than the whole repository, significantly reducing token overhead.
    • Enforce acceptance gates: All AI output must pass compilation, testing, and code review pipelines—bypassing quality gates to merge directly is not allowed.

    4.4 Designing the Human–AI Workflow

    A recommended collaboration pattern is “AI leads, humans review”: humans handle requirement clarification and task decomposition; AI handles implementation and first-round debugging; humans retain acceptance testing, edge-case handling, and architectural decisions. This leverages the model’s scale advantage in long-horizon implementation while avoiding its unreliability on critical judgment calls.

    5. Conclusion

    The real value of this leaderboard is not ranking models from strongest to weakest, but using multi-dimensional data to force the industry to face a fact: AI coding capability has entered a stage where capability, speed, and cost cannot all be maximized at once. Pass rate represents the ceiling, completion represents usability, and runtime and cost determine scalability. The rational approach for teams is to find their own optimal point on this spectrum based on their task distribution and budget constraints, supported by engineered mechanisms for evaluation, routing, and acceptance. Leaderboards will keep updating—but this “data-driven selection mindset” is the methodology that delivers lasting value.