AI Coding Benchmarks: A Selection Guide for Long-Horizon Mobile Development Tasks

    This article breaks down what modern AI coding benchmarks actually measure and how to read them when the workload is mobile app development rather than a single coding puzzle. It explains the four core metrics (pass rate, average completeness, average time, and cost per benchmark run), interprets a recent set of scores ranging from 17.3% to 28.7% pass rate and from $9.9 to $317.4 per run, and analyzes why long-horizon tasks are hard. It then offers a practical selection framework organized by task risk and complexity, an engineering approach for making long-horizon tasks controllable and verifiable (with budget-aware agent orchestration), guidance on building an internal benchmark, and the key trends shaping model-plus-agent evaluation.

    In recent years, the way AI coding capability is evaluated has changed markedly. Early benchmarks mostly measured single-file function completion or the correctness rate on standalone programming problems. Such tasks have clear boundaries and are simple to verify, which makes them suitable for gauging a model’s coding fundamentals. But in real engineering, development is often a long chain: a single goal may require cross-file edits, dependency installation, build compilation, and test verification before it can be delivered—what the industry commonly calls “long-horizon tasks.”

    Mobile app development is a typical high-difficulty scenario. A feature has to span the UI, business logic, the data layer, and build configuration, and it must pass every stage—compilation, static analysis, unit tests, simulator installation, and runtime verification—before it counts as truly done. For this reason, industry benchmarks aimed at real engineering scenarios have emerged, testing whether a model can complete tasks such as app creation, feature development, code migration, and cross-platform conversion.

    Recently, a mobile development platform publicly released a new set of benchmark scores. Across three representative results, pass rates ranged from 17.3% to 28.7%, and the cost of a single benchmark run ranged from $9.9 to $317.4. Behind these numbers lies not only the gap in model capability, but also the engineering complexity of long-horizon tasks and the impact of cost efficiency on selection. This article interprets what these metrics really mean, analyzes the capability profiles of models at different tiers, and offers practical selection and implementation advice.

    1. Understanding AI Coding Benchmarks: Four Core Metrics

    An AI coding benchmark for mobile development typically has a model or agent operate autonomously in a near-real repository environment, carrying a preset task through the full pipeline—understanding requirements, planning an approach, editing code, and verifying the build—and then judges the result against objective, repeatable criteria. To read such a benchmark, you first need to understand four core metrics.

    1. Pass rate: did the task actually get done

    The pass rate is the share of tasks that fully satisfy every acceptance condition, and it is the most discriminating hard metric. “Passing” generally means the build succeeded, all relevant tests passed, and the behavior of key features matches expectations. It emphasizes the correctness of the outcome, not merely that “code was written” or that the task “looks done.” Because failure at any single stage causes the whole task to fail, the pass rate reflects a model’s end-to-end delivery reliability fairly well.

    2. Average completeness: how far from the finish line

    Average completeness measures how much of the verifiable subgoals a model completes on average across tasks. These subgoals usually come from build artifacts, check results, or the feature points covered by tests. It complements the pass rate:

    • The pass rate answers “did it get done?”;
    • Average completeness answers “how far did it get?”

    A common combination is high completeness with a low pass rate. This indicates that the model can usually write the main functionality but repeatedly gets stuck at the finishing stages. In such cases, more thorough verification feedback and limited human intervention can often raise output fairly quickly.

    3. Average time: wall-clock time for long-horizon tasks

    Average time records how long a model takes to complete a single task on average, covering model inference, code editing, build compilation, test execution, and even waiting for a simulator to launch. Time directly affects the experience in three scenarios:

    • Interactive coding assistants: users often tolerate only seconds to minutes of waiting;
    • Overnight batch jobs: tasks can run unattended for a long time;
    • Continuous integration pipelines: task duration determines feedback speed.

    Average times on the order of 27 hours versus 12 hours correspond to an “unattended long-running” mode, which requires supporting task orchestration and failure-recovery mechanisms.

    4. Cost per benchmark run: the underestimated scaling variable

    The cost per benchmark run is the total spend when a model runs the full benchmark suite, covering token consumption from model calls as well as the accumulating multi-turn context created by tool calls. It is worth noting that token consumption on long-horizon tasks tends to grow rapidly with the number of turns:

    • Every interaction turn has to carry the prior conversation history;
    • The process of reading source code, executing commands, and inspecting error logs keeps expanding the context.

    As a result, the cost of end-to-end long-horizon tasks is far higher than a per-call estimate would suggest. During prototype validation, cost differences may be negligible; but once you move into batch development or organization-wide adoption, cost becomes the key factor that determines whether a solution can scale.

    2. Reading the Scores: The Distribution of Capability and Cost

    From the published representative results, capability levels roughly fall into three tiers. The model types in the table below are my own categorization for analytical convenience; they do not represent any specific product or vendor, and the data reflects only this set of results:

    Tier Representative Type Pass Rate Avg. Completeness Avg. Time Cost per Run (approx.)
    Tier 1 Top-tier general-purpose LLM 28.7% Not disclosed Not disclosed Not disclosed
    Tier 2 Leading LLM + mature agent 24.0% 82.3% 27.4 hours ~$317.4
    Tier 3 Lightweight model + agent 17.3% 71.2% 12.0 hours ~$9.9

    1. The leading model is ahead, but the gap is limited

    The top model’s 28.7% is higher than the second tier’s 24.0%—roughly two additional tasks completed out of 30. This shows that a model with stronger general capability still has an edge on engineering tasks, but the lead is limited. The gap shows up more in the stability on complex tasks than in a simple “can do it or can’t.”

    2. The “finishing bottleneck” behind high completeness

    The second-tier model has an average completeness of 82.3% but a pass rate of only 24.0%. Put differently, about 60% of tasks completed most of the work yet failed to pass overall. Given the nature of mobile app development, this usually means the model tends to make mistakes at stages such as:

    • The main code is done, but build configuration or resource references are missing something;
    • The functional logic is largely correct, but boundary tests or compatibility checks fail.

    This also implies that thorough automated validation and limited manual patching could meaningfully raise the final usable rate.

    3. Lightweight models: trading some capability for an order-of-magnitude cost advantage

    The third-tier model’s pass rate (17.3%) is slightly below the second tier’s, and its average completeness (71.2%) trails by about 11 percentage points—but the cost per run is only about 1/32 of the second tier, and its time is about 44% of it. For tasks that are well-structured, clearly bounded, and highly repetitive—such as page template generation, boilerplate code completion, and unit-test draft writing—this level of capability already meets the basic requirements.

    4. Cost differences per unit of effective output

    For budget-sensitive teams, a more useful metric than “cost per run” is “cost per task passed”:

    Metric Tier 2 Model Tier 3 Model Difference
    Avg. tasks passed (30 × pass rate, estimated) ~7.2 ~5.2 —
    Cost per run $317.4 $9.9 ~32×
    Cost per task passed (estimated) ~$44.2 ~$1.9 ~23×

    Even after accounting for success rate, the third-tier model’s cost per task passed is still only about 1/23 of the second tier’s.

    These figures are extrapolated from public data and reflect only the order-of-magnitude relationships in this set of results, not the price of any specific product. They show that although the cost gap is partially offset by success rate, the cost advantage remains significant. So using lightweight models for high-frequency, low-risk tasks, then applying human review or a stronger model as the key gate, is a combination strategy worth evaluating.

    5. Comparability of results: variables beyond the leaderboard

    When comparing across models, note a few variables that do not show up in the scores:

    • Agent framework: the same model paired with agents of differing maturity can perform very differently in planning, tool invocation, and error recovery;
    • Runtime environment: build caching, how the simulator is launched, and test-environment stability all affect time and success or failure;
    • Cost accounting: whether it includes context caching, tool calls, and repeated attempts can change the numbers noticeably.

    3. Why Pass Rates Are Generally Low: Sources of Failure in Long-Horizon Tasks

    Even the top model fails to pass more than 70% of tasks. This reflects the inherent difficulty of long-horizon tasks and is exactly where engineering improvements can pay off. The common causes fall into five categories:

    1. Long-range state consistency is hard to maintain: migration or conversion tasks involve many files and multiple steps, and early design decisions (such as directory structure, naming conventions, and dependency versions) must stay consistent throughout. As context accumulates, early constraints tend to fade, and mid-to-late changes can drift away from the original agreements.
    2. Local errors accumulate downstream: for example, introducing an incompatible dependency, or changing an interface signature without updating all callers. Such local issues keep amplifying in later steps and eventually surface as build failures or runtime exceptions.
    3. Verification feedback is weak and slow: mobile verification depends on time-consuming stages such as building, installing, and launching the simulator, so a single “modify-and-verify” cycle can take minutes or longer. Feedback is slow and coarse-grained, making it hard for the model to pinpoint the root cause quickly, so it tends to make repeated ineffective edits.
    4. Implicit constraints are hard to enumerate: requirements such as target platform versions, screen adaptation, permission declarations, and localized resources are rarely stated explicitly in the task description and require engineering common sense to cover.
    5. Cost constraints limit the search space: once turn or token budgets are constrained, the model may rush to finish without fully investigating, or the task may be interrupted when the budget runs out.

    4. A Selection Framework for Mobile Development

    Selection should not aim for “the strongest model” but for “a sensible match between tasks and tools.” You can build an evaluation framework around four dimensions.

    1. Task tiering: route by risk and complexity

    Task Type Typical Examples Recommended Strategy
    High-value, high-complexity Core module development, cross-platform migration, architecture-level refactoring Prefer more capable models, paired with mature agents and strict human review
    Medium complexity Routine feature development, bug fixes Start with a lightweight model to draft a solution, then escalate to a stronger model as needed
    High-frequency, low-risk Template code, boilerplate screens, test drafts, code comments Prefer lightweight, low-cost models; optimize for throughput and cost efficiency

    2. Timeliness and budget: make the constraints explicit

    In interactive scenarios, prefer models that respond quickly and pass reliably; in overnight batch scenarios, you can accept longer times in exchange for higher completeness; in budget-constrained scenarios or those that need to scale out, focus on the cost per task passed and total cost of ownership.

    3. Agent and toolchain: they set the ceiling on capability

    The same model paired with different agents can perform noticeably differently. The capabilities that matter most include:

    • Planning and task decomposition;
    • Accuracy of tool invocation;
    • Error recovery;
    • Completeness of the verification stage.

    Teams should evaluate the “model + agent” as a whole.

    4. Security and compliance: a line that cannot be crossed

    Prefer model services that support private deployment or dedicated data protection. For scenarios involving core code, also establish mechanisms to filter sensitive information and audit access, ensuring that code and business data are not leaked.

    5. Engineering Implementation: Making Long-Horizon Tasks Controllable, Verifiable, and Reusable

    1. Build a reliable verification loop

    Mobile development tasks are hard largely because the standard for “done” is fuzzy. Teams should make acceptance conditions explicit and automate them as much as possible:

    1. Requirements → generate acceptance checklist → compile/static analysis → unit tests → simulator install & smoke test → result judgment

    The acceptance checklist should cover dimensions such as build success, functional behavior, compatibility, and resource integrity. The more objective and automated the verification, the more efficient the model’s trial-and-error, and the more credible the evaluation conclusions.

    2. Adopt a budget-aware orchestration strategy

    For long-running tasks, it is advisable to add a “budget guard” at the orchestration layer that checks the time and cost budget before each step. When the budget is insufficient, wrap up in time or hand off to a human to avoid wasted consumption:

    1. # Illustrative code: a generic agent execution loop with budget constraints and objective verification
    2. MAX_STEPS = 40 # maximum number of steps
    3. MAX_COST = 5.0 # cost cap per task (currency unit)
    4. DEADLINE = 3 * 3600 # timeout per task (seconds)
    5. def run_task(task, agent, workspace, verify):
    6. used_cost, start = 0.0, now()
    7. history = []
    8. for _ in range(MAX_STEPS):
    9. # Budget guard: save current progress and exit when the limit is exceeded
    10. if used_cost >= MAX_COST or now() - start >= DEADLINE:
    11. return {"status": "budget_exceeded", "history": history}
    12. action, cost = agent.next_action(task, workspace, history)
    13. used_cost += cost
    14. outcome = agent.execute(action, workspace)
    15. history.append(outcome)
    16. # Objective verification: rely on repeatable standards such as build, static analysis, and tests
    17. report = verify(workspace)
    18. if report.passed:
    19. return {"status": "passed", "cost": used_cost, "history": history}
    20. # On failure, feed structured error information back to the agent to avoid repeated ineffective edits
    21. task = task.with_feedback(report.errors)
    22. return {"status": "max_steps_reached", "history": history}

    In the example, both the “agent” and the “verifier” are abstract interfaces. The actual implementation can choose models and verification tools as needed while keeping the logic generic.

    3. Support long-horizon tasks with general cloud capabilities

    Long-horizon tasks run for a long time, consume significant resources, and have a high probability of failure, which places clear demands on the underlying infrastructure:

    • Container platform: provide each task with an isolated, one-off execution environment so builds and tests do not interfere with one another, and reclaim it automatically when the task ends;
    • Object storage: centrally store task context, build artifacts, test reports, and run logs for traceability and retrospective analysis;
    • Message queue: smooth out peaks and troughs, carry asynchronous tasks, and support long-running jobs and failure retries;
    • Logging and monitoring/alerting: continuously track task success rate, average time, and cost consumption to catch anomalies early.

    Teams can build this kind of evaluation and execution platform on their own using general cloud capabilities such as those offered by Baidu AI Cloud (container platform, object storage, message queue, logging, and monitoring/alerting). This helps track model and agent performance continuously at a controllable cost.

    4. Establish a quality gate centered on “pass rate”

    In a continuous integration flow, you can set an independent quality gate for AI-generated changes. It should include at least:

    • The code must compile successfully;
    • Static analysis has no blocking-level issues;
    • All relevant unit and smoke tests pass;
    • The change goes through human code review.

    Only changes that meet every condition may be merged, which prevents mistaking high completeness for high quality.

    6. Continuous Evaluation: Build a Benchmark for Your Own Scenarios

    Public benchmarks reflect a model’s general level, but every team’s business scenarios, coding standards, and tech stacks differ. It is advisable for teams to draw 20 to 30 representative tasks from their own repositories, covering the main types of day-to-day development, build a small internal evaluation set, and continuously track the following metrics:

    • Pass rate and average completeness;
    • Cost per task passed;
    • Average time;
    • Human rework rate—the proportion of AI output that gets modified by people.

    In practice, two common pitfalls should be avoided:

    • Focusing only on pass rate while ignoring rework cost and long-term maintenance burden;
    • Testing only in ideal environments while ignoring real-world complications such as dependency conflicts and legacy code.

    The evaluation environment should stay as close to production reality as possible.

    The latest scores reveal several trends:

    • Capability gradation: the difference between leading and lightweight models shows up more in stability on complex tasks than in “can it do it or not”;
    • Cost structure: the cost of long-horizon tasks rises quickly as context accumulates, making the cost per unit of effective output a core metric;
    • Verification engineering: objective, automated verification mechanisms are becoming the “infrastructure” that determines the quality of AI coding output;
    • System-level competition: what is being compared is no longer just the model itself, but the overall effectiveness of “model + agent + toolchain + verification system.”

    It is worth remembering that model versions iterate quickly, and any single benchmark reflects only the state at a particular point in time and should not be treated as a fixed conclusion. The value of a leaderboard lies in revealing technical trends and providing a comparison baseline, not in replacing validation in real business scenarios. For development teams, a more pragmatic approach is to:

    • First clarify the distribution of your own tasks and their constraints;
    • Then establish a tiered selection and objective verification mechanism;
    • Finally, calibrate continuously with your own evaluation set;
    • So as to find the best balance among quality, speed, and cost.