Completion and Cost Trade-offs in Mobile App AI Coding Benchmarks

    This article examines how long-horizon mobile app AI coding benchmarks shift evaluation from code generation to closed-loop task delivery. It explains four core metrics—pass rate, completion, average duration, and benchmark run cost—and compares high-completion, long-running approaches with low-cost, fast approaches. It also outlines practical methods for task tiering, effective cost calculation, gray-release routing, and observable evaluation pipelines, while highlighting the limits of public benchmarks as production decision inputs.

    Completion and Cost Trade-offs in Mobile App AI Coding Benchmarks

    1. Background: Long-Horizon Tasks Shift Evaluation from “Can Write Code” to “Can Deliver a Closed Loop”

    A cloud provider recently updated an AI coding benchmark for mobile app development and published results for several long-horizon agentic coding approaches on real engineering tasks such as app creation, feature development, code migration, and app conversion. Unlike single-file completion or function-level generation, long-horizon tasks require an agent to understand requirements, break down steps, modify multiple files, run validation, read failure information, and decide whether to retry. In other words, the evaluation target is no longer just “code generation quality” but the ability to close the task loop.

    This also explains why, under the same set of tasks, different approaches show clear divergence in pass rate, completion, duration, and cost. One approach may complete most pages and logic but fail at a critical compilation, test, or migration validation step. Another may have a lower pass rate but respond faster and cost less, making it suitable as an initial screening layer or a first pipeline stage for low-risk tasks. Engineering teams should not simply choose the “highest score.” They need an evaluation and routing mechanism that explains the relationship among quality, speed, and cost.

    2. Four Core Metrics: Interpret Together, Not in Isolation

    1. Pass rate reflects the proportion of reliable deliveries. Pass rate usually refers to the percentage of tasks that fully succeed under given acceptance conditions. It is closest to the question “can this be delivered directly?” but strict pass rates are highly sensitive to acceptance criteria: test coverage, compilation success, functional execution, and code mergeability can each cause a task to fail. A high pass rate therefore means a lower probability of rework, but it does not by itself represent high completion.

    2. Completion reflects coverage of task elements. Completion measures the proportion of requirement elements, file modifications, feature points, or migration steps covered. High completion means an approach “did a lot,” but not necessarily “did it correctly.” For example, page structure, API calls, and state management may all be generated, but missing permission checks or exception branches can still cause acceptance failure. Completion is useful for determining whether a failure occurred because the agent gave up early or because it was close to completion but failed a key validation.

    3. Average duration affects feedback cycles and concurrency capacity. The average duration of long-horizon tasks can reach several hours or longer. The longer the duration, the longer developers wait for feedback, the more concurrent resources the evaluation pipeline consumes, and the higher the cost of timeouts and retries. For scenarios sensitive to budget and response time, duration itself is part of usability.

    4. Benchmark run cost determines scalability. Cost includes not only model calls but potentially sandbox execution, dependency installation, test execution, log storage, and failure retries. A more meaningful metric is “effective cost”: total cost divided by the number of passed tasks. In the public example, a high-completion approach running 30 long-horizon tasks had a pass rate of about 24%, average completion of about 82%, average duration of about 27 hours, and a benchmark run cost on the order of $300. A low-cost approach had a pass rate of about 17%, completion of about 71%, average duration of about 12 hours, and a cost on the order of $10. Based on a rough estimate from the example, the high-completion approach’s effective cost was on the order of $40 per passed task, while the low-cost approach’s was on the order of $2. The former pays for quality and validation depth on complex tasks; the latter trades a lower budget for faster throughput.

    3. Two Approach Profiles: High-Completion, Long-Running and Low-Cost, Fast Are Not Simply Better or Worse

    High-completion, long-running approaches are typically more willing to perform multi-round planning, cross-file modification, execution validation, and failure repair, so they are more likely to approach a deliverable state on complex tasks. Their advantage is reducing the number of manual interventions, making them suitable for high-risk tasks such as core business logic, data migration, and permission- or payment-related changes. The trade-offs are long wait times, high concurrency occupancy, high per-run cost, and volatility in long-horizon tasks, where retries can further increase budget.

    Low-cost, fast approaches are characterized by shorter feedback cycles and lower budgets. They suit tasks with clear requirements, short validation paths, and easy rollback, such as scaffold generation, style adjustments, low-risk page completion, initial screening and localization, and simple subtasks within batch migration. Their pass rate and completion may be lower than those of high-completion approaches, but if used in the front stage of a pipeline to “run once and expose problems quickly,” they can improve overall throughput.

    Engineering decisions should therefore avoid a “pass-rate-only” view. A more reasonable approach is to tier tasks by risk level, cross-module scope, acceptance automation, and budget constraints: high-risk tasks use high-quality modes with human review retained; low-risk tasks use low-cost modes with automated testing; intermediate tasks use balanced modes or staged execution.

    4. Implementation: Build a Closed Loop with Task Tiering, Effective Cost, and Gray-Release Admission

    First, build a tiered task sample set. Divide evaluation tasks into smoke, cross-file, migration, and regression tiers. The smoke tier focuses on whether a runnable skeleton can be generated quickly; the cross-file tier focuses on API contracts and state consistency; the migration tier focuses on dependency replacement and behavioral compatibility; the regression tier focuses on whether existing tests pass. Different tiers should use different approaches so that a single average score does not hide structural differences.

    Second, collect complete metrics rather than only final results. In addition to pass rate, completion, duration, and cost, teams should record retry counts, timeout counts, manual intervention time, test failure types, and artifact mergeability. Samples with high completion but low pass rate deserve particular analysis, as they often reveal missing acceptance scripts, environment dependencies, or key validation steps.

    Third, use effective cost for budget decisions. Effective cost = total run cost / number of passed tasks. For long-running approaches, queue wait time and concurrency resources should also be factored in; for low-cost approaches, manual rework cost should be included. Only when rework, review, and reruns are included does cost comparison approach real production conditions.

    Fourth, adopt gray-release admission and routing strategies. A low-cost approach can first handle initial screening and simple subtasks, then route failed samples or high-risk tasks to a high-completion approach. Set quality gates before release: core paths must pass automated tests and human review, while non-core paths may accept lower completion depending on risk. Use gray releases to continuously calibrate routing thresholds.

    Fifth, build an observable evaluation pipeline. In a Baidu AI Cloud environment, teams can use general categories such as container platforms, object storage, logging, and monitoring/alerting to build a reproducible chain for task execution, artifact archiving, and metric collection; specific specifications, quotas, and costs should be evaluated against real workloads. The evaluation environment should isolate code and artifacts, follow the principle of least privilege, and retain task inputs, run logs, and acceptance results for reproducibility and auditing.

    5. Illustrative Pseudocode: Routing and Effective Cost Estimation

    1. # Task routing and effective cost estimation (illustrative)
    2. def effective_cost(total_cost, task_count, pass_rate):
    3. passed = max(1, task_count * pass_rate)
    4. return total_cost / passed
    5. def route_task(task):
    6. if task.risk_level == "high" or task.cross_module:
    7. return "high_completion_mode"
    8. if task.budget_sensitive and task.fast_feedback:
    9. return "low_cost_fast_mode"
    10. return "balanced_mode"

    The pseudocode expresses two principles: first, amortize cost over passed tasks instead of looking only at per-run price; second, route by risk and feedback speed instead of sending every task through the same mode. In practice, teams should also add timeout budgets, maximum retry counts, and human-review thresholds.

    6. Limitations and Conclusion: Benchmarks Are Decision Inputs, Not Production Conclusions

    Public benchmarks cannot fully represent production environments. Task distribution, dependency versions, acceptance scripts, and agent orchestration all affect results; the duration and cost of long-horizon tasks may also fluctuate significantly with retry strategies. High completion does not mean code is mergeable, and a high pass rate does not mean the architecture is sound. Teams should reproduce small samples locally, evaluate continuously, and record version differences.

    Overall, high-completion, long-running approaches are an investment in quality and validation depth for complex tasks, while low-cost, fast approaches are a tool for improving throughput and controlling budgets. The rational approach is to build a closed loop of task tiering, effective cost accounting, gray-release admission, and observable evaluation so that different approaches can play appropriate roles, rather than replacing engineering judgment with a single leaderboard ranking.