Mobile Agent Coding Benchmarks: How to Balance Capability, Time, and Cost

    A practical guide to mobile agent coding benchmarks. It explains why evaluation has shifted from function-level tasks to complete engineering tasks, breaks down four core metrics (pass rate, average completion, average runtime, and per-run cost), and shows how to convert them into per-task economics such as cost per successful task and completion per hour. It also covers the mechanisms behind capability gaps, a five-step framework for building reproducible evaluations, a constraint-based model-selection framework, common pitfalls, and risk-control practices.

    1. The Evaluation Focus Is Shifting: From “Writing One Correct Piece” to “Finishing the Whole Task”

    Historically, coding-model capability was measured mostly on function-level, offline tasks: completing an implementation, fixing a bug, or making a set of unit tests pass. These tasks have clear boundaries and unambiguous pass/fail criteria, but they fail to answer one critical question: when requirements are relatively vague, changes span multiple files, and correctness can only be confirmed through build and test feedback, can an agent carry a task all the way to completion?

    Mobile app development is an ideal scenario for testing long-horizon capability, for three reasons:

    • Long requirement chains: a single feature often runs through the UI layer, the business-logic layer, and the data layer, and any broken link makes the whole thing unusable;
    • Complex environments: build toolchains, dependency versions, and differences across devices and OS versions stack up, so builds fail for countless reasons;
    • Verifiable outcomes: whether it compiles, whether tests pass, and whether the output matches expectations all have relatively clear acceptance criteria.

    Based on this reasoning, agent coding benchmarks aimed at real engineering tasks have emerged. Their tasks cover areas such as app creation, feature development, code migration, and platform conversion. What they measure is the ability to complete real engineering work, not the correctness of isolated snippets of code.

    For engineering teams, the value of these benchmarks goes beyond a leaderboard. More importantly, they provide a reusable measurement framework: how tasks are designed, how success is judged, how time is counted, and how cost is accounted for. Only by understanding the relationships among these four metrics can a team turn “which agent is stronger” into “which agent is a better fit for our current team.”

    2. Reading the Results: No Metric Can Be Skipped

    In recent evaluation results, two technical approaches show a sharp contrast:

    Metric Route A: High-completion Route B: Lightweight and fast
    Pass rate ~24.0% ~17.3%
    Average completion ~82.3% ~71.2%
    Average runtime ~27.4 hours ~12.0 hours
    Cost per benchmark run ~$317 ~$9.9

    1. Pass Rate: Was It “Actually Done”

    The pass rate is the share of tasks that meet all acceptance criteria. It is the strictest outcome metric. Its meaning is that the code not only runs but also satisfies every constraint on functionality, compatibility, and quality. Route A’s pass rate is about 6.7 percentage points higher than Route B’s. With a sample of 30 tasks, the relative improvement is around 39%, but the statistical significance of the gap still needs to be judged alongside the task-difficulty distribution and results from repeated runs.

    2. Average Completion: Measuring “How Close It Got”

    Average completion reflects, on average, how large a fraction of the work the agent finished on each task, including tasks that did not pass all acceptance checks. It is an important window into the capability ceiling: even though Route B has a lower pass rate, its average completion still reaches about 71.2%. This shows that many failures are not because the agent “had no idea where to start,” but because it got stuck on a specific step near the end, such as one build-configuration error or one missed edge case. For teams, these “one step away” tasks are the most valuable, because a single targeted fix can turn them into usable output.

    3. Average Runtime: The Real Time Cost

    Long-horizon tasks often run for hours, and runtime directly determines whether an agent can fit into a daily development rhythm. Runtime depends not only on the model’s raw inference speed but also on orchestration strategy:

    • whether it uses a multi-round “build → test → fix” loop;
    • whether it performs retrospective analysis on failures;
    • whether it allocates sufficient execution time to long tasks.

    Route A’s average runtime is about 2.3 times Route B’s, which closely matches its “verify thoroughly, fix repeatedly” approach.

    4. Cost per Run: Can It Be Used at Scale

    The cost per run is the inference and execution cost of completing one round of the benchmark. The per-run cost differs by about 32x, the largest gap among the four metrics. However, per-run cost reflects the price of “trying once,” not the price of “finishing a task.” To answer the latter, you must combine it with the pass rate.

    3. From Per-Run Cost to Per-Task Cost: Redefining “Cost-Effectiveness”

    Imagine a team needs an agent to complete 100 similar tasks. The difference in per-run cost only becomes decision-relevant once it is converted to “per successful task.” Define:

    • S: cost per run
    • P: pass rate
    • C: average completion
    • T: average runtime

    This yields four derived metrics:

    1. Cost per successful task: S / P.
    2. Expected completion: P × C. If failed tasks are completely unusable, the overall average output can only reach “pass rate × completion.”
    3. Cost per unit of expected completion: S / (P × C).
    4. Expected completion per hour: (P × C) / T, suitable for scenarios sensitive to delivery timelines.

    A worked example:

    1. def efficiency(single_run_cost, pass_rate, completion, hours):
    2. """Derive per-task economics from per-run metrics."""
    3. return {
    4. "cost_per_passed_task": single_run_cost / pass_rate, # cost per successful task
    5. "expected_completion": pass_rate * completion, # expected completion
    6. "cost_per_expected_completion": single_run_cost / (pass_rate * completion),
    7. "completion_per_hour": (pass_rate * completion) / hours, # expected completion per hour
    8. }
    9. # Route A (high-completion) example
    10. efficiency(317.4, 0.240, 0.823, 27.4)
    11. # Route B (lightweight and fast) example
    12. efficiency(9.9, 0.173, 0.712, 12.0)

    The results are summarized below:

    Derived metric Route A Route B
    Cost per successful task ~$1,323 ~$57
    Expected completion ~19.8% ~12.3%
    Cost per unit of expected completion ~1,607 ~80
    Expected completion per hour ~0.72% ~1.03%

    Three takeaways:

    1. The absolute cost gap narrows. Switching from “per-run cost” to “cost per successful task” shrinks the gap from about 32x to about 23x. The high-cost route is not wasted money; its higher pass rate partly offsets the expensive price.
    2. The gap is even smaller on an expected-value basis. If failed tasks can only be partially reused, the measure becomes “cost per unit of expected completion,” and the gap narrows further to about 20x. This is directly tied to Route B’s ~71% average completion: even when it does not fully pass, a large amount of intermediate output can still be used.
    3. “Cheaper” depends on the constraint. On pass rate alone, the high-completion route has an advantage of about 39%; but measured by “expected completion per hour,” the lightweight route is actually slightly higher. The reason is that the lightweight route takes only about half the time. So there is no single answer to “which route is more cost-effective.” The answer depends on which constraint is tightest among budget, timeline, and quality.

    4. Where the Capability Gap Comes From: Three Mechanism-Level Reasons

    External metrics alone make it hard to pinpoint the cause of the difference, but the characteristics of long-horizon execution allow us to distill three key factors.

    1. Depth of the Verification Loop

    In long-horizon tasks, the first generated code almost never passes on the first try. The gap often comes from the ability to “self-correct”:

    • High-completion route: tends to build, run tests, analyze failure logs, locate root causes, and revise again, until acceptance criteria are met;
    • Lightweight route: relies more on a single high-quality generation, with a shorter verification loop.

    Multi-round loops raise the success rate, but they also stack up runtime and cost.

    2. How Failures Are Handled

    Faced with a build error or a test failure, a fundamental question is: patch locally, or re-plan?

    • Local patching: small, low-cost changes, but it easily falls into a cycle of repeated trial and error, where a single patch gradually turns into multiple rounds of refactoring;
    • Re-planning: requires retaining enough task context and re-evaluating the overall solution; the thinking cost is higher, but it often resolves the root cause in one shot.

    The two routes make different trade-offs here, which is reflected directly in the combination of runtime and completion.

    3. Task Planning and Execution Granularity

    Long-horizon tasks must be decomposed into several verifiable sub-goals. A sensible planning order is usually: first build a runnable skeleton, then implement feature modules one by one, and finally handle compatibility adaptation and detail polishing. If planning granularity is too coarse, errors surface all at once later, making rework scope large; if it is too fine, it increases the number of calls and context consumption. This explains why differences in planning strategy ultimately show up across all four dimensions: pass rate, completion, runtime, and cost.

    5. Building Reproducible Engineering-Task Evaluations: A Practical Guide

    External benchmarks provide reference coordinates, but every team’s tech stack and business scenario differ. Building your own evaluation system is the only way to make choices that closely match reality. An evaluation system can be broken into five stages.

    1. Task-Set Design

    • Cover multiple scenario types: creating an app from scratch, adding a feature, fixing a bug, migrating code, converting platforms. Different tasks place very different demands on capability.
    • Set machine-verifiable acceptance criteria: layer the gates with static checks, compile/build, unit tests, integration tests, and compatibility checks, avoiding the subjective judgment of “looks done.”
    • Record difficulty labels: this makes it easy to compute stratified statistics by difficulty and avoid overall averages masking structural differences.

    2. Environment Isolation and Reproducibility

    • Prepare a separate sandbox environment (such as a container) for each task run, with fixed base images and dependency versions.
    • Snapshot the initial workspace before running, and roll back with one click after a failure, so that every run starts from the same point.
    • Restrict network access to avoid introducing uncontrolled factors during execution.

    3. Observability and Cost Accounting

    • Record build logs, test reports, the agent’s step-by-step action trace, as well as token usage, tool-call counts, and resource consumption.
    • The value of this data shows up in three ways: it enables post-mortem root-cause analysis on failure, it provides a basis for horizontal comparison, and it makes cost traceable to specific tasks.

    4. Statistical and Analysis Methods

    • Long-horizon task results are stochastic; repeated runs of the same agent can fluctuate noticeably, so run multiple times and report both the mean and the interval.
    • When comparing different agents, use paired comparisons with the same task set and the same run budget whenever possible.
    • Beyond overall averages, also provide distributions by scenario and by difficulty to identify capability weak spots.

    5. Sample Code: Task Execution and Metric Aggregation

    1. from dataclasses import dataclass
    2. @dataclass
    3. class TaskResult:
    4. passed: bool # whether all acceptance criteria passed
    5. completion: float # completion, in the range 0 to 1
    6. hours: float # runtime (hours)
    7. cost: float # cost of this run
    8. def run_task(task, agent, sandbox, budget):
    9. """Run a single task in an isolated environment and return a structured result."""
    10. # 1. Restore a clean initial workspace from the snapshot
    11. workspace = sandbox.restore(task.snapshot_id)
    12. # 2. The agent completes the task under budget constraints
    13. trace = agent.solve(task.description, workspace,
    14. max_hours=budget.hours, max_cost=budget.cost)
    15. # 3. Check each acceptance item and compute completion
    16. checks = [check(trace) for check in task.acceptance_checks]
    17. return TaskResult(
    18. passed=all(checks),
    19. completion=sum(checks) / len(checks),
    20. hours=trace.elapsed_hours,
    21. cost=trace.cost,
    22. )
    23. def summarize(results):
    24. """Aggregate per-run metrics and derive per-task economics."""
    25. n = len(results)
    26. pass_rate = sum(r.passed for r in results) / n
    27. completion = sum(r.completion for r in results) / n
    28. hours = sum(r.hours for r in results) / n
    29. cost = sum(r.cost for r in results) / n
    30. return {
    31. "pass_rate": pass_rate,
    32. "completion": completion,
    33. "avg_hours": hours,
    34. "avg_cost": cost,
    35. "cost_per_passed_task": cost / pass_rate if pass_rate else None,
    36. "expected_completion": pass_rate * completion,
    37. "completion_per_hour": pass_rate * completion / hours if hours else None,
    38. }

    6. Selection Advice: Choose a Route by Constraints, Not by Ranking

    The point of benchmark rankings is to help teams find the option that best matches their own constraints. The following decision framework can be used as a reference:

    Team constraint Suggested preference Rationale
    Limited budget, tasks are splittable, retries are acceptable Lightweight and fast route Low per-task cost, plenty of room to experiment
    High value per delivery, high rework cost High-completion route Longer runtime and cost can be seen as an investment in delivery quality
    Tight delivery timeline, manual review allowed Prefer lightweight route, escalate failed tasks Produce drafts quickly and shorten the overall delivery cycle
    Mostly migrating and converting existing code Pilot on a small scale before scaling out Such tasks carry many legacy constraints; stability and controllability must be validated

    The sample rule below turns constraints into a selection logic:

    1. def choose_route(budget_per_task, delivery_hours, need_verified_delivery):
    2. # Hard delivery deadline is tight: prioritize speed, leave output to manual review
    3. if delivery_hours <= 24:
    4. return "lightweight"
    5. # A fully verified result must be delivered, and budget can cover multiple repair rounds
    6. if need_verified_delivery and budget_per_task >= 100:
    7. return "thorough"
    8. # Otherwise: start lightweight, then escalate failed tasks
    9. return "lightweight_then_escalate"

    7. Common Pitfalls and Risk Control

    1. Judging only by pass-rate ranking. Ignoring cost and runtime may lead to choosing the “strongest but unaffordable” option; looking only at per-run cost may underestimate the hidden rework cost of failures.
    2. Drawing conclusions from a single run. Long-horizon results fluctuate significantly, and a single run is not enough to represent the true level.
    3. Ignoring benchmark-set contamination. If tasks or code already appear in the model’s training data, results will be inflated. A self-built task set should retain a degree of privacy and a regular update cadence.
    4. Treating benchmark scores as business outcomes. A gap exists between benchmark tasks and a team’s real needs, so run a small-scale validation on the real codebase before going live.
    5. Lacking cost guardrails. Long-horizon tasks can fall into “dead-loop” style retries, so set timeouts, budget caps, and automatic circuit breakers to prevent costs from spiraling out of control.

    8. Conclusion

    Long-horizon task evaluation reveals a simple engineering truth: capability, time, and cost form a mutually constraining “triangle,” and no option wins on every dimension. The high-completion route trades more time and money for a higher pass rate and completion, while the lightweight route covers a broader space of attempts with lower cost and faster turnaround. Understanding the conversion relationships among pass rate, average completion, runtime, and cost is more valuable than remembering any single ranking.

    For teams that want to build their own evaluation system, cloud-based, on-demand composable building blocks can lower the barrier to getting started:

    • containerized, isolated execution environments;
    • object storage for archiving run traces and reports;
    • logging services and monitoring alerts for observing task-execution status and catching anomalies promptly.

    On Baidu AI Cloud, teams can select general-purpose cloud product capabilities such as containers, object storage, logging services, and monitoring alerts as needed to build their own agent coding evaluation pipeline. For example, they can start task sandboxes on demand with elastic compute, preserve each run’s trace in object storage, and set budget alerts to control evaluation costs. Availability and access of these services should be confirmed against the official documentation for each region.

    The endpoint of evaluation is not the leaderboard, but making it clear to the team: under the constraints of a given budget and timeline, which option can deliver the most reliable result at an acceptable cost.