AI Coding Benchmarks: Balancing Completion, Time, and Cost

    A mobile development platform recently published the results of an AI coding benchmark that pits coding agents against real-world mobile app tasks—from scaffolding an app and building features to code migration and app conversion. This article explains how to read the four metrics that matter (pass rate, completion, time, and cost), why mobile development is a demanding proving ground for agents, and what the tradeoffs mean in practice. Comparing a high-completion, long-runtime agent with a low-cost, fast one, it shows why single-run cost is misleading, how to estimate the true total cost including retries and human fixes, and how teams should route tasks and add budget guardrails to turn benchmark scores into real productivity.

    Introduction: Beyond the Leaderboard Hype

    The official team behind a mobile development platform recently published the results of an AI coding benchmark, and the findings sparked broad discussion in the community. This kind of benchmark targets real mobile app development scenarios, asking agents to autonomously complete end-to-end tasks such as creating apps, building features, migrating code, and converting applications. The gap between agents shows up not only in whether a task ultimately passes acceptance, but also in the time and money spent to get there.

    The more interesting story is the contrast between two agents:

    Profile Pass rate Average completion Average time Cost per run (illustrative)
    High-completion, long-runtime 24.0% 82.3% ~27 hours ~$317
    Low-cost, fast 17.3% 71.2% ~12 hours ~$9.9

    The second agent has a clearly lower pass rate than the first, yet its several-fold speed advantage and roughly 30x cost advantage make it an important option for teams that are sensitive to budget and response time. So what engineering tradeoff does the combination of high completion and long runtime actually reflect, and how can a leaderboard’s numbers be turned into decisions a development team can act on? This article breaks the question down across four layers: task characteristics, what the metrics actually mean, cost structure, and selection strategy.

    Why Mobile Development Is a Demanding Test Track for AI Coding

    Mobile long-horizon tasks challenge agents for five main reasons.

    1. Long task chains with many steps. A complete task runs from creating the project skeleton, configuring build scripts, and writing UI and business logic, through passing compilation, running tests, and packaging an installable app. That’s more than a dozen sequential steps, and a failure at any one of them can sink the whole task.
    2. Slow, expensive feedback. Building and testing mobile apps depends on a real toolchain—cold starts, dependency resolution, compilation, and tests can each take minutes. Every trial-and-error cycle an agent runs consumes real wall-clock time, so the cost of mistakes is far higher than for tasks that run lightweight scripts.
    3. Strong error propagation. Errors in earlier steps often surface only at the final stage. For example, an early interface signature change can trigger compilation failures across many call sites, and a missing resource may not be discovered until packaging. Both are costly to diagnose and fix.
    4. Frequent tool calls that depend on a real environment. Agents must switch repeatedly between the file system, build tools, and simulators or test frameworks. Whether the execution environment is stable and whether tool output is clear directly affect whether the task can move forward.
    5. Relatively clear acceptance criteria. Whether an app compiles, whether tests pass, and whether it packages successfully can all be judged automatically. This makes results relatively objective and lets developers carry the benchmark approach over to their own projects.

    It’s worth noting that mobile projects involve large project structures, build scripts, and many resource files, which quickly fill an agent’s context window. Whether code retrieval is precise, whether dependency analysis is correct, and whether irrelevant information can be filtered out often become the deciding factors.

    Reading the Results: Four Metrics, Four Different Questions

    The four metrics on a leaderboard answer different questions, and understanding that is the prerequisite for interpreting results correctly.

    Metric How it’s measured Question it answers What to watch
    Pass rate Tasks passing acceptance ÷ total tasks Can the agent actually complete and deliver the task Final delivery capability
    Average completion Average completion ratio of unfinished tasks How far from success the failed tasks were Size of the failure gap
    Average time Average wall-clock time per task How long it takes to reach the goal Error-correction efficiency and environment performance
    Cost per run Cost to complete one evaluation run How expensive reaching the goal is Budget feasibility at scale

    Pass rate answers “can it do it.” It is a strict outcome metric that usually requires a task to pass all acceptance conditions. One agent achieved a 24.0% pass rate. That number looks modest, but given the difficulty of long-horizon tasks, it reflects fairly strong end-to-end delivery capability. The other agent’s 17.3% pass rate means roughly eight in ten tasks can’t be completed independently and need human intervention or must be abandoned.

    Completion answers “how far off.” It measures task progress and is closer to how developers actually feel about a run. One agent may have finished the UI, data integration, and most of the business logic, missing only final compatibility adjustments or resource cleanup; another may have gone in the wrong direction back at the requirements stage. The first might sit at 90% completion and the second at just 20%. These two kinds of failure mean completely different things to a developer: the first is close to usable and needs only minor manual fixes, while the second basically has to be redone from scratch.

    Time answers “how long.” Wall-clock time includes not only the agent’s inference time but also the waiting time for tools like build and test execution. Long runtime can come from many reasoning rounds or slow context processing, or from a slow toolchain or repeated build failures—so it needs to be diagnosed against run logs.

    Cost answers “is it worth it.” Cost per run is usually made up of model call fees, execution environment fees, and human review cost. In agent mode, model call fees tend to grow linearly with the number of rounds, so long-running tasks easily rack up large bills.

    The four metrics need to be read together: high completion doesn’t mean a high pass rate, and a high pass rate doesn’t mean low cost. Some agents finish most of the work but get stuck at the final acceptance step; others have a mediocre pass rate but complete a large number of tasks in very little time and at very low cost.

    The Core Insight: The Engineering Tradeoff of High Completion vs. Long Runtime

    The two agents in the table above represent two typical strategy orientations in long-horizon coding tasks.

    High-completion, long-runtime: trading time for completion. These agents tend to keep exploring and re-verifying until the task is as close to done as possible. Their strengths:

    • Resilience: when a build fails, dependencies conflict, or interfaces don’t match, they can analyze the error and adjust the approach rather than giving up early;
    • Near-usable completion: even when they ultimately fail acceptance, the deliverable is often close to a deliverable state, so the human fix cost is low;
    • Suited to complex migration and refactoring: code migration and app conversion tasks involve many file changes and compatibility handling, which require an agent that can push forward persistently.

    But the costs are equally clear:

    • High time cost: an average runtime of over twenty hours means evaluating a single task can take an entire day, which rules it out for day-to-day development that needs quick feedback;
    • High cost: a few hundred dollars per run makes large-scale use uneconomical;
    • Heavy resource usage: long runs tie up large amounts of compute and need a stable execution environment to support them.

    Low-cost, fast: trading efficiency for completion. These agents tend to produce a solution quickly and verify it quickly, finishing within a limited number of rounds. Their strengths:

    • Speed: an average runtime of about 12 hours, more than twice as fast as the high-completion profile;
    • Very low cost: under ten dollars per run, suitable for large-scale batch tasks;
    • Timely responses: friendlier to budget-limited teams that need fast iteration.

    But the limitations are just as obvious:

    • Low pass rate: about eight in ten tasks can’t be completed independently and need human intervention;
    • Insufficient completion: an average completion of about 71% means many tasks are only half to mostly done;
    • Struggles on complex tasks: it does relatively poorly on hard tasks like code migration and app conversion.

    From an engineering standpoint, the difference between these two agents is essentially the difference between a “depth-first” and a “breadth-first” strategy. The former is willing to invest more time and resources in a single task to chase the highest possible completion; the latter aims to complete more tasks within limited time and cost. Neither is absolutely better—the key is the task type and the team’s constraints.

    Calculating Total Cost: Retries and Human Effort Are Often Underestimated

    When evaluating cost, comparing only “cost per run” isn’t enough. The true total cost of a real project can be roughly expressed as:

    1. Total cost ≈ agent run cost + retry cost + human fix cost + latency cost
    • Agent run cost: cost per run multiplied by the actual number of task runs.
    • Retry cost: a low pass rate means many tasks need multiple attempts. The low-cost, fast agent is cheap per run, but if a task needs three or more attempts on average, the cumulative cost can exceed a high-completion, long-runtime agent’s approach that succeeds close to the first try.
    • Human fix cost: insufficient completion translates into engineers’ fix time. The high-completion agent’s output is closer to a deliverable state, so the human intervention cost may be lower.
    • Latency cost: for teams that need fast iteration, waiting twenty-plus hours for a result can cost far more than the model call fees saved.

    You can estimate cost-effectiveness with a simple set of ratios (all values are illustrative):

    Profile Average time Cost per run Completion per hour Completion per dollar
    High-completion, long-runtime 27.4 hours $317.4 ~3.0 ~0.26
    Low-cost, fast 12.0 hours $9.9 ~5.9 ~7.19

    By unit cost, the low-cost, fast agent is about 28x more cost-effective than the high-completion, long-runtime one; by unit time, it also has nearly twice the output efficiency. But this conclusion assumes “tasks can be retried and human fix cost is controllable.” If a task can run only once, or failure is very costly—such as a production data migration—then pass rate and completion matter far more than the cost-effectiveness ratios.

    Selection Strategy: There’s No Best Agent, Only the Right Configuration

    Based on the analysis above, development teams can choose agent configurations by task type.

    Prefer the high-completion, long-runtime profile for:

    • Code migration and app conversion: many file changes and compatibility handling require persistent progress;
    • Legacy system refactoring: complex tasks with a high cost of failure, where you want to get close to done in one pass;
    • Critical-path development: feature modules that directly affect core business, where quality requirements are high;
    • Offline batch processing: tasks that can run for a long time, aren’t on the critical path, and are not time-sensitive.

    Prefer the low-cost, fast profile for:

    • Day-to-day feature development: clear requirements and relatively simple tasks that call for fast iteration;
    • Prototyping and experiments: situations that need fast trial and error where the cost of failure is low;
    • Large-scale batch tasks: for example, generating test cases in bulk or fixing low-risk issues in bulk;
    • Budget-limited teams: those that want to complete as many tasks as possible within a limited budget.

    A hybrid strategy is often better. For complex projects, you can adopt a layered approach: use the low-cost, fast agent for simple tasks and the high-completion, long-runtime agent for complex ones, routing tasks to balance cost and quality.

    1. # Example: route to different agent profiles based on task complexity
    2. COMPLEXITY_KEYWORDS = {"migrate", "refactor", "convert", "core"}
    3. HIGH_PROFILE_BUDGET = 350.0 # Per-task budget cap for the high-completion profile (currency units)
    4. LOW_PROFILE_BUDGET = 12.0 # Per-task budget cap for the low-cost profile (currency units)
    5. MAX_RETRY_SAME_ERROR = 2 # Max retries when the same error repeats
    6. def select_profile(task):
    7. """Select an agent profile from the task description; tune thresholds and keywords per team."""
    8. text = task.description.lower()
    9. if any(k in text for k in COMPLEXITY_KEYWORDS) or task.scope == "core":
    10. return {"name": "deep", "budget": HIGH_PROFILE_BUDGET}
    11. return {"name": "fast", "budget": LOW_PROFILE_BUDGET}
    12. def should_stop(error_history, spent):
    13. """Stop repeated trial-and-error to avoid unproductive loops."""
    14. if error_history[-1] in error_history[:-1]:
    15. return error_history.count(error_history[-1]) > MAX_RETRY_SAME_ERROR
    16. return spent >= HIGH_PROFILE_BUDGET

    Putting this into practice also calls for two supporting mechanisms. The first is budget guardrails: set a cost cap and a runtime cap for each task, and abort and hand off to a human as soon as either is hit, so an abnormal task can’t consume resources indefinitely. The second is a failure circuit breaker: stop promptly when the same error repeats consecutively, so the agent doesn’t get stuck in an unproductive loop. Together, these two mechanisms keep uncontrollable long runtimes and high costs “caged.”

    Building Your Own Long-Horizon Eval: Port the Leaderboard Approach to Your Codebase

    The task set on a leaderboard differs from a real project’s tech stack, so a more reliable approach is to build your own evaluation around your own codebase.

    1. Extract tasks from real requirements. Start with requirements that were completed in the past and have clear acceptance criteria—for example, fixing a certain class of defect, adding a feature module, or migrating an old component to a new framework.
    2. Establish automatically judgeable acceptance criteria. A successful compile, passing unit tests, and passing smoke tests on key paths can all be judged by script, reducing the cost of manual evaluation.
    3. Record all four metrics. Beyond pass rate, also record completion, time, and cost so you can fully assess different agents.
    4. Control variables. Fix the execution environment, dependency versions, and acceptance scripts so every agent is evaluated under the same conditions.
    5. Analyze results in layers. Look at the overall pass rate first, then analyze the distribution of failure causes to find common bottlenecks.
    6. Combine evaluation with governance. Feed results back into team processes to optimize agent configuration, the execution environment, and the human review step.

    Building your own eval can draw on the container platform on Baidu AI Cloud to quickly set up isolated, reproducible execution environments, use object storage to centrally manage build artifacts and run logs, and track task status and pinpoint failure causes through log service and monitoring/alerting capabilities. This substantially improves the comparability and traceability of evaluation results.

    Conclusion: Read AI Coding Benchmarks Through an Engineering Lens

    The value of an AI coding benchmark lies not in the rankings themselves, but in the way it reveals agents’ capability boundaries and cost structures on real engineering tasks. Pass rate, completion, time, and cost together form a multi-dimensional evaluation space. Without the task scenario and the team’s constraints, comparing any single metric in isolation can easily lead to misleading conclusions.

    For development teams, the more pragmatic approach is to: understand the engineering tradeoffs behind different agent configurations, calculate the total cost including retries and human intervention, choose the right configuration by task type, and keep risk within an acceptable range using budget guardrails and a failure circuit breaker. On that foundation, building a continuous evaluation mechanism around your own codebase is what truly turns benchmark scores into productivity.

    On Baidu AI Cloud, teams can use general cloud capabilities such as a container platform, object storage, a log service, and monitoring/alerting to build isolated, reproducible, and observable agent evaluation and runtime environments, applying an engineering approach so that every bit of investment can be justified.