Mobile Coding Agent Benchmarks: Balancing Completion Rate and Cost
This article explains how to read coding agent benchmark results for mobile app development. It covers four metrics: pass rate, average completion, duration, and cost per run. It compares high-completion and low-cost approaches, then offers a four-layer framework for choosing an agent based on task type, quality gates, total cost, and observability.
1. What Benchmark Results Actually Tell Us
Coding agent benchmarks for mobile app development are usually not algorithm puzzles. They are long-horizon tasks that resemble real engineering work: scaffolding an app, building out pages and interactions, connecting a data layer, migrating legacy code, restructuring a project, and fixing build and test failures. These benchmarks test how well an agent plans, edits, verifies, and repairs its own work in a complex context. Therefore, reading the results should not focus on a single pass rate. Four dimensions matter:
- Pass rate: the share of tasks where all acceptance items pass. It reflects the hard delivery threshold.
- Average completion: the coverage of task goals and acceptance points. It shows how usable a partial result is.
- Average duration: the time from task start to a verifiable result, including planning, generation, execution, testing, and retries.
- Cost per run: the model calls, sandbox execution, and infrastructure consumed by one benchmark run.
Together, these four metrics define the constraints of an engineering decision. Looking only at pass rate ignores cost and waiting time. Looking only at cost may shift rework and manual review costs to later stages.
2. What a High-Completion, Long-Duration Approach Implies
Publicly disclosed results show one recurring pattern: a pass rate of about 20%, average completion above 80%, average duration over one day, and a cost of hundreds of US dollars per run. This suggests the agent keeps its goals and checks its own work well over long tasks. It can push many subtasks close to completion. The trade-off is longer feedback cycles and higher model call costs.
When does this pattern fit? First, when the task tolerates errors well, such as batch code migration, project structure cleanup, test coverage expansion, and documentation refactoring. Second, when manual intervention is expensive and the team prefers a near-complete draft followed by focused review. Third, when rerunning a failed task is costly, so it is better to make each run more thorough and reduce repeated rework.
The risks are also clear. If the benchmark tasks do not match the real business workload, high completion may only reflect overfitting to the acceptance points. If the duration exceeds the team’s feedback rhythm, agent output can block later reviews.
3. Where Low-Cost, Fast Approaches Fit
Another recurring pattern: a pass rate of about 17%, average completion around 70%, average duration of half a day, and a cost on the order of ten US dollars per run. This is not simply worse. It reduces resource input to get faster iteration and lower trial-and-error costs. For teams sensitive to budget and response time, this approach suits prototype exploration, suggestions for local changes, candidate patch screening, and batch processing of low-risk tasks.
However, a low-cost, fast approach is not production-ready by default. An average completion of about 70% means many boundary conditions, exception paths, and platform adaptations still need manual work. A low pass rate means that generated output is not the same as delivered output. A sound approach places this method at the front of the funnel: generate several candidates cheaply, filter them with automated tests and static checks, then apply a higher-completion approach to a small set of high-value tasks.
4. A Four-Layer Framework from Leaderboard to Selection
Layer 1: Task stratification. Divide development work into levels such as scaffolding, local feature changes, cross-module migration, complex defect fixing, and performance and compatibility governance. Each level has different requirements for pass rate, completion, and latency. A single overall score should not drive every decision.
Layer 2: Quality gates. Define non-negotiable acceptance conditions first. For example, the build must pass, core test cases must pass, static checks must show no blocking issues, and critical paths must show no regressions. Pass rate is the gate. Completion is the basis for ranking. The order should not be reversed.
Layer 3: Cost model. Cost per run is only the visible cost. Manual review, reruns of failed tasks, environment waiting time, code review, and rework time must also be counted. The following pseudocode aggregates evaluation records into comparable engineering metrics:
def run_bench(agent, task_set, budget):records = []for task in task_set:trace = agent.execute(task.prompt, timeout=task.timeout)report = task.verify(trace.patch)records.append({"task": task.id,"pass": report.all_required_checks_pass,"completion": report.completion_ratio,"duration_h": trace.duration_h,"cost": trace.cost})return summarize(records, budget)
Layer 4: Observability and regression. Each evaluation should keep task inputs, execution traces, test reports, cost, and duration, forming a traceable record. At the infrastructure level, teams can use general-purpose services on Baidu AI Cloud, such as object storage, container platforms, log services, and monitoring and alerting, to build a repeatable evaluation pipeline. The tools are only the vehicle. What matters is whether the task set, the acceptance checkers, and the regression mechanism reflect real engineering work.
5. Rollout Recommendations and Risk Control
When introducing coding agent benchmarks, teams should first build a small, verifiable internal task set and expand it gradually. Do not treat public leaderboards as direct conclusions for production selection. Instead, calibrate three things: task distribution, acceptance checkers, and cost. For long-duration, high-completion approaches, set staged checkpoints to avoid long waits. For low-cost, fast approaches, set automated gates to keep low-quality patches out of the main branch.
Ultimately, completion, pass rate, duration, and cost are not four isolated metrics. They form a set of engineering trade-offs. High-completion, long-duration approaches suit high-value tasks that can wait and where manual review is costly. Low-cost, fast approaches suit exploration, screening, and budget-sensitive scenarios. Mature selection does not mean finding the top leaderboard entry. It means finding a combination strategy that matches the team’s task structure, delivery rhythm, and risk tolerance.
