Mobile Development Benchmarks Decoded: Efficiency Trade-offs of Long-Horizon Coding Agents
This article explains how to read mobile development coding benchmarks for long-horizon coding agents. It breaks down the multi-step engineering loop, argues that pass rate, average completion, wall-clock time, and cost per run must be interpreted together, and examines why high completion with long wall-clock time can be a deliberate convergence trade-off. It also identifies three sources of bias when applying public leaderboard results to internal projects, proposes a reproducible CI evaluation setup with declarative task cards and weighted scoring, and outlines cost governance through profile routing, budget gates, checkpointing, context budgeting, fast failure feedback, and least-privilege sandboxes. Finally, it describes how cloud capabilities can support the pipeline and why evaluation should become part of CI.
1. What Is Actually Being Evaluated?
A mobile operating system’s official developer account recently published a batch of coding benchmark results for application-development scenarios. These results cover several mainstream large models across tasks such as app creation, feature development, code migration, and cross-platform app conversion. This kind of evaluation is fundamentally different from a typical “algorithm problem benchmark.” It does not measure correctness in a single question-answer round. Instead, it measures whether an agent can advance a multi-step task in real repositories, real build pipelines, and real time scales.
Translated into engineering terms, a long-horizon coding task usually follows this loop:
- Read the repository structure and dependencies to understand the existing implementation.
- Locate the change points and generate patches across multiple files.
- Trigger builds and static checks, then parse the errors.
- Run unit and integration tests to determine whether a regression has been introduced.
- Revise based on failure information, repeat steps 3–4, and continue until convergence or timeout.
The meaning of “long-horizon” lies here: success is not determined by the first patch, but by iterative convergence ability. This also explains why the same model can perform well on short question-answering tasks but degrade significantly on long-horizon tasks. Long-horizon tasks simultaneously push context length, tool-call reliability, error recovery, and cost constraints to their limits.
2. Read the Four Metrics Together
Public leaderboards usually provide pass rate, average completion, average wall-clock time, and cost per run. Each of these numbers is easy to misread in isolation.
- Pass rate: the percentage of samples in which all checks pass. This is a strict metric that reflects “whether the work can actually be delivered.”
- Average completion: a weighted completion ratio across checks or scoring dimensions. This is a looser metric that reflects “how far the work got.” Its gap from pass rate describes the distribution of partially finished tasks. A large gap means many tasks are stuck on the last one or two checks; such tasks can often be closed with a small amount of human intervention.
- Average wall-clock time: directly reflects team waiting cost and concurrency occupancy.
- Cost per run: the total cost of the entire evaluation chain, including model calls and execution overhead for tools and sandboxes.
A schematic way to read these numbers is:
Deliverability = pass rateCloseness = average completion - pass rate # A larger gap means higher value of human closureResource intensity = average wall-clock time × concurrency occupancyEconomic cost = cost per run / passed tasks # Closer to the true unit price of a usable delivery
The last line deserves special emphasis. When the cost difference between configurations is much larger than the pass-rate difference, “cost per successful delivery” often reflects the real economics better than “cost per run.” In the public data, compared with a low-cost lightweight configuration, the higher-completion configuration has roughly more than twice the wall-clock time and more than an order of magnitude higher cost per run, while its pass-rate lead is only about ten percentage points. This means the two configurations have completely different applicability boundaries, rather than a simple “who is stronger” conclusion.
3. What Engineering Trade-off Does “High Completion + Long Wall-Clock Time” Reflect?
This is the point most worth scrutinizing. High completion combined with long wall-clock time usually does not mean the model has become “dumber.” More often, the strategy layer has chosen a more conservative convergence path.
First, budget is exchanged for verification depth. More thorough exploration means more build and test triggers, more rounds of self-review, and longer context backtracking. These behaviors directly turn into wall-clock time, but they also significantly reduce the probability of “the fix is correct but a regression was introduced.” For high-risk tasks such as code migration, this exchange is usually worthwhile.
Second, long wall-clock time can mask insufficient parallelism. If the agent’s iterations are strictly serial—change one place, wait for a full build, change another place—the wall-clock time will be linearly amplified by the build duration. A better engineering approach is to split checks that can run in parallel: run unit tests, static checks, and compatibility checks separately, aggregate the failure information, and then give it to the model, rather than letting the model try one by one in serial.
Third, completion is fuzzy, but risk is not. Projects with an average completion of around 80% often include many samples that are “basically correct but missing a boundary-condition check” or “tests are not fully supplemented.” It is risky to merge such outputs into the mainline. Checks, not completion, must serve as the gate; completion must not be used as the release criterion.
Fourth, cost grows superlinearly with the number of iterations. As iterations increase, the input context of each round also grows, so cost approximately rises with the square of the number of rounds. This is why “running more rounds will eventually get better” does not hold in engineering. Explicit budget and iteration limits are required.
4. From the Leaderboard to Your Own Scenario: Three Sources of Bias
Public results should not be treated directly as selection conclusions. At least three sources of bias need to be corrected.
Agent framework differences. The same model paired with different agent frameworks can produce very different results. A leaderboard score is a joint result of “model + framework.” When attributing the result, it is important to distinguish whether it comes from model capability or from the framework’s context management, tool orchestration, and failure-retry strategy.
Task distribution differences. A sample size of about 30 tasks still has non-trivial statistical variance. In addition, public task sets tend to cover general mobile development and may not match your team’s technology stack, coding standards, or historical debt structure. The implicit conventions in your repository that no outsider knows are often exactly where an agent fails.
Measurement differences. Whether wall-clock time includes queueing and build waiting, whether cost includes sandbox and image pulling, and whether pass/fail includes human readability review—these definitions are not consistent across leaderboards. Directly comparing numbers across leaderboards is not rigorous.
The conclusion is straightforward: use public leaderboards to filter out clearly unsuitable options, and use a self-built evaluation set for the final decision.
5. Building a Reproducible Evaluation Set: A Minimum Viable Plan
The goal is not to replicate the leaderboard, but to build an evaluation that can run stably in CI and answer the question: “If we switch to this configuration, how will our delivery efficiency and cost change?”
Task card design. Each task should be described declaratively, avoiding any implicit conventions:
task_id: migrate-feature-to-modern-uirepo_snapshot: "content-sha256:9f3c1a..." # Pinned repository snapshot for reproducibilitycategory: code_migration # create | feature | migrate | convertrisk: high # low | medium | highconstraints:max_wall_clock_minutes: 90max_iterations: 12budget_usd: 5.0checks:- name: buildcommand: "<project build command>"- name: unit_testscommand: "<project unit test command>"required: true- name: static_checkcommand: "<project static check command>"required: false- name: diff_reviewkind: rubric # Human or rule-based readability reviewweight: 0.2
Scorer design. The key is to turn “partial completion” into a comparable numeric value rather than a boolean:
def evaluate(task, workspace, rubric_model=None):results = {c["name"]: run_check(c, workspace) for c in task.checks}weighted = 0.0total_weight = 0.0for c in task.checks:w = c.get("weight", 1.0)if c.get("required", False):score = 1.0 if results[c["name"]] else 0.0elif c.get("kind") == "rubric":score = rubric_model(c["rubric"], workspace) # 0~1else:score = 1.0 if results[c["name"]] else 0.0weighted += w * scoretotal_weight += wrequired_ok = all(results[c["name"]] for c in task.checks if c.get("required", False))return {"task_id": task.id,"passed": required_ok, # Pass-rate metric: strict"completion": weighted / total_weight, # Completion metric: weighted"cost_per_pass": None, # Computed by the aggregation layer based on passed tasks}
The aggregation layer then produces a report using the four metrics from Section 2, with an additional calculation of cost_per_pass = total cost / passed tasks. This table is what is truly persuasive at a selection meeting.
Three things must be pinned down: the repository snapshot, the random seed and temperature parameters, and the tool and runtime versions. If any of these drift, the evaluation results lose comparability.
6. Cost Governance: Layered Routing and Budget Gates
Since different configurations each have strengths, the optimal solution is usually not a binary choice but routing based on task characteristics:
def select_profile(task):if task.risk == "low" and task.scope == "single_file":return "light" # Low latency, low cost: formatting, renaming, adding tests, simple bug fixesif task.horizon_minutes >= 30 or task.risk == "high":return "deep" # High completion: cross-module migration, architecture changes, compatibility conversionreturn "balanced"
Supporting engineering practices include:
- Budget gates. Set dual upper limits of
max_iterationsandbudgetfor each task. When exceeded, output the “completed portion + to-do list” rather than retrying indefinitely. - Checkpoint and resume. Persist the completed workspace and check status. After an interruption, long tasks can resume from a checkpoint, avoiding repeated payment.
- Context budget. Use repository-level retrieval plus a summary index instead of stuffing the entire repository into the context. This significantly reduces input tokens and wall-clock time.
- Fast failure feedback. When a build fails, return only the aggregated key errors and the first root cause, not the complete log.
- Sandbox and least privilege. The agent executes in an isolated container and is granted only the necessary read/write scope and network policy. Build artifacts and patches remain fully rollbackable throughout.
7. How the Cloud Supports This System
The engineering complexity of a self-built evaluation lies mainly not in the model side, but in the “reproducible execution environment.” General capabilities can be combined as needed:
- Object storage: store repository snapshots, evaluation task cards, run logs, and artifact packages, naturally supporting per-task archiving and traceability.
- Container platform: spin up an isolated sandbox for each task, ensuring consistent dependencies and toolchain versions while supporting elastic concurrency.
- Message queue: decouple scheduling from execution to handle long-task queueing, retries, and peak shaving.
- Logging and monitoring/alerting: collect wall-clock time, iterations, token consumption, and failure reasons, and configure alerts for cost thresholds and pass-rate drops.
- Key and configuration management: centrally manage credentials and avoid spreading them into evaluation code.
On Baidu AI Cloud, these capabilities can be composed into a complete evaluation pipeline: object storage carries tasks and artifacts, the container platform carries sandbox execution, and logging and monitoring drive continuous observation of cost and quality. For selection, it is recommended to start with a small controlled experiment and then decide whether to scale up, avoiding a large investment in compute before the metric definitions are unified.
8. Conclusion
When facing a mobile development benchmark result, the real value is not the ranking but three judgments: pass rate determines deliverability, completion determines the value of human closure, and cost and wall-clock time determine whether the approach can scale. “High completion and long wall-clock time” is a rational engineering trade-off: it exchanges wall-clock time for deeper verification, but the cost is that budget gates and parallel checks must be configured well, otherwise the benefit will be eaten by build waiting.
For teams, the correct sequence is: use public leaderboards for coarse filtering, use a self-built reproducible evaluation set for fine filtering, use layered routing to apply the two configurations to suitable tasks, and use monitoring to manage drift in cost and quality. When evaluation itself becomes part of CI, the debate over model selection will shift from subjective impressions to engineering metrics that can be continuously verified.
