Mobile Development Benchmark Reveals Engineering Trade-offs in Agentic Coding
This article examines public results from a mobile development benchmark that evaluates agentic coding tools on 30 long-horizon tasks covering application creation, feature development, code migration, and application conversion. It explains why pass rate, average completion, wall-clock time, and cost must be read together, and highlights the performance gap between strict acceptance and partial progress. It also discusses why the agent harness itself matters, how to build a credible internal evaluation, and how to select strategies according to task value density and workflow requirements.
1. Three key signals from a mobile coding benchmark
A cloud provider recently published results from a benchmark aimed at mobile application development scenarios. The benchmark evaluates multiple intelligent coding tools. It does not ask them to write a quicksort algorithm. Instead, it tests 30 long-horizon tasks that are close to real engineering work, covering four categories: application creation, feature development, code migration, and application conversion. The agents must work in a real build environment, read code, modify code, run builds, fix errors, and continue until the task is closed.
Three groups of data are worth attention:
- The highest pass-rate tier is about 28.7%.
- A solution using an agent harness achieved a 24.0% pass rate, 82.3% average completion, 27.4 hours average time, and about $317 per benchmark run.
- A lighter-weight solution achieved a 17.3% pass rate, 71.2% average completion, 12.0 hours average time, and about $9.9 per run.
Three engineering-relevant signals can be drawn from these data.
Signal 1: Pass rates for long-horizon tasks are still in the 20%–30% range. Passing about 8 of 30 tasks is already first-tier performance. This is not because the models are “not smart enough.” It follows from the nature of the tasks: acceptance is compound. The code must compile, install successfully, pass existing tests, deliver the expected new functionality, and avoid breaking existing behavior. If any one condition fails, the entire task is considered failed.
Signal 2: There is a clear gap between average completion and pass rate. A completion rate of 82.3% corresponds to a pass rate of 24.0%; a completion rate of 71.2% corresponds to a pass rate of 17.3%. Completion measures how far the task was advanced. Pass rate measures whether all acceptance conditions were strictly met. The gap of tens of percentage points is consumed mostly by “last-mile” details: boundary handling, exception branches, resource cleanup, configuration adaptation, and similar issues.
Signal 3: Time and cost can differ by an order of magnitude. The long-running solution costs about $11.6 per hour, while the lightweight solution costs about $0.83 per hour. On a per-unit-completion basis, the figures are about $3.86 and $0.14, respectively. The difference in speed and cost is not a minor adjustment; it is an order-of-magnitude difference.
It is also worth noting that the long-running solution uses a different agent framework from most of the other solutions, yet its pass rate is only about 4.7 percentage points below the best result. This suggests a fact that is easy to overlook: for long-horizon tasks, the engineering quality of the agent framework—task decomposition, tool invocation, and error recovery—can matter as much as the underlying model itself.
2. How to read the four core metrics correctly
| Solution type | Pass rate | Average completion | Average wall-clock time | Per-run cost | Hourly cost | Cost per completion unit |
|---|---|---|---|---|---|---|
| Long-running, high-completion | 24.0% | 82.3% | 27.4 h | ≈ $317 | ≈ $11.6 | ≈ $3.86 |
| Best pass-rate tier | 28.7% | — | — | — | — | — |
| Lightweight, fast | 17.3% | 71.2% | 12.0 h | ≈ $9.9 | ≈ $0.83 | ≈ $0.14 |
(The table is an abstract summary of public leaderboard data. It is used only to illustrate the magnitude of the metrics and their trade-offs. It is not an evaluation or recommendation of any product.)
- Pass rate is a hard metric for full acceptance. It marks the boundary for whether a tool can operate unattended. It is measured at task level and is insensitive to partial success within a single task.
- Average completion is a continuous metric. It reflects the proportion of the task that was advanced and is useful for observing trends and optimization direction. However, it systematically overestimates real usability.
- Average wall-clock time is the actual delivery cycle. The leaderboard reports only averages, but real engineering should also look at P50/P90, because long-tail tasks determine whether human fallback is required.
- Per-benchmark-run cost is the total cost of model calls and tool execution. It is a direct cost item for automated pipelines.
The original four metrics are not enough. Three derived metrics are also useful:
Unit cost efficiency = average completion / per-run cost # Progress purchased per dollarEffective throughput = pass rate / average wall-clock time # Successfully delivered tasks per unit timeCompletion gap = average completion - pass rate * 100 # Severity of the “last mile” weakness
By these calculations, the lightweight option’s unit cost efficiency is about 27 times that of the long-running option, while its effective throughput is about 1.1 times. Cheap does not mean slow, and expensive does not mean fast. This is the key to selection.
3. Why “high completion + long duration” is a reasonable engineering trade-off
First, the cost of long-horizon tasks is superlinear. For every step, the agent must reread context, locate relevant files, trigger a build, parse errors, and decide what to do next. When a task reaches 70% completion, the remaining 30% is often the long tail: a test behaves differently across devices or OS versions, a dependency fails to download in an offline environment, or a migrated API signature does not match. The exploration budget needed to handle the long tail usually far exceeds the budget needed for the main development work earlier in the task.
Second, pass rate behaves like a step function, while completion is a continuous function. Improving completion from 71% to 82% only increases pass rate from 17% to 24%. This shows that marginal investment is mainly spent on filling acceptance details. If the business requires a very high first-pass rate—for example, unattended nightly batch processing—then the extra 7 percentage points and the extra 11 percentage points are worth it. If human review fallback is cheaper, there is no need to pursue those gains.
Third, duration determines orchestration. A 27.4-hour single-task duration is not suitable for an interactive IDE scenario, but it can fit an overnight batch process: submit the task in the evening and review it the next morning. A 12-hour duration is closer to a same-day turnaround. Therefore, duration is not an absolute disadvantage. It determines which workflow the solution can fit into.
4. Turning the trade-offs into engineering practice
1) Task decomposition and planning. Long-horizon tasks must be explicitly split into verifiable subgoals instead of asking the model to “think everything through and then write.” A recommended loop structure is:
def run_task(goal, env, budget):plan = planner(goal, repo_index=env.index) # Produce a list of verifiable subgoalsfor step in plan:if budget.exhausted():return checkpoint(budget) # Preserve state when budget is exhausted; do not force itpatch = executor(step, env.context())result = verify(step, patch, env) # Compile -> static analysis -> unit tests -> integration testsif not result.passed:feedback = summarize_failure(result.log) # Return only key errors to avoid context bloatpatch = executor(step + feedback, env.context())result = verify(step, patch, env)if result.passed:env.commit(step, patch) # Commit when passed to create rollback-able progressreturn verify_goal(goal, env)
2) Reproducible execution environment. A large share of build failures are not model-capability problems but environment drift. It is advisable to freeze the compiler toolchain, dependency versions, system images, and network proxy policies into container images. Dependencies should be distributed through an internal mirror repository so that agents do not waste budget on dependency installation failures.
3) Keep the verification loop short. Agents need to run verification after every iteration. The slower verification is, the worse the long-duration problem becomes. Layered verification is an effective approach: run second-level static checks and unit tests first, then run minute-level builds and integration tests later, and only trigger the latter after the earlier checks pass.
4) Context governance. Putting the entire codebase into the context is both expensive and inefficient. A practical approach is to build code indexes and semantic retrieval, dynamically retrieve files relevant to the task, summarize historical error logs instead of feeding them back verbatim, and summarize long sessions in segments so that conclusions are kept while processes are discarded.
5) Cost and progress budgets. Agents need hard limits to avoid infinite exploration on wrong paths:
If spent cost > budget * 0.6 and completion < 40%:Switch to a minimal-fix strategy and converge earlyIf 3 consecutive verification rounds fail and the error signature remains unchanged:Treat this as a directional error and roll back to the latest passed checkpoint
6) Tiered routing. Simple completion, formatting, and comment generation can be assigned to small models. Cross-file refactoring, migration, and difficult defects should be assigned to higher-capability solutions. This is the most practical way to improve both unit cost efficiency and pass rate.
5. Running a credible evaluation on your own codebase
A leaderboard can only serve as an external reference. Real decisions require an internal evaluation set. It can be built around five elements:
- Task set: Extract sanitized tasks from real tickets and stratify them across the four task categories: application or module creation, feature development, code migration, and framework or platform conversion. The set does not need to be large; 20–30 tasks are enough to cover the layers. The key is that every task has clear acceptance conditions that can be judged automatically.
- Environment isolation: Give each task a separate container instance, lock dependencies, disable access to external networks, and reset to the same initial commit before each run so that different solutions face the same initial state.
- Acceptance criteria: At least three layers are needed: compilation passes, existing tests do not regress, and targeted test cases for new functionality pass. Human scoring rubrics can be added when necessary, with 1–5 scores for code readability and completeness of exception handling.
- Metric collection: Write every round of tool calls, model requests, token consumption, and build logs into a logging service. Use dashboards and alerts for cost attribution and latency distribution so that teams do not end up with “the run finished, but we do not know where the money went.”
- Contamination and variance control: Check whether evaluation tasks overlap with model training data. Run each solution at least three times and report the mean and range, because agent paths are random and a single result is not enough to support a conclusion.
6. Selection guidance: tier by task value density
| Scenario characteristics | Recommended strategy | Reason |
|---|---|---|
| Nightly batch migration, unattended operation | Long-running, high-completion solution | Cost per unit time is less sensitive, and first-pass success has the highest value |
| Interactive daily development, assisted completion | Lightweight, fast solution | Response time is sensitive and human fallback cost is low |
| Critical paths, pre-launch modifications | High-capability solution plus mandatory human review | Failure cost is far higher than invocation cost |
| Exploratory prototype validation | Prefer lightweight solutions | The goal is iteration count rather than single-pass quality |
Three boundaries should also be kept in mind:
- Comparability is limited. Different benchmarks may differ in task sets, agent frameworks, time-measurement methods, and cost accounting methods. Directly comparing ranks across leaderboards has limited meaning. Horizontal comparison should return to the internal task set.
- A low pass rate does not mean unusable. A 17%–29% pass rate means that more than 70% of tasks require human intervention. However, those tasks often already have more than 70% of the code skeleton completed, and humans only need to finish them. The return on investment can still be positive.
- Compliance comes first. The license source of generated code, the scope of external transmission of enterprise code, and the masking strategy for sensitive data should all be clarified before agents are integrated, not remedied afterward.
7. Implementation checklist
- Build an internal evaluation set of 20–30 tasks with automatic acceptance scripts
- Freeze the build environment as container images and route dependencies through an internal mirror
- Configure triple budgets for cost, steps, and duration, along with checkpoint rollback for agents
- Implement layered verification and push high-frequency verification down to second-level response
- Connect logging services and monitoring alerts for cost attribution and latency distribution analysis
- Use tiered model routing and allocate compute according to task complexity
- Build evaluation baselines and a failure-case library to support prompt and process iteration
As intelligent coding tools enter long-horizon engineering tasks, the standard for evaluating them has shifted from “whether the code looks like it was written by a human” to “whether the task can be closed reliably under controllable cost.” Completion determines how much work can be replaced. Pass rate determines how much responsibility can be entrusted. Time and cost determine which workflow the tool can fit into. Understanding the relationship among these three factors is more valuable than remembering the rank of any leaderboard.
