Coding Agent Benchmarks for Long-Horizon Tasks: How to Weigh Efficiency Against Cost
As AI coding agents move into day-to-day engineering work, evaluation has shifted from single-file code generation to end-to-end delivery on long-horizon tasks. This article breaks down what a mobile-development coding benchmark actually measures — four task categories, an agent-driven execution loop, and a two-layer verdict based on pass rate and completion — and explains the four headline metrics (pass rate, average completion, average runtime, cost per run) along with the derived indicators that expose the real gaps between configurations. Drawing on three representative configurations (24.0% pass rate at $317.4 per run versus 17.3% at $9.9), it explains why high completion does not equal task completion, why marginal cost climbs sharply near full delivery, and how enterprises should build their own benchmarks and match agent configurations to scenario, timeline, and budget.
1. Introduction: When the Test Subject Shifts from “Model” to “Agent”
In the early days of LLM code evaluation, developers asked a narrow question: “Is this function written correctly?” Benchmarks centered on single-file, single-function generation, used test-case pass/fail as the primary criterion, and rarely required more than one or two interaction turns.
That center of gravity has moved. As code agents — with an LLM at the decision-making core — enter everyday development workflows, what developers and enterprises actually care about is no longer “can the model write code,” but “can a vague requirement be carried end to end to completion inside a real engineering codebase.”
The test subject has therefore expanded from a model to a composite system: model + agent orchestration framework + tool chain + runtime environment. The model handles understanding and decisions, the agent handles planning and execution, the tool chain provides file I/O, command execution, and test running, and the runtime environment determines whether a task can actually be verified.
Mobile application development is one of the most representative proving grounds for this capability, for three reasons:
- Complex project structure: A shippable mobile app typically involves a multi-module project layout, build scripts, resource files, and permission declarations, while also accommodating different screen sizes and OS versions.
- Long verification chain: A single change often has to pass through compile, package, install, and run before you can confirm it actually took effect.
- Pronounced ecosystem differences: Build tooling and dependency management vary widely, some platforms mandate specific languages and project templates, and cross-platform migration or conversion is considerably harder than an ordinary backend change.
That is also why a set of programming benchmark results for real development tasks, published by a mobile platform’s official developer account, quickly drew attention from the developer community. This article systematically unpacks what such benchmarks test, how their metrics are defined, and the engineering trade-offs behind the numbers, and then outlines a practical method for enterprises that want to build their own evaluations and choose configurations rationally.
2. What the Benchmark Actually Measures: From Task Sets to the Verdict Pipeline
2.1 Four categories of real engineering tasks
Mobile-oriented coding benchmarks are typically distilled from real development workflows and cover four task types:
- App creation: Building a runnable project from scratch under given requirements and directory constraints, including project structure, baseline screens, and navigation logic.
- Feature development: Adding new capability to an existing project — a new business screen, integration with a class of data service, or a specific interaction flow.
- Code migration: Moving existing code from one framework version to another, or upgrading the build approach and project structure, with behavioral equivalence as the core requirement.
- App conversion: Porting an existing application across platforms or application form factors, which involves adapting both the cross-platform project structure and the runtime environment.
These tasks share one characteristic: long chains, many dependencies, delayed feedback, and errors that accumulate. A single task may require the agent to complete dozens or even hundreds of consecutive steps — planning, file edits, command execution, result verification, and error correction. A step that looks correct in isolation does not mean the final result is usable.
2.2 Execution model: agent-driven end-to-end delivery
In this kind of evaluation, the model does not simply “answer a question.” It is embedded in an agent framework and drives the whole task autonomously. The orchestration layer supplies the model with task context and available tools; the model emits decisions; tools perform file edits, builds, and tests; results are fed back to the model, closing the loop until the agent decides the task is done or a run limit is hit.
Consequently, benchmark results reflect the capability of the entire chain: agent strategy, tool quality, and the runtime environment all directly affect the final score. The same model plugged into different agent frameworks can produce noticeably different results.
2.3 Scoring: pass/fail plus a completion score
Final verdicts typically use a two-layer scoring system:
- Task pass (binary verdict): The agent’s final output is handed to a judge. A task counts as passed only if it fully satisfies every requirement — the project compiles, the feature runs, key tests pass.
- Completion (process score): For tasks that do not pass, the judge checks the agent’s actual progress item by item against the task requirements — whether files were created correctly, whether the core logic was implemented, whether the app ran — and converts that into a weighted completion score.
The design intent is clear: pass rate measures whether the work can be delivered, completion measures how far it got. Only the two together describe an agent’s real performance on long-horizon tasks.
3. The Four Core Metrics: How to Read the Leaderboard Numbers
3.1 Pass rate: the end-to-end delivery rate for long-horizon tasks
Definition: The share of test tasks whose final output is judged to fully satisfy the requirements.
This benchmark contains 30 long-horizon tasks. The base is small, which matters when reading the numbers: on a 30-task set, one more or fewer pass moves the pass rate by roughly 3.3 percentage points. Configurations with similar rankings may not differ significantly in statistical terms, and a gap of a few tenths of a percentage point should not be read as a generational difference in capability.
For enterprises, pass rate is the metric closest to “usable value.” Only fully passed tasks genuinely reduce human effort; a task that does not pass requires a person to finish it, no matter how high its completion score is.
3.2 Average completion: how far the work got
Definition: The arithmetic mean of completion scores across all tasks. A higher number means the agent is, overall, closer to getting the job done.
Its value lies in distinguishing the quality of failure:
- A task completed to 90% may need only a final configuration tweak or test fix, so topping it up by hand is cheap;
- A task completed to 30% may be flawed in its very approach and usually has to be redone from scratch.
Completion has limits, though. Scoring rules are set by humans and are hard to compare across benchmarks, and the question a dev team cares about most — “can this ship as-is?” — is ultimately answered by pass rate.
3.3 Average runtime: efficiency and the human-machine collaboration rhythm
Definition: The mean end-to-end wall-clock time for an agent to finish a single task, normally including model inference waits, tool calls, and execution queuing.
Runtime matters beyond “faster vs. slower”:
- The longer a task runs, the longer the agent’s decision sequence, and the greater the risk of error accumulation and context bloat.
- From a collaboration standpoint, runtime sets the pace of human-machine interaction. Tasks exceeding ten hours usually mean cross-day runs, which require checkpoint/resume and mid-run intervention mechanisms; tasks under twelve hours can complete a full feedback loop within one working day.
3.4 Cost per run: the direct measure of real investment
Definition: The cost of running the benchmark once end to end, composed mainly of input/output token charges from model calls plus the compute consumed by tool execution.
Long-horizon tasks have a distinctive cost structure: cost accelerates as the task progresses. At every step, the agent submits the task description, conversation history, code already read, and command output as context. As steps accumulate, context keeps expanding, so per-step token consumption in later stages is markedly higher than in early ones. Configurations with longer runtimes therefore not only tend to have a higher unit cost — their cumulative consumption is more easily amplified.
3.5 Three representative configurations compared
Within the same benchmark, the newly added configuration achieved a 24.0% pass rate, above the three configurations already on the leaderboard. Its cost per run was $317.4, high among the publicly reported configurations. A low-cost configuration completed the evaluation at a 17.3% pass rate, 71.2% average completion, 12.0 hours average runtime, and $9.9 per run; the remaining configurations fell in between. The three representative configurations compare as follows:
| Configuration type | Pass rate | Avg. completion | Avg. runtime | Cost per run |
|---|---|---|---|---|
| High-completion, long-runtime | 24.0% | 82.3% | 27.4 hours | $317.4 |
| Middle | Between the two | Between the two | Between the two | Between the two |
| Low-cost, fast-responding | 17.3% | 71.2% | 12.0 hours | $9.9 |
3.6 Four metrics are not enough: adding derived indicators
Raw metrics each answer one question. Combining them is what reveals the real gap between configurations:
| Derived indicator | Formula | High-completion, long-runtime | Low-cost, fast-responding |
|---|---|---|---|
| Cost per pass-rate point | Cost per run ÷ pass rate | ≈ $13.2 / point | ≈ $0.57 / point |
| Completion output per hour | Avg. completion ÷ avg. runtime | ≈ 3.0 points / hour | ≈ 5.9 points / hour |
| Marginal cost | Cost delta ÷ pass-rate delta | ≈ $45.9 per +1 point of pass rate | — |
| Human-machine collaboration window | Avg. runtime ÷ 24 hours | More than 1 day | Within half a day |
The most revealing is marginal cost. Compared with the low-cost, fast-responding configuration, the high-completion, long-runtime configuration gains 6.7 percentage points of pass rate and 11.1 points of completion, but costs $307.5 more per run — about $45.9 for each additional percentage point of pass rate, roughly 3.5× its average cost per pass-rate point ($13.2). In other words, once a certain baseline of capability is in place, buying additional end-to-end delivery capability is far more expensive than the average suggests.
4. Deep Dive: Why “High Completion” Is Not “Task Done”
4.1 Shared weaknesses exposed by the two configurations
Read together, the two configurations send three signals:
- There is still a clear gap between “writing code” and “delivering an app.” With pass rates of 24.0% and 17.3%, the majority of tasks still cannot go straight into delivery even after average runs of twenty-plus hours.
- Completion consistently exceeds pass rate. For both configurations, average completion is more than 50 percentage points above the respective pass rate, which means a large share of tasks stall at the last mile.
- The two data sets describe very different capability profiles. The high-completion, long-runtime configuration leads on both outcome metrics but takes longer and costs more; the low-cost, fast-responding configuration scores lower on pass rate and completion, yet finishes in less than half the time at roughly 1/32 of the cost.
4.2 The engineering logic behind long runtimes and high cost
Long runtimes and high cost are not merely “slower and more expensive.” They usually reflect a more thorough verification strategy:
- More thorough self-verification: After a change, the agent proactively runs builds and tests and iterates on the errors — close to a senior engineer’s “edit, verify, edit” habit. This raises success rates but also multiplies the number of model calls.
- Longer reasoning and larger context: Long-horizon tasks span many rounds of file edits and command execution, context keeps accumulating, and per-step token consumption rises substantially in later stages.
- Why high cost coexists with a low pass rate: Cost per run is averaged across all 30 tasks and includes the resources burned by many failed tasks. The deeper a task goes, the larger the context and the more correction loops, so cost grows faster — and yet a few missing critical pieces can still cause a failure.
4.3 The trade-off logic of low cost and fast response
The low-cost configuration reflects a different set of choices:
- Efficiency and cost control: Completion output per hour is about 2× that of the high-completion, long-runtime configuration, and tasks typically finish within half a day, returning results to developers faster.
- The price of that gain: Shorter runs generally mean less exploration of environment setup and fewer build/test rounds, so non-core-code concerns such as dependency configuration and runtime setup are handled with less depth. This is exactly why its profile reads “acceptable completion, low pass rate”: the main business logic is largely implemented, but the work still falls short of complete delivery.
4.4 Applying this to enterprise selection: three kinds of cost are not the same
When enterprises use leaderboard data, three different types of cost should be kept separate:
- Call cost: token and compute charges, which convert directly into a monetary figure.
- Time cost: developer hours spent waiting, correcting, and accepting the work — the hidden expense most often underestimated for long-runtime approaches.
- Opportunity cost: other work the team could have advanced in the same window.
Assessment should therefore be based on the total resources consumed per task — the total cost of one successfully delivered task — not on the price of a single run.
5. Enterprise Practice: Five Key Steps to Building Your Own Mobile Coding Benchmark
5.1 Build a representative task set
- Real sources: Draw tasks from the backlog, the defect tracker, and planned technical overhauls, prioritizing work items with clear boundary conditions and quantifiable acceptance criteria.
- Broad coverage: Cover all four task types — app creation, feature development, code migration, app conversion — and weight them according to your team’s actual business mix.
- Data desensitization: Remove sensitive and internal information, and keep task descriptions and acceptance criteria consistent in scope.
- Controllable scale: Balance statistical reliability against runtime cost; at least 10 tasks per category is a reasonable starting point, expandable in stages.
5.2 Design acceptance rules that can be judged automatically
Judging quality determines evaluation quality. Use “automation first, human spot-checks as backup”:
- Build: Whether the project compiles successfully is the most basic acceptance condition for a mobile task.
- Tests: Whether unit tests and key interaction cases pass.
- Behavioral consistency: Migration and conversion tasks should focus on verifying equivalence with the original behavior.
- Code conventions: Static analysis and adherence to team coding standards.
An illustrative acceptance flow, abstracted away from any specific tooling:
def judge_task(task, workspace):# 1. Baseline quality gatesbuild_ok = run_command("build", cwd=workspace)tests_ok = run_command("test", cwd=workspace)if not (build_ok and tests_ok):return Verdict(passed=False,completion=partial_score(task, workspace))# 2. Behavioral equivalence checks (critical for migration/conversion tasks)for case in task.acceptance_cases:if not run_case(case, workspace):return Verdict(passed=False,completion=partial_score(task, workspace))# 3. Convention checkslint_ok = run_command("lint", cwd=workspace)return Verdict(passed=lint_ok,completion=100.0 if lint_ok else 95.0)
Here partial_score walks the task checklist item by item — for example, “was the screen created,” “is the core logic implemented,” “did it run” — and computes partial credit by weight.
5.3 Build an isolated, reproducible evaluation environment
- Sandbox isolation: Run each task in its own container or cloud development environment so tasks cannot interfere with one another or pollute the baseline codebase.
- Pinned versions: Lock build tool and dependency versions, ideally distributing dependencies through a unified artifact repository so the environment stays consistent across evaluation rounds.
- Run constraints: Set maximum runtime, token budget, and step limits per run to prevent individual tasks from looping indefinitely.
- Stable feedback: Build failures, test errors, and command timeouts must be returned in full — this is the foundation of agent self-correction.
5.4 Establish observability, data collection, and cost accounting
- Execution traces: Record every step in full — file edits, command executions, error messages — to enable post-hoc review.
- Resource metering: Track token consumption, tool invocation counts, and total runtime separately, then aggregate per task.
- Failure attribution: Classify failures into categories such as environment setup, dependency management, business understanding, and runtime verification, to identify the main weak spots.
- Cost allocation: Break cost per run down to each task and compute the total cost per successfully delivered task, rather than relying on averages alone.
5.5 Design the evaluation process and interpret results soundly
- Repeated runs: Run each configuration at least 2–3 times, take the mean, and observe the variance band instead of drawing conclusions from a single run.
- Same environment: When comparing configurations, keep the agent framework, tool chain, runtime environment, and task set identical to reduce noise from unrelated variables.
- Layered reporting: Report pass rate, average completion, average runtime, and cost per run together, plus the derived indicators — not just a single ranking.
- Caution with small samples: On a benchmark of a few dozen tasks, one task’s variance can flip rankings, so avoid over-interpreting small differences.
6. Rational Selection: Match the Configuration to the Scenario
There is no universally optimal configuration in mobile development. The more sensible approach is to match the setup to task value, time constraints, and budget:
| Scenario | Recommended configuration | Rationale |
|---|---|---|
| Prototyping, scaffolding | Low-cost, fast-responding | Frequent iteration and a need for quick feedback; a little manual top-up is acceptable |
| Routine feature work within a release cycle | Mid-tier cost | Balances quality against cost |
| Multi-platform migration, app conversion | High-completion, long-runtime | These tasks demand correctness and complete delivery, and their total volume is limited |
| High-volume repetitive development | Judge by total cost per successfully delivered task | Cheap individual calls do not mean lower overall spend once rework is counted |
Whichever configuration you choose, three supporting mechanisms are worth putting in place:
- Cost guardrails: Set token and spend caps per run; on breach, terminate automatically and preserve completed progress so a single task cannot drain the budget.
- Human-in-the-loop: For complex work such as code migration and app conversion, add human checkpoints to review plans and outputs at key points, reducing rework risk.
- Cloud-side capabilities: Cloud providers’ agent development platforms, container platforms, logging services, and monitoring and alerting capabilities can support unified scheduling, run traceability, and cost accounting for evaluation tasks. Confirm actual availability against your platform’s documentation.
7. Conclusion
The distance between “AI writes code” and “AI delivers an application” is precisely what mobile coding benchmarks try to quantify. Pass rate answers “is it usable,” average completion answers “how far did it get,” and average runtime plus cost per run answer “what does it take.” Choosing between a long-runtime, high-completion configuration and a low-cost, fast-responding one is fundamentally trading time and money for completion quality and delivery certainty — and the marginal cost of moving closer to full delivery is clearly higher than the average.
For developers and enterprises, the value of a leaderboard is not a single answer but a reusable evaluation framework: build your own benchmark from your own tasks and real constraints, and make selection decisions with data rather than impressions. That is what turns AI coding tools into genuine gains in development efficiency.
