Evaluating AI Agents on Long-Horizon Coding Tasks: Benchmark Results and Cost Trade-offs
A developer team at a mobile app development platform published results from a benchmark of 30 mobile app development tasks. A long-horizon agent with high completion and long runtime passed 24.0% of tasks at 82.3% average completion and $317.4 per run, while a lightweight opencode-style agent passed 17.3% at 71.2% average completion and $9.9 per run. This article explains what pass rate, average completion, average duration, and cost per run each measure, why capability and cost scale non-linearly, and how task tiering, sandboxing, observability, and private evaluation sets can turn public benchmark data into sound engineering decisions.
1. Benchmarking Background: AI Coding Moves from Code Completion to Long-Horizon Agents
The developer team at a mobile app development platform recently published benchmark results for a set of large language model (LLM) agents on mobile app development tasks. The standout was a long-horizon approach, defined by high completion and long runtime. Across 30 tasks, it achieved:
- A pass rate of 24.0%
- An average completion of 82.3%
- An average duration of 27.4 hours
- A cost of $317.4 per benchmark run
A lighter-weight entry built on an opencode-style agent trailed it with a 17.3% pass rate, 71.2% average completion, and 12.0 hours average duration. Its cost was only $9.9 per run.
The value of these numbers lies less in the rankings than in what they measure. Coding models have traditionally been judged on metrics such as code completion accuracy and unit test pass rates, which assess how well a few lines of code are written. Long-horizon benchmarks measure something different: whether an agent can complete the full loop on its own, from understanding requirements and modifying code to build and verify the result, without step-by-step human guidance. Mobile app development is a common choice for this kind of evaluation for three reasons:
- Complete task chains. Creating apps, building features, migrating code, and converting apps all involve continuous sequences of reading code, editing files, and running builds and tests.
- Objective verification. Whether an app compiles, installs, and launches, and whether critical paths pass their tests, can be judged against clear criteria rather than subjective scores.
- Realistic engineering environments. Build toolchains are sensitive to version differences, and UI behavior depends on the target device or emulator. These conditions closely match everyday development work.
For teams, the results ultimately answer one question: for which tasks, at what cost, and over what time frame, can an agent be trusted to work independently?
2. Reading the Metrics: What Each Number Means
2.1 Pass rate: the share of tasks fully completed end to end
Pass rate is the share of tasks an agent completes autonomously while satisfying every verification condition. The judgment is strictly all-or-nothing. A failed build, a missing core feature, or any test that does not pass counts as a failure. A 24.0% pass rate does not mean the other 76% of tasks produced nothing of value. It means they have not yet reached a deliverable standard.
2.2 Average completion: progress is more revealing than success
Average completion measures the stage reached on average across all tasks, including failures. A higher figure means failed attempts got closer to success. The two agents differ by 6.7 percentage points in pass rate but by 11.1 percentage points in completion. Failed attempts therefore also vary considerably in quality. Looking at pass rate alone tends to undervalue near-miss approaches that may prove worthwhile in real iteration cycles.
2.3 Average duration: wall-clock time, including waits and retries
Average duration is the wall-clock time from task dispatch to final result. It is not pure inference time. It combines inference, tool execution, retries after failures, and waits for human takeover when needed. At 27.4 hours, a typical task runs for more than a working day. That raises the bar for task hosting, progress tracking, and cost control.
2.4 Cost per run: the price of one task
Cost per run is the agent usage fee for one complete benchmark run covering all 30 tasks. Per task, it works out to:
- Long-horizon approach: $317.4 ÷ 30 ≈ $10.6
- Lightweight approach: $9.9 ÷ 30 ≈ $0.33
These figures cover only what the agents pay for model calls. They exclude infrastructure such as build machines, emulators, and storage, as well as the time engineers spend reviewing and taking over work.
3. Side-by-Side Comparison: Cost-Performance Matters More Than Rank
| Entry | Pass rate | Average completion | Average duration | Cost per run |
|---|---|---|---|---|
| Top-ranked approach | 28.7% | Not disclosed | Not disclosed | Not disclosed |
| Long-horizon agent | 24.0% | 82.3% | 27.4 hours | $317.4 |
| Lightweight agent | 17.3% | 71.2% | 12.0 hours | $9.9 |
Three conclusions stand out:
- Cost and capability do not scale linearly. The long-horizon agent’s pass rate is about 1.4 times that of the lightweight agent, but its cost per run is about 32 times higher.
- Marginal cost rises sharply as capability improves. Starting from the lightweight agent, lifting the pass rate by 6.7 percentage points costs roughly $45.9 per additional percentage point per run.
- Small samples exaggerate ranking gaps. The benchmark has only 30 tasks, so a single task is worth 3.3 percentage points. The 6.7-point gap between the two agents amounts to just two tasks. Statistical variance is therefore significant, and selection decisions should be based on results across multiple runs.
4. Engineering Trade-offs: The Price of High Completion, Long Runtime, and High Cost
The long-horizon agent has a clear profile: high completion, long runtime, high cost, and a leading pass rate. In engineering terms, this is trading compute for correctness. The trade-off plays out on three fronts:
- Exploration depth. Letting the agent run more cycles of edit → build → test → read logs → edit again lets it cover more edge cases and implicit constraints. The price is added time and cost.
- Context overhead. Context grows as a long-horizon task accumulates steps, and every step carries a large history with it. Input token usage tends to climb quickly with the number of steps. This is the main driver of long-task cost.
- Verification strength. A stricter verification pipeline reduces results that appear to pass but are not actually usable. However, every build, test run, and device launch takes extra time.
One distinction is easy to overlook. Completion measures how much work was done, while pass rate measures how much of it was done correctly. High completion combined with a lower pass rate often means many tasks stall in the last mile. Typical examples are UI interaction details, exception-path handling, and compatibility with system components. These issues are costly to fix and hard to reproduce, which makes them the hardest part of long-horizon tasks.
The lightweight agent, by contrast, delivers about 60% of the top-ranked agent’s pass rate for roughly 3% of the long-horizon agent’s cost per run. Its absolute pass rate is modest. Even so, for short, well-defined tasks such as generating boilerplate code in bulk or quickly testing technical feasibility, it responds fast enough to support a “get it running first, then refine” workflow.
Selection should therefore start not with “which model is strongest” but with three questions:
- How expensive is it to fix a failed task?
- Can the team tolerate tasks that take hours or even days?
- How many deliverable results do you need per unit of budget?
5. Engineering Practices: Turning Benchmark Results into Developer Productivity
5.1 Task tiering and model routing
Group tasks by complexity and the cost of failure, and assign agents of different capabilities and costs to each group. Enforce budget caps as hard cutoffs.
# Task-tier routing: match agent capability and a budget cap to each task type (illustrative)TASK_TIERS = {"boilerplate": {"agent": "light", "max_steps": 20, "budget_usd": 0.5},"feature": {"agent": "light", "max_steps": 60, "budget_usd": 3.0},"migration": {"agent": "long", "max_steps": 300, "budget_usd": 15.0},}def run_agent(task):tier = TASK_TIERS[task.category]state = {"messages": load_context(task), "spent": 0.0, "steps": 0}while state["steps"] < tier["max_steps"]:resp = call_model(agent=tier["agent"], messages=state["messages"])state["messages"].append(resp)state["steps"] += 1state["spent"] += estimate_cost(resp.usage) # Bill by input/output token usage of each call# Cost guardrail: hand off to a human once the per-task budget is exceeded, to keep long-horizon tasks in checkif state["spent"] > tier["budget_usd"]:return escalate_to_human(task, state, reason="budget_exceeded")if resp.is_final:return verify_and_commit(task, resp) # Run builds and tests; commit only if all passresult = run_tool(resp.tool_call, sandbox=task.sandbox)state["messages"].append(result)return escalate_to_human(task, state, reason="step_limit")
5.2 Sandboxed environments and reproducible builds
Long-horizon tasks often run for tens of hours. Retries caused by unstable environments then inflate both duration and cost. A few practices help:
- Give each task its own sandbox, with a container platform handling scheduling.
- Prebuild base images that include the required toolchains and dependencies.
- Store build artifacts and dependency caches in object storage so they can be shared and reused across tasks.
When a task times out or fails, its environment can simply be destroyed and recreated. This prevents a “dirty” environment from contaminating later results.
5.3 Observability and cost governance
Costs rarely spiral because of a single expensive call. More often, they accumulate from repeated retries in long tasks and from carrying the full context on every step. Log structured metrics for every task and feed them into centralized logging, monitoring, and alerting, with thresholds on per-task cost, step count, and duration:
{"task_id": "migrate-settings-page","category": "migration","status": "failed","completion": 0.82,"duration_hours": 27.4,"steps": 186,"input_tokens": 2350000,"output_tokens": 96000,"cost_usd": 10.6,"human_interventions": 1}
With this data, you can see whether cost is dominated by inference, tool execution, or human intervention. Organizations that want to onboard multiple models and run unified evaluations and cost accounting can rely on one-stop platforms such as Baidu AI Cloud’s Qianfan large model platform for model management and data governance. The sandboxes themselves can run on container platforms such as Baidu AI Cloud’s container services. Paired with object storage, logging, and monitoring alerts, they provide environment isolation and cost tracking.
5.4 Human-in-the-loop collaboration and private evaluation sets
Long-horizon tasks should default to a “supervised with checkpoints” mode. The agent pauses for human approval at key points such as requirement confirmation, solution selection, and final submission, and runs automatically in between.
Meanwhile, once a public benchmark’s tasks and scoring criteria become familiar to the community, rising scores do not necessarily indicate real capability gains. A more reliable approach is to mine real tasks from the team’s own ticket history, such as feature migrations between past releases or bug fixes. Use them to build a private evaluation set and run it on a schedule through the CI pipeline, with metrics aligned to those of public leaderboards. That way, every change to a model or agent configuration can be judged by its actual benefit.
6. Common Pitfalls and How to Avoid Them
- Treating completion as success. Completion reflects progress, not usability. Acceptance must always rest on objective build and test results.
- Ignoring statistical variance. With only 30 tasks, one or two results can change the ranking. Focus on averages across multiple runs and their confidence intervals.
- Counting only inference costs. Engineer review, environment resources, and failed retries are costs too, and should be included in total cost of ownership.
- Generalizing from a single benchmark. Findings from mobile app development do not necessarily hold for backend services or data engineering. Base selection decisions on evaluation sets that reflect your own workloads.
7. Conclusion
The value of long-horizon benchmarks is that they move the evaluation of AI coding agents from isolated coding ability to complete engineering tasks. High-completion, long-running approaches suit complex work with little room for error and a high cost of failure. Low-cost, fast-responding approaches suit bulk and exploratory work. Understanding how pass rate, completion, duration, and cost relate to one another, and turning public results into sound decisions through tiered routing, sandbox isolation, observability, and private evaluation sets, is the right path for teams looking to improve engineering productivity.
