AI Coding Agents for Long-Horizon Tasks: Benchmark Results, Cost Analysis, and Tool Selection
This article breaks down a recent benchmark comparing AI coding agents on long-horizon, end-to-end engineering tasks. It explains how pass rate, completion level, runtime, and per-run cost are defined and measured, why a higher completion score does not necessarily translate into a higher pass rate, and how cost and time can outweigh a modest gain in pass rate. Using two anonymized tools (Tool A and Tool B), it shows how to calculate cost per passing task, why reproducibility and atomic task design matter, and how to integrate these metrics into CI/CD. It closes with a scenario-based selection method and a reproducible scoring example, helping engineering teams decide which agent fits their workload rather than chasing a single "best" tool.
1. The Setup: Long-Horizon Tasks Put “Smart” and “Affordable” on Opposite Sides of the Scale
A development community around a mainstream mobile operating system recently published a batch of coding benchmark results aimed at application development scenarios. The benchmark placed AI coding solutions of different positioning side by side in a single comparison. Because the original report did not disclose the full evaluation setup, this article refers to them uniformly as “Tool A” and “Tool B,” treats the numbers as illustrative, and focuses on methodology, measurement conventions, and engineering trade-offs.
| Metric | Tool A | Tool B |
|---|---|---|
| Long-horizon task pass rate | 24.0% | 17.3% |
| Average completion level | 82.3% | 71.2% |
| Average elapsed time | 27.4 hours | 12.0 hours |
| Per-benchmark-run cost | $317.4 | $9.9 |
Two differences stand out. First, Tool B’s pass rate is only 6.7 percentage points lower than Tool A’s, yet its per-run cost is roughly 1/32 of the other’s. Second, Tool A averages more than 27 hours per task, while delivering only an 11.1-percentage-point edge in completion. So the question of whether “high completion, long runtime” is actually worth it becomes something no team can avoid when selecting a tool.
2. First, Understand the Measurement Convention: Grading Agents on Real Engineering Tasks
This kind of coding benchmark is not a simple code-completion test. It evaluates an agent’s end-to-end capability on real engineering tasks. Typical tasks include application creation, feature development, code migration and version upgrades, and application conversion. They usually share three traits: long horizon, heavy environment dependence, and strict acceptance.
Concretely, a single task often requires the agent to go through dozens of rounds of a “read code → write code → run → read errors → revise again” loop; it must genuinely build successfully on a specific version of the compiler and system software; and it must ultimately be built automatically, run, and tested by the system. This means the evaluation is essentially a task-level, end-to-end evaluation, and the candidate is usually not a code model alone but a combination of “agent framework + code model.”
Four core metrics need clear calculation conventions:
- Pass rate: A task counts as passed only when every acceptance condition is met. It is a binary criterion, close to a formal “release gate.” The formula is typically: tasks passed / total tasks.
- Average completion level: Judged in segments against predefined checkpoints, where partially completed features can still earn points. It measures “how much value was delivered.” Values are typically 0 to 1 or 0% to 100%.
- Average elapsed time: The end-to-end wall-clock time for a single task, including thinking, generation, execution, and verification, measured in hours.
- Per-benchmark-run cost: The invocation and execution overhead of running the entire benchmark. This is a “whole-run cost,” not a per-task cost. To get a per-task cost, you must divide again by the number of tasks.
A task set of 30 is enough to observe trends, but there is still room for fluctuation at the decimal level. When reading the results, look at the overall picture rather than agonizing over ±1 percentage point.
3. Reading the Leaderboard: Three Key Signals
3.1 Why Is Completion High but Pass Rate Low?
From the data, Tool B’s completion is 11.1 percentage points lower than Tool A’s, yet its pass rate is only 6.7 percentage points lower. In other words, the pass-rate gap is actually smaller, while the completion gap is larger. Dividing one by the other, the pass-rate gap is about 0.60 times the completion gap; the two curves are not parallel.
Look closer: Tool A has 82.3% completion but only 24.0% pass rate; Tool B has 71.2% completion but only 17.3% pass rate. Both solutions show a clear “completion-to-pass-rate” drop-off. This indicates that failures often concentrate in the “last mile”: the interface may already be shaped, the logic largely correct, and local features may run, but the automatic build, full test suite, or compatibility checks—the veto items—are not passed.
Mature teams use a similar acceptance logic: even a single blocking defect means the release cannot ship. So the pass rate correlates more strongly with final delivery quality. Completion reflects “was it built,” while pass rate reflects “dare we ship it.” When evaluating, treat the pass rate as the core metric and completion as a diagnostic aid.
3.2 The Cost Advantage Far Outweighs the Pass-Rate Disadvantage: Roughly a 23x Gap in Cost per Passing Task
Cost cannot be judged by “per-benchmark-run cost” alone; you also need to see whether it corresponds to a whole-run cost or a per-task cost. In this case, both $317.4 and $9.9 are whole-run costs for completing the entire benchmark.
If you compute “amortized cost per passing task,” the formula is:
Cost per passing task = per-benchmark-run cost / (task count × pass rate)
So:
- Tool A: 317.4 ÷ (30 × 24.0%) ≈ $44.1
- Tool B: 9.9 ÷ (30 × 17.3%) ≈ $1.9
The two differ by roughly 23x. If the failed portion can be filled in manually, Tool B’s cost-performance advantage widens further. Under a fixed budget, a low-cost solution can cover more attempts, giving the team more room to experiment.
In addition, some articles use “cost amortized per unit of pass rate” for relative comparison, that is:
Cost per unit of pass rate = per-benchmark-run cost / pass rate
By this measure:
- Tool A: 317.4 ÷ 24.0% ≈ $1,322.5
- Tool B: 9.9 ÷ 17.3% ≈ $57.2
The two also differ by roughly 23x. But note that this is not the cost of a single passing task; it is a convention used for relative comparison. For engineering selection, prefer “cost per passing task,” because it is more intuitive and easier to fold into budget planning.
3.3 Elapsed Time Is a Long-Underestimated Cost Item
Tool A averages 27.4 hours, Tool B averages 12.0 hours, a difference of roughly 2.3x. If converted to “average elapsed time per passing task”:
- Tool A: 27.4 ÷ 24.0% ≈ 114.2 hours
- Tool B: 12.0 ÷ 17.3% ≈ 69.4 hours
Elapsed time may look like mere waiting, but it actually brings multiple costs: a dedicated execution environment is occupied for a long time; developers wait for feedback, incurring high context-switching costs; task queues grow, making capacity planning for nightly batch jobs more complex. More importantly, a single long-horizon task already reaches tens of hours, sometimes hundreds, far exceeding the minute-level time budget of a typical CI pipeline. In execution environments billed by usage or by the minute, long elapsed times translate directly into the bill.
4. Where Do the Differences Come From: Where Do Agents Fall Short?
4.1 Exploration Depth and Search Budget
Different agents handle long-horizon tasks differently. One type tends to build a finer task-decomposition tree, compare and select among multiple candidate implementations, and allocate more retry and self-check rounds to each branch. The other tends to quickly produce a runnable first version. The former is closer to “slow thinking,” more likely to take the right path in cross-module reasoning and complex refactoring, but at the cost of longer time.
4.2 Verification Loop Quality
Whether an agent can correctly run the build, read compiler and test logs, and trace errors to their root cause—rather than patching symptoms—is a key differentiator. Tool A takes nearly 2.3x the time of Tool B, and one likely reason is that it invests in more “generate → run → diagnose → fix” loops. When diagnosis is accurate, the extra time buys correctness; when diagnosis repeatedly misses, the time is simply wasted.
4.3 Long-Horizon Context Management
As task scale grows, the codebase and execution logs exceed the context window. Mechanisms such as segment summarization, retrieving relevant files, and writing intermediate conclusions into persistent notes determine whether the agent still remembers the round-1 objective at round 50. In long-horizon tasks, context loss often shows up as “early requirements forgotten,” “the same file modified repeatedly,” or “conflicts with already-verified conclusions.”
4.4 Reproducibility of the Execution Environment
Dependency version drift, invalidated build caches, and missing permission or signing configuration can all cause “the code is correct but the build fails.” This kind of failure hurts completion especially, because it does not reflect a genuine lack of model capability yet still drags down the pass rate significantly.
5. Selection Method: Match by Scenario, Don’t Chase a Single Champion
| Scenario characteristics | Suggested emphasis | Rationale |
|---|---|---|
| Complex refactoring, cross-platform migration, hard core-module work | High-completion solution | One-shot delivery requirements are high, rework is costly, and time can be scheduled for nightly batch jobs |
| Requirement exploration, style tweaks, small feature iterations | Low-cost, fast-response solution | Failure cost is low, trial count is high, and feedback speed sets the iteration pace |
| Concurrent multi-task workloads with a fixed budget | Decide by effective task cost | An expensive per-task cost shrinks the range of tasks you can cover |
| Pre-release compliance and quality gates | Acceptance centered on pass rate | Strict acceptance aligns with release-gate logic |
In practice, tiered scheduling often beats a binary either/or choice:
- Routine work such as requirement clarification, implementation drafts, and test completion goes to the fast and cheap solution;
- Tasks that repeatedly fail automated builds or tests, or that show no progress after two or three fix rounds, get escalated to the deep solution;
- All results flow into a unified quality gate, avoiding the notion that “switching models means lowering the bar.”
6. Engineering Implementation: Metric Conversion and CI/CD Integration
6.1 Establish a Unified Scoring Convention
Effective task cost is a foundational metric when choosing a model. If you want to balance quality, time, and cost, you can use a weighted score. The Python code below gives a reproducible illustrative implementation. Note that it takes aggregated metrics directly, avoiding the ambiguity that arises when back-calculating from task samples.
# pass_rate and completion come from the benchmark summary; run_cost is the whole-run benchmark costdef metrics(pass_rate, completion, avg_hours, run_cost, task_count):cost_per_task = run_cost / task_count# amortized cost per passing taskcost_per_pass = run_cost / (task_count * pass_rate) if pass_rate else float("inf")# cost per unit of pass rate, for relative comparison onlycost_per_pass_rate = run_cost / pass_rate if pass_rate else float("inf")return {"pass_rate": pass_rate,"completion": completion,"avg_hours": avg_hours,"run_cost": run_cost,"cost_per_task": cost_per_task,"cost_per_pass": cost_per_pass,"cost_per_pass_rate": cost_per_pass_rate,}# Calibrate weights against business risk; lower cost scores higher on the cost term,# and longer runtime scores lower on the time termdef weighted_score(m, w_pass=0.6, w_done=0.2, w_cost=0.1, w_time=0.1,cost_ref=50.0, time_ref=80.0):cost_score = cost_ref / (cost_ref + m["cost_per_pass"])time_score = time_ref / (time_ref + m["avg_hours"])return (w_pass * m["pass_rate"]+ w_done * m["completion"]+ w_cost * cost_score+ w_time * time_score)jia = metrics(pass_rate=0.240,completion=0.823,avg_hours=27.4,run_cost=317.4,task_count=30,)yi = metrics(pass_rate=0.173,completion=0.712,avg_hours=12.0,run_cost=9.9,task_count=30,)print(round(jia["cost_per_pass"], 1)) # ~44.1print(round(yi["cost_per_pass"], 1)) # ~1.9print(round(jia["cost_per_pass_rate"], 1)) # ~1322.5print(round(yi["cost_per_pass_rate"], 1)) # ~57.2print(round(weighted_score(jia), 3))print(round(weighted_score(yi), 3))
The key point of this code: cost_per_pass represents “the amortized cost per passing task,” while cost_per_pass_rate represents only “the relative cost per unit of pass rate.” The two cannot be used interchangeably.
6.2 Task Atomization and Acceptance-First Design
Don’t hand the agent vague tasks like “refactor the entire module.” Instead, break work into atomic tasks with explicit acceptance conditions. A task card can be written directly into a ticket in the requirement management system.
task: add coupon amount calculation to the order modulescope: [order/domain, order/tests] # whitelist of directories allowed to changeforbid: [order/api, shared] # directories that must not be touchedacceptance:- build command runs successfully with exit code 0- all unit tests pass- new cases cover three scenarios: full-reduction, discount, and mutually exclusive stackingbudget:max_rounds: 8 # at most 8 rounds per taskmax_minutes: 30 # abort on timeout, then hand off to a human or escalate the modelpath_guard:mode: whitelist_only # only paths within scope may be modifiedci_check: reject_if_touch_forbid
One point deserves emphasis here: a forbid configuration alone is not enough. CI should enforce constraints based on the scope whitelist and path validation. If any patch touches a forbid directory, the pipeline should block it outright, rather than relying on the agent’s self-discipline.
The value of acceptance-first design is this: the agent can judge for itself whether it is done, and failure information flows back to the fix step immediately, instead of surfacing only during human acceptance dozens of hours later.
6.3 Reproducible Execution Environment and Tiered Pipeline
- Containerize the environment: Fix dependency versions, the build environment, and permission configuration into a container image; store build caches and code artifacts in object storage or an artifact repository for cross-round reuse, reducing repeated compilation.
- Tiered execution: A code commit triggers fast checks that run in seconds to minutes, such as static analysis, unit tests, and interface tests; compilation or long-running compatibility verification goes into nightly batch jobs, staggered away from long-running tasks.
- Gates and failure classification: Have results recorded uniformly through build artifacts, test reports, and log services. Classify and count failures as “compilation errors, assertion failures, timeouts, resource limits, environment anomalies.” Environment-type failures should not count toward model capability assessment; otherwise you underestimate the model and mislead tool selection.
6.4 Cost Governance and Observability
- Record each run’s invocation count, execution time, and resource usage uniformly, and aggregate across dimensions such as task, model, and agent configuration, alerting promptly on abnormal consumption;
- Trim the injected context, enable result caching, and prioritize reusing retrievable knowledge snippets, avoiding the repeated injection of large blocks of background on every round;
- Set retry limits and budget circuit breakers for long tasks to avoid infinite loops and keep the bill within a predictable range.
7. Conclusion: Fix the Convention First, Then Talk Selection
The benchmark results yield a single selection thread: the pass rate determines “whether it can ship,” completion reflects “how much was delivered,” and time and cost determine “how many rounds you can run.” Tool A, at roughly 32x the cost and 2.3x the time, buys a 6.7-percentage-point lift in pass rate, making it better suited to complex migrations and hard core work; Tool B completes most features at less than one-tenth the cost, making it better suited to high-frequency iteration and budget-sensitive teams.
For engineering teams, the more worthwhile move is not to ask “who is the strongest,” but to do three things:
- Clarify your own task structure, matching high-frequency small tasks and low-frequency hard tasks to solutions with different capabilities;
- Build your own small regression task set, using the same convention to continuously track pass rate, completion, time, and cost;
- Solidify the “foundation”—reproducible environments, atomized tasks, and tiered pipelines—because the ceiling of an agent’s performance often depends on the quality of the engineering substrate.
