Agent Long-Horizon Task Evaluation: How to Read the Metrics and Pick the Right Solution

    This article explains how to evaluate AI agents on long-horizon tasks and how to read leaderboard results without falling into common traps. Drawing on a recent benchmark that added both a long-horizon autonomous agent and a lightweight agent, it breaks down what the four headline metrics actually measure—pass rate, average completion, average runtime, and per-run cost—and where each one can mislead. It shows why per-run cost is the wrong number to compare on, how to compute cost per successful task instead, and why task distribution, evaluation methodology, and human fallback costs matter more than a single ranking. Readers will find practical guidance for triangulating quality, cost, and delivery time, along with a reproducible evaluation pipeline covering task-set design, environment isolation, full trajectory logging, statistical design with budget guardrails, an executor skeleton, and tiered routing, so teams can build a baseline that reflects their own workload.

    1. Introduction: The Benchmark Changed, So Should the Way You Read It

    According to public industry information, a recent mobile-development benchmark added results for two new categories of agent solutions. The first is strong at long-horizon autonomous execution: across 30 long-horizon tasks it reached a 24.0% pass rate, 82.3% average completion, 27.4 hours average runtime, and a per-run cost of roughly $317.4 (about ¥2,123). The second is a lightweight solution with a 17.3% pass rate, 71.2% average completion, 12.0 hours average runtime, and a per-run cost of only about $9.9 (about ¥66.2).

    The numbers themselves are not complicated, but many teams stop at “just pick the one with the higher pass rate,” which leads to two common misjudgments:

    • Treating leaderboard scores as your team’s delivery efficiency. Benchmark tasks are deliberately designed to be hard and time-consuming, while everyday team work is mostly short- and mid-horizon. The two distributions are not the same.
    • Treating per-run cost as your real spend. A leaderboard counts only the cost of “one evaluation,” ignoring retries after failure, solution screening, test writing, and human review and rework.

    In practice, the object of evaluation has long since shifted from “generate a snippet of code” to “complete a long-horizon task.” Long-horizon tasks have three typical characteristics:

    1. Cross-file and cross-module: they require coordinated changes across UI, business logic, data persistence, and test code.
    2. Multiple tool-call rounds: the agent repeatedly builds, runs tests, reads logs, and fixes code until the artifact passes acceptance.
    3. The output must actually run and be verifiable: whether it passes tests and produces an installable artifact is the final judgment.

    In this context, a leaderboard no longer measures the generation quality of a single model. It measures the combined result of “model capability × agent framework × toolchain × budget strategy.” So the right way to read a leaderboard is not to find “number one,” but to answer three questions:

    • What does each metric actually measure?
    • Under what conditions does its conclusion hold?
    • Under which of those conditions does my team sit?

    The rest of this article addresses these three questions.

    2. Understanding the Core Metrics: What Each Number Tells You

    2.1 Pass Rate: A Hard Metric, but Bounded by Sample Size and Scoring Rules

    Pass rate is the proportion of evaluation tasks that fully pass acceptance. It is the most direct signal of how usable the tasks really are. It answers “how many tasks can be delivered as-is,” not “does the code look right.” It usually maps directly to real rework cost: tasks that fail need human takeover, which often costs more than the failed run itself.

    Two things must be kept in mind when reading pass rate:

    • Small sample, large variance. If pass rate is counted per task, each of the 30 tasks is worth about 3.3 percentage points, so a few tasks succeeding or failing can shift the result noticeably. By a binomial-distribution estimate, a solution with a true pass rate of about 24% has roughly a 95% chance of landing anywhere in the 12%–39% range across 30 independent evaluations. A leaderboard reports a single run, so a gap of a few percentage points between adjacent solutions may well fall within that noise band.
    • The scoring rule needs confirmation. Neither 24.0% nor 17.3% divides evenly into an integer share of 30 tasks (7/30 ≈ 23.3%, 5/30 ≈ 16.7%), which suggests the leaderboard likely uses partial scoring, averaging over multiple runs, or per-task weighting. This is a reasonable inference, not a rule the leaderboard has disclosed. Two implications follow:
      • Pass rate should be read together with completion.
      • Confirming the scoring rule with the leaderboard owner or your internal evaluation platform is a necessary step before citing the data.

    For teams, it is more valuable to establish a tiered pass rate: stratify tasks by difficulty and type and report pass rates per stratum, rather than looking at a single overall number.

    2.2 Average Completion: Measures “How Far Off,” but Can Be Overestimated

    Average completion captures how far a task was pushed. It answers “how much of the unfinished tasks actually got done” and helps gauge how far a failed task is from usable. For example:

    • A failed task at ~70% completion may only need edge-case handling and tests filled in.
    • A task with lower completion may need a redesign at the solution level.

    In this dataset, the long-horizon solution’s completion is 11.1 points higher than the lightweight one, indicating it is also better at “pushing tasks further.” But completion is usually based on rule-based scoring (e.g., number of checks passed) or model judgment, and it carries a risk of overestimation:

    • An agent may produce code that looks structurally complete but does not actually pass.
    • The judging criteria may be more lenient toward solutions that “produce more content.”

    For this reason, high completion is best treated as potential value or room for follow-up investment, not a delivery guarantee. A checkpoint-based evaluation is recommended: score each item on a checklist, then calibrate with automated tests and human review.

    2.3 Average Runtime: Drives Interaction Experience and Task Throughput

    Average runtime affects both the interaction experience and parallel throughput. The long-horizon solution averages 27.4 hours—more than a full working day—while the lightweight solution takes 12.0 hours, about half as long (precisely 43.8%). In a team setting, this number has to be read together with the concurrency strategy:

    • Serial execution: 10 tasks queue one after another, giving cumulative wait times of roughly 274 hours versus 120 hours.
    • Parallel execution: wait time depends on the slowest task, but parallelism also amplifies the peak cost and demands more from build machines, emulators/physical devices, network quotas, and other resources.

    Note that the high dispersion of long-horizon runtimes tends to weaken the usefulness of the average. In practice you should track P50 and P95 runtimes as well, because the tail tasks usually determine the delivery cadence.

    2.4 Per-Run Cost: The “Average Price Tag” That’s Easy to Misread

    Per-run cost usually means “the model-call cost of completing one benchmark evaluation.” It is directly affected by:

    • model unit price
    • input and output token volume
    • reasoning effort and retry count
    • number of tool-call rounds

    The long-horizon solution’s per-run cost is about $317.4 (¥2,123), roughly 32.1× the lightweight solution’s ~$9.9 (¥66.2). That number gives a first impression of “too expensive,” but it overlooks a key fact: only tasks that actually complete produce business value. A more reasonable comparison is therefore average cost per successful task, i.e., per-run cost divided by pass rate:

    • Long-horizon: ~$317.4 ÷ 24.0% ≈ $1,322 (about ¥8,846)
    • Lightweight: ~$9.9 ÷ 17.3% ≈ $57 (about ¥383)

    Even under this metric, the two still differ by roughly 23×. So “which is more cost-effective” is essentially a business question: if the human fallback cost of a failed task is far below the price gap, the lightweight solution is more economical; if the task itself is high-value and failure is costly, the long-horizon solution’s higher cost may be substantially offset.

    Also be wary of the intuition that “higher price means higher quality.” In relative terms, the long-horizon solution’s pass rate is about 38.7% higher while its cost is about 31× higher. Judged only by these two numbers, the higher spend did not buy a proportional gain in effectiveness. This conclusion should not be generalized to all long-horizon tasks or all engineering scenarios.

    2.5 Cross-Metric View: Read the Four Numbers Together

    Metric Long-horizon Lightweight Comparison (long-horizon vs. lightweight)
    Benchmark tasks 30 long-horizon tasks 30 long-horizon tasks Same benchmark
    Pass rate 24.0% 17.3% +6.7 points (~38.7% relative)
    Average completion 82.3% 71.2% +11.1 points
    Average runtime 27.4 hours 12.0 hours ~2.3×
    Per-run cost ~$317.4 (~¥2,123) ~$9.9 (~¥66.2) ~32.1×
    Cost per successful task ~$1,322 (~¥8,846) ~$57 (~¥383) ~23.1×

    (The last two rows are estimated from the leaderboard data; exchange-rate fluctuations affect the exact figures, so use them only for order-of-magnitude comparison.)

    Combining the metrics yields two more actionable reading rules:

    1. Low pass rate + high completion means “close to usable.” These failed tasks may be one step short of done, making them prime candidates for human takeover and process optimization.
    2. High pass rate + high completion means fewer capability gaps. But look further: are the newly passed tasks genuinely hard ones, or ones that were relatively easy to begin with?

    3. A Cross-Sectional View: How the Two Solutions Are Positioned

    Comparing across the board, the long-horizon solution scores higher than several other types of agent solutions, but below the top solution in the leaderboard (28.7% pass rate). This tells us two things:

    • Under the same benchmark, “long-horizon” does not equal “strongest.” An agent’s task-decomposition strategy, context management, and tool-call orchestration all affect the result significantly.
    • A leaderboard is a moving coordinate system. Each update can change the reference frame, so direct comparisons across leaderboards or across time should be made cautiously.

    Returning to the two new solutions, their positioning can be summarized as:

    • Long-horizon solution: trades time and cost for pass rate and completion. Better suited to structurally complex tasks with strict acceptance and a high cost of error.
    • Lightweight solution: trades pass rate and completion for speed and low cost. Better suited to high-frequency, low-risk tasks that are sensitive to response time.

    They are therefore more complementary than mutually substitutive. A common practice among teams is hybrid scheduling:

    • Use the lightweight solution for batch, low-risk tasks to quickly obtain verifiable, incremental results.
    • Hand complex tasks to the long-horizon solution, or escalate to it for a retry after the lightweight solution fails.
    • For high-risk tasks, have humans review at key checkpoints.

    4. An Engineering View: Turning Leaderboard Conclusions into Selection Criteria

    4.1 From “Best on a Single Metric” to a Triangle of Cost, Quality, and Delivery Time

    The classic software-engineering trade-offs apply equally to agent selection:

    • Quality: measured by pass rate and rework rate.
    • Cost: includes both the per-run model cost and the human fallback cost of failed tasks.
    • Delivery time: determined by average runtime, P95 runtime, concurrency, and queueing strategy.

    A schematic formula captures this trade-off (for illustration only; actual weights must be calibrated with team data):

    1. Composite score = business value × pass rate + remaining value × completion
    2. − runtime cost − expected fallback labor cost − waiting cost
    • High-value, long-cycle, one-off tasks (e.g., complex architecture migrations, hard-to-diagnose defects): prioritize pass rate and completion.
    • High-frequency, short-cycle, low-risk tasks (e.g., routine bug fixes, boilerplate code generation, batch copy and config changes): prioritize per-run cost and response time.

    4.2 Task Distribution Is the Decisive Variable

    The 30 tasks in the leaderboard are deliberately designed, usually harder and more time-consuming than everyday needs. If a team’s work is mostly short- and mid-horizon, the real pass-rate gap for the lightweight solution is likely smaller than the leaderboard suggests. A more reliable approach is to:

    • Sample tasks from your real backlog (bug tickets, feature requests, code-review comments) to build your own task set.
    • Evaluate candidate solutions on that task set.

    4.3 Evaluation Rules, Versions, and Comparability

    Cross-comparison holds only when evaluation conditions are consistent. Whether you cite a leaderboard or run your own evaluation, at minimum confirm the following:

    1. Task definition and acceptance criteria: whether test cases, build commands, and checklists are public and reproducible.
    2. Runtime environment and tool versions: whether SDK versions, build-tool versions, and base images match.
    3. Number of runs and scoring method: single run or averaged over multiple runs, and whether partial scoring is used.
    4. Budget constraints: whether per-task timeouts and cost caps are set, and how tasks that hit a cap are scored.
    5. Cost definition: whether it includes only model-call costs, and whether retries and failed tasks are counted.

    At the same time, recognize that leaderboards update dynamically. New solutions change the rankings, and model and agent version iterations quickly make historical data stale. It is better for teams to establish their own baseline and run regular regressions than to depend on external leaderboard rankings for the long term.

    4.4 Organize a Selection Matrix with “Scenario Tags”

    Tag tasks by “risk × complexity × time sensitivity” and pre-assign a default strategy to each tag:

    Task type Typical traits Suggested default strategy
    Boilerplate and batch tasks Low risk, fixed pattern, high volume Lightweight first, escalate on failure
    Routine bug fixes Low-to-medium risk, well-scoped Lightweight primarily, verified by automated tests
    Complex bugs and performance issues Medium-to-high risk, multi-round verification Long-horizon or human-led
    Architecture and cross-module changes High risk, wide impact Human-led, long-horizon assists with generation and verification
    Release-critical tasks High risk, time-sensitive Human decision, agent assists only

    5. Practical Guidance: Build a Reproducible Long-Horizon Evaluation Pipeline

    External leaderboards only offer reference. What a team really needs is an evaluation baseline consistent with its own engineering environment. A practical pipeline usually has the following six stages.

    5.1 Task-Set Construction: Realistic, Stratified by Difficulty, with a Unique Acceptance Standard

    • Extract tasks from real requirements and bug records, stratified by difficulty and type.
    • Attach a clear acceptance standard to each task (test cases, build commands, checklists).
    • Prepare the initial code snapshot and a reproducible runtime environment in advance.

    Schematic task definition (YAML, format illustration only):

    1. task_id: fix-crash-on-back-press
    2. category: bugfix # task type: feature / bugfix / refactor / test
    3. difficulty: medium # difficulty: easy / medium / hard
    4. prompt: "Fix the intermittent crash when pressing Back repeatedly from a secondary page, and add corresponding unit tests"
    5. acceptance:
    6. build_command: "<build and packaging command>" # must produce an installable artifact
    7. tests:
    8. - "<unit test or UI automation test targeting this bug>"
    9. budget:
    10. max_wallclock_hours: 8 # per-task wall-clock time limit
    11. max_cost: 20 # per-task cost limit (unit defined by the evaluator)

    5.2 Environment Isolation and Elastic Resources

    • Environment isolation: run each task in its own container, destroyed when the task ends, to prevent cross-contamination.
    • Elastic resources: provision build machines and emulator/physical-device resources on demand, and cap parallelism for long-running tasks.
    • General cloud capabilities: on Baidu AI Cloud, for example, evaluation containers can run on a container platform; artifacts and execution traces can be archived to object storage; call logs can be aggregated into a log service; and runtime and cost can be monitored through alerting services that trigger notifications when thresholds are crossed.

    5.3 Log the Full Trajectory: Failed Samples Are More Valuable Than Passed Ones

    Beyond pass/fail, record the full tool-call sequence, token consumption, per-stage timing, and failure reasons. This data can be used to:

    • locate runtime bottlenecks (model inference, build, or test execution);
    • analyze failure modes (compile failure, test failure, timeout, cost overrun);
    • optimize prompts and the agent workflow.

    5.4 Statistical Design and Budget Guardrails

    • Multiple runs and interval estimates: ratio metrics like pass rate have high variance. Run the same solution multiple times, report confidence intervals, and cover the same batch of tasks. When conditions allow, a paired design (running the same tasks for each solution) more effectively determines whether a difference is significant.
    • Sanity checks: if a claimed pass rate cannot divide evenly into “passes / total,” it uses weighting or partial scoring; understand the rule before citing it.
    • Budget guardrails: set a wall-clock time and cost cap for each task, and score tasks that hit the cap as “not completed,” preventing a few runaway tasks from inflating overall cost.

    5.5 Evaluation Executor Skeleton

    The following Python pseudocode outlines the core idea of execution and aggregation:

    1. import math
    2. from concurrent.futures import ThreadPoolExecutor
    3. from dataclasses import dataclass, asdict
    4. @dataclass
    5. class Budget:
    6. max_wallclock_hours: float
    7. max_cost: float
    8. def run_one(task, agent, budget: Budget) -> dict:
    9. """Execute a single task in an isolated environment and return a structured result"""
    10. sandbox = create_sandbox(image=task.env_image) # isolated container environment
    11. try:
    12. result = agent.run(
    13. prompt=task.prompt,
    14. sandbox=sandbox,
    15. budget=budget,
    16. )
    17. passed = run_acceptance(result, sandbox) # build + tests + checklist
    18. return {
    19. "task_id": task.task_id,
    20. "category": task.category,
    21. "passed": passed,
    22. "completion": estimate_completion(result), # checkpoint-based completion
    23. "wallclock_hours": result.wallclock_hours,
    24. "cost": result.cost,
    25. "terminated_by_budget": result.hit_budget,
    26. }
    27. finally:
    28. sandbox.destroy() # reclaim resources promptly, success or failure
    29. def summarize(records: list[dict]) -> dict:
    30. """Summarize metrics and return the 95% confidence interval for pass rate (normal approximation)"""
    31. n = len(records)
    32. assert n > 0, "task set cannot be empty"
    33. passes = sum(r["passed"] for r in records)
    34. pass_rate = passes / n
    35. ci_low = max(0.0, pass_rate - 1.96 * math.sqrt(pass_rate * (1 - pass_rate) / n))
    36. ci_high = min(1.0, pass_rate + 1.96 * math.sqrt(pass_rate * (1 - pass_rate) / n))
    37. def mean(key):
    38. return sum(r[key] for r in records) / n
    39. avg_cost = mean("cost")
    40. return {
    41. "sample_size": n,
    42. "pass_rate": pass_rate,
    43. "pass_rate_ci95": [ci_low, ci_high],
    44. "avg_completion": mean("completion"),
    45. "avg_wallclock_hours": mean("wallclock_hours"),
    46. "p95_wallclock_hours": sorted(r["wallclock_hours"] for r in records)[math.ceil(0.95 * n) - 1],
    47. "avg_cost": avg_cost,
    48. # average cost per successful task: failed tasks consume resources too
    49. "cost_per_pass": avg_cost / pass_rate if pass_rate else float("inf"),
    50. # failure attribution, used to locate capability gaps
    51. "fail_reasons": {
    52. reason: sum(
    53. 1 for r in records
    54. if not r["passed"] and r.get("fail_reason") == reason
    55. )
    56. for reason in {r.get("fail_reason") for r in records if not r["passed"]}
    57. },
    58. }
    59. def evaluate(tasks, agent, budget_factory, repeats=1, max_workers=4) -> list[dict]:
    60. """Run the same task set repeats times, aggregate to task level; repeat is only for statistical stability"""
    61. all_records = []
    62. with ThreadPoolExecutor(max_workers=max_workers) as pool:
    63. for task in tasks:
    64. runs = list(pool.map(
    65. lambda _r, t=task: run_one(t, agent, budget_factory(t)),
    66. range(repeats),
    67. ))
    68. all_records.extend(runs)
    69. # aggregate by task: a task passes if any of its runs passes
    70. task_level = {}
    71. for r in all_records:
    72. t = task_level.setdefault(r["task_id"], {"passed": False})
    73. t["passed"] = t["passed"] or r["passed"]
    74. # keep raw records for offline analysis
    75. return all_records, task_level

    Three details in this code are worth noting:

    1. Failure attribution: fail_reasons groups failures by cause (compile failure, test failure, timeout, cost overrun, etc.). Compared with a single pass rate, it better reveals which stage the capability gap is in.
    2. Paired comparison: to implement paired comparison, join records from different solutions by task_id before aggregating, then compute the performance difference on the same task. This reflects the real gap better than comparing two independent population means.
    3. Aggregation rule: here, “a task passes if any run passes” is only used to illustrate the aggregation idea. When citing external data, you must first confirm the scoring rule and must not apply this directly.

    5.6 Tiered Routing: Encode the Selection Strategy into Agent Configuration

    You can distill selection decisions into versionable routing rules, for example:

    1. routing:
    2. - match: {category: [chore, test], risk: low}
    3. use: lightweight
    4. fallback: long_horizon # escalate on failure or repeated non-passes
    5. - match: {complexity: high, risk: high}
    6. use: human_led # human-led, agent-assisted
    7. assist: long_horizon
    8. budgets:
    9. lightweight: {max_wallclock_hours: 4, max_cost: 5}
    10. long_horizon: {max_wallclock_hours: 32, max_cost: 50}

    Note that agents are probabilistic systems. However carefully a solution is designed, it must still pass testing, code review, and security scanning before it can be merged into the main branch. A human in the loop is not a loss of efficiency but part of the quality system.

    6. A Selection Checklist: Six Questions to Answer Before Deciding

    Before recommending a category of solution to your team, answer these six questions first:

    1. What is my task distribution? What share of tasks is long-horizon? What share is high-risk?
    2. What is the real cost of a failed task? Include human rework, delivery delays, and opportunity cost.
    3. How long a delivery cycle can I accept? Can the tail latency of the long-horizon solution be absorbed by the business process?
    4. What is the budget cap? Is it a per-run cap, a monthly cap, or priced per successful task?
    5. Are the evaluation rules aligned? Are task definitions, environment versions, run counts, and scoring and cost rules consistent?
    6. Is there a regression mechanism? Can the baseline be rerun automatically after each model or agent version change?

    7. Conclusion

    The two new categories in this leaderboard correspond neatly to two typical engineering needs:

    • The long-horizon solution serves complex, high-value tasks with higher pass rate and completion.
    • The lightweight solution serves high-frequency, budget- and time-sensitive tasks with lower per-unit cost and shorter runtime.

    The key point is that this comparison is not simply a matter of “high vs. low,” but a quantified presentation of differently positioned solutions under a unified benchmark. Its real takeaway for developers is:

    1. Read the metrics carefully. Pass rate, completion, runtime, and cost each have boundaries; only reading them together avoids misjudgment.
    2. Compare from the same source. Conclusions are useful only when evaluation rules are consistent, and statistical variance cannot be ignored.
    3. Match selection to the business. Using real task distribution and human fallback cost as the basis, build a sustainable internal evaluation system with regular regression, so that agents become a genuine increment to engineering efficiency rather than a new cost black box.