Long-Horizon Agent Evaluation: How to Balance Pass Rate and Cost

    This article explains what long-horizon agent benchmarks measure, how to standardize key metrics such as pass rate, completion, latency, and cost, and why a higher pass rate may come with longer runtime or higher expense. It also covers when lightweight, fast agents are suitable, how to build reproducible evaluation pipelines, and how to translate evaluation metrics into routing policies, budget guardrails, and observable decision-making systems.

    1. What Do Long-Horizon Benchmarks Actually Measure?

    A programming benchmark for mobile application development is not simply measuring code completion accuracy. It places models into a quasi-realistic engineering timeline and observes whether they can independently advance a multi-stage deliverable. Long-horizon tasks in such benchmarks usually cover four typical scenarios:

    • Application creation: Starting from an empty project and setting up directory structure, build configuration, basic UI scaffolding, and dependency management.
    • Feature development: Adding business modules to an existing codebase while understanding current conventions and keeping the style consistent.
    • Code migration: Porting an implementation from one technology stack to another while preserving behavioral equivalence.
    • Application conversion: Refactoring across platforms or architectural forms, often involving adjustments to state management, lifecycle, and resource models.

    Compared with short Q&A evaluations, the key difference is that long-horizon tasks do not converge within a single turn. The agent goes through a cycle of read code → generate → compile → fail → diagnose → fix → rerun. As a result, evaluation outcomes are affected by four types of factors: model reasoning capability, tool-call reliability, context-management ability, and external build-environment stability.

    More precisely, tool-call reliability is often intertwined with model capability, agent scaffolding, and search strategy. It should not be treated as a completely independent variable. When building your own evaluation, you should combine observations such as termination reasons, average iteration counts, and build failure rates to determine whether differences come from model capability, engineering implementation, or environmental fluctuation.

    2. How Should Metrics Be Standardized Before Horizontal Comparison?

    Before interpreting any leaderboard score, metric definitions should be standardized first. Otherwise, horizontal comparisons can easily become misleading.

    Metric Meaning Main influences Common misinterpretation
    Pass rate Proportion of tasks judged fully successful; a binary result Model capability, toolchain, validation strictness Treating pass rate as the only objective
    Average completion score Weighted progress score for unfinished tasks; shows how close the task is to completion Generation quality, whether iteration converges Ignoring rubric weights and judgment criteria
    Average elapsed time End-to-end wall-clock time, not model inference time Iteration count, build duration, retry strategy Mistaking build-environment differences for model capability
    Cost per run Compute cost required to complete one benchmark run Interaction turns, context size, retry count Looking only at token unit price without total turns

    The aggregation logic can be expressed in pseudocode, which is useful for reusing the approach in self-built evaluations:

    1. for task in task_set:
    2. result = run_agent(task, budget=max_turns, timeout=max_hours)
    3. passed[task] = judge(result, test_suite) # 0 / 1
    4. completion[task] = weighted_score(result, rubric) # [0, 1]
    5. wall_time[task] = result.end_ts - result.start_ts
    6. cost[task] = sum(turn.token_in * p_in + turn.token_out * p_out)
    7. pass_rate = mean(passed)
    8. completion = mean(completion)
    9. cost_per_pass = sum(cost) / max(1, sum(passed))

    Here, cost_per_pass represents the actual cost for each successfully completed task. It is often more useful for decision-making than comparing cost per run alone.

    A statement such as “when pass rate improves from 24% to 28%” should specify the benchmark name and version, task-set size, measurement period, and number of runs if it is used for formal selection. If the source cannot be provided, it should be marked as illustrative data, not measured.

    For example, assume a self-built task set has 50 tasks, each run 3 times and averaged. Scheme A has a pass rate of 24%, while Scheme B has a pass rate of 28%. The following is illustrative data, not measured:

    • Scheme A: about 12 passed tasks, monthly total cost 120, completion weighting factor 0.42.
    • Scheme B: about 14 passed tasks, monthly total cost 240, completion weighting factor 0.86.

    The cost per pass can be calculated as:

    1. cost_per_pass_A = 120 / (50 * 24%) = 10
    2. cost_per_pass_B = 240 / (50 * 28%) ≈ 17.1

    If only pass rate is considered, Scheme B looks better. However, when looking at completion-weighted cost:

    1. completion_weighted_cost_per_pass_A = 120 / (50 * 24% * 0.42) ≈ 23.8
    2. completion_weighted_cost_per_pass_B = 240 / (50 * 28% * 0.86) ≈ 19.9

    This shows that pass rate, completion, latency, and cost should be viewed together as a four-part tuple. Ranking by any single metric may lead to the opposite conclusion.

    3. Why Does Higher Completion Often Come With Longer Elapsed Time?

    A common illustrative pattern in public benchmarks is that a certain type of agent achieves higher completion on long-horizon tasks but has noticeably longer average elapsed time. Because the statement does not specify a leaderboard, version, task-set size, measurement period, or number of runs, it should be understood as illustrative data, not measured.

    Usually, this is not a defect but the result of a search strategy. However, if the root cause is an unstable toolchain, poor context management, or invalid loops, then it should be considered an implementation defect. To distinguish these cases, look at termination reasons and average iteration counts. If many tasks end because of normal completion but with high iteration counts, this may indicate deeper exploration. If tasks frequently hit turn limits, time out, or encounter environment errors, the issue is more likely related to implementation or environment.

    This phenomenon can be understood at three levels.

    1. Exploration depth is positively correlated with elapsed time. A higher-completion implementation path usually means the agent performs more hypothesis-verification cycles. It does not stop after the first compile failure. Instead, it continues reading stack traces, tracing call chains, and trying different implementation branches until tests pass or the budget limit is reached. Every extra exploration step is paid for in wall-clock time.

    2. Context growth introduces nonlinear overhead. In long-horizon tasks, the context window continuously accumulates build logs, error traces, and historical patches. As context size increases, inference latency and token consumption rise together, creating a curve where later stages become slower and more expensive. This also explains why elapsed-time distributions often have long tails: a small number of tasks repeatedly fall into repair loops and pull up the average.

    3. Convergence strategies differ under budget constraints. Conservative agents tend to “try several approaches before finalizing,” while aggressive agents tend to “generate once and deliver quickly.” The former usually achieves higher completion with fewer failed tasks, while the latter has shorter elapsed time and lower cost but leaves more tasks in a partially completed state.

    These two approaches correspond to different product positioning. The routing logic can be expressed as:

    1. if task_difficulty is high and delivery_value_per_run is high:
    2. choose a high-completion strategy that can tolerate longer elapsed time
    3. elif task_volume is large and fast feedback is required:
    4. choose a low-latency, low-cost strategy
    5. else:
    6. use tiered routing rather than a single model

    4. What Tasks Are Suitable for Low-Cost, Fast Solutions?

    A group of lightweight, fast-oriented solutions may lag behind top solutions by a few percentage points in pass rate. This comparison is illustrative data, not measured. If used for formal selection, the benchmark name and version, task-set size, measurement period, and number of runs should be supplemented.

    These solutions are not truly suited to the hardest migration and refactoring tasks. They are more suitable for high-frequency, low-risk scenarios that can be quickly verified by humans:

    • Boilerplate code and scaffolding generation, with templated build configuration.
    • Single-file modifications, copy or constant replacements, and logging instrumentation.
    • Unit tests and test-case supplementation, as well as compile-time error fixes.
    • Code comments, documentation generation, and API description organization.
    • Mechanical and repetitive rewrites in migration tasks.

    For budget-sensitive teams that need short feedback loops, routing the majority of high-frequency, low-risk requests to a fast solution and escalating a small number of high-value complex tasks to a high-completion solution is usually a better performance-to-cost combination. However, the actual benefit depends on rework rate and routing accuracy. It should not be assumed that this approach is always cheaper in every scenario.

    The core premise is to establish a task triage mechanism, rather than measuring all tasks with the same ruler.

    5. How Can You Build a Reproducible Self-Constructed Pipeline?

    Benchmarks can be referenced, but self-built evaluations are the basis for reproducible engineering decisions. Reproducibility is the first principle. The following steps are recommended.

    (1) Task-set construction. Sample from a real requirement pool, retaining the original context and acceptance criteria. Each task should include an automatically executable validation script to avoid relying on subjective human judgment. Tasks should be stratified by difficulty and annotated with estimated effort ranges. The task set should cover at least the four scenarios: application creation, feature development, code migration, and application conversion.

    (2) Environment isolation and consistency. Each task should run in an independent sandbox. Build toolchain versions should be fixed, dependency repositories should use local mirrors or offline caches, and network egress should be controlled. Otherwise, differences in build duration will contaminate latency metrics and make horizontal comparisons meaningless.

    1. runtime:
    2. isolation: container_per_task
    3. toolcache: pinned_offline
    4. network: restricted_allowlist
    5. timeout_seconds: 7200
    6. budget:
    7. max_turns: 40
    8. max_cost_per_task: 1.0

    (3) Scorer design. Use a two-layer approach combining automatic tests and a structured rubric. Automatic tests are responsible for behavioral equivalence, while the rubric handles completion weighting, including whether the build passes, whether core paths are usable, whether exception handling is complete, and whether code style is consistent. The judging logic itself should be versioned to avoid metric drift.

    (4) Budget and failure handling. Explicitly define maximum iteration counts, timeout thresholds, and retry limits. Record the termination reason for each task, including normal completion, hitting the turn limit, timeout, and environment error. Data without termination reasons cannot be used for attribution analysis.

    (5) Metric aggregation and attribution. In addition to the four main metrics, at least three observation dimensions should be added: average iteration count, build failure rate, and task-level cost distribution at P50/P90. Distribution data is often more representative of real experience than averages alone.

    (6) Review report template. To align benchmark results with your own baseline, an evaluation report should include at least the following fields:

    • Benchmark name and version.
    • Task-set size and task stratification.
    • Measurement period.
    • Number of runs or pass@k.
    • Pass rate.
    • Average completion score.
    • cost_per_pass.
    • P50/P90 elapsed time.
    • P50/P90 cost.
    • Termination-reason distribution.
    • Exception task list and reproduction method.

    With these fields, a team can determine whether a benchmark score can be transferred to its own tasks and can avoid directly comparing results across different benchmarks or versions.

    6. How Can a Cost Model Become Routing and Guardrails?

    A complete selection decision requires converting metrics into comparable terms. Start by calculating total monthly cost:

    1. total_monthly_cost = task_volume * average_iterations * token_cost_per_turn
    2. + task_volume * environment_build_overhead
    3. + human_review_hours * human_cost_per_hour

    On this basis, two more granular unit-cost metrics can be calculated:

    1. cost_per_pass = total_monthly_cost / (task_volume * pass_rate)
    2. completion_weighted_cost_per_pass =
    3. total_monthly_cost / (task_volume * pass_rate * average_completion_weight)

    It should be noted that pass_rate * average_completion_weight is a heuristic measure, not a measured effective output. Different teams have different task difficulty, acceptance criteria, and rework costs. The weighting factor needs to be calibrated according to the task profile.

    The most easily overlooked cost here is human review effort. If a solution has a low cost per run but produces intermediate code that is hard to read and requires more manual rework, the real cost may be higher. Therefore, the human cost converted from review effort should be included in total cost in the numerator, and then divided by effective output. It should not be excluded from total cost or incorrectly placed into the denominator.

    At the engineering organization level, four practices are recommended:

    1. Tiered routing: Select different strategies based on task type and risk level. High-risk tasks should be forced through manual review gates.
    2. Budget guardrails: Set token and turn limits for each task to prevent a single task from iterating indefinitely and slowing down the pipeline.
    3. Result caching and reuse: Reuse build dependencies, code indexes, and similar task results to significantly reduce repeated overhead.
    4. Full-link observability: Bring each tool call, build output, and judgment result of the agent into logging services and monitoring alert systems, forming traceable evaluation data assets.

    A minimal set of log fields can include:

    1. task_id
    2. turn_index
    3. token_in
    4. token_out
    5. termination_reason
    6. judgment_score
    7. wall_time

    These fields map directly to evaluation observations: turn_index can be used to calculate average iterations; termination_reason and judgment_score can be used to calculate build failure rates; token_in, token_out, and wall_time can be used to generate task-level cost distributions and P50/P90 metrics.

    When implemented on the cloud side, such a pipeline is usually composed of several general capabilities: a container platform hosts isolated sandbox tasks, object storage preserves evaluation artifacts and historical baselines, a message queue handles task distribution and rate limiting, and logging services plus monitoring alerts provide visual attribution for latency and cost. The amount of modification depends on the completeness of the existing container, storage, messaging, and logging systems.

    It is also important to note that evaluation artifacts and logs may contain business source code, dependency credentials, and personal information. Storage should enable private access and encryption, logs should mask sensitive fields, and cross-team sharing should be restricted.

    7. What Should You Pay Attention to When Choosing an Evaluation Approach?

    Common misinterpretations in actual selection include the following:

    • Treating pass rate as the only objective. In long-horizon tasks, a few percentage points of improvement in pass rate are often accompanied by significant increases in cost and elapsed time. Marginal benefit should be judged in the context of business value.
    • Ignoring baseline comparability. Different evaluations may use different agent harnesses, build environments, and judgment criteria. Directly ranking them across benchmarks can easily lead to incorrect conclusions.
    • Looking only at averages and not distributions. Long-tail tasks determine the team’s real experience. P90 elapsed time and P90 cost often have more decision-making value.
    • Ignoring the deviation between the benchmark and real tasks. Benchmark tasks tend to favor automatically judgeable cases, while real requirements often involve ambiguous constraints and cross-team communication. Internal task sets are needed for calibration.

    Therefore, when choosing an evaluation approach, do not only ask “which model has a higher score?” Also ask: Which benchmark version is this score based on? How large is the task set? How many times was it run? Is the pass-rate improvement accompanied by increased cost and elapsed time? Do termination reasons include environment errors? These questions determine whether benchmark results can be converted into reliable engineering decisions.

    Conclusion: Use Task Profiles for Explainable Selection

    The value of these development-oriented benchmarks is not to produce a ranking of “who is stronger.” It is to quantify both the capability boundaries of agents on real long-horizon tasks and their resource costs.

    For engineering teams, the correct way to read them is: first characterize your own task profile, then use the four-part tuple of pass rate, completion, elapsed time, and cost to match strategies, and finally use tiered routing and budget guardrails to keep costs within an expected range.

    A four-dimensional scoring rubric can be used for task profiling, with each dimension scored from 1 to 3:

    Dimension 1 point 2 point 3 point
    Task difficulty Single-file, low-risk Multi-file, with dependencies Cross-module migration or complex refactoring
    Task volume Small single-request volume Medium batch processing Large-scale continuous development
    Acceptable latency Can tolerate tens of minutes Can tolerate minutes Requires second-level feedback
    Rework tolerance Can tolerate multiple manual edits Limited rework Must pass in one shot or be low-risk

    Routing example: If any dimension scores 3, or the total score is 10 or higher, prioritize a high-completion solution. If the total score is low and fast feedback is emphasized, prioritize a low-cost, fast solution. If the score is between these cases, use tiered routing and set manual review gates for high-risk tasks.

    The purpose of evaluation is to help teams make explainable, reproducible, and continuously optimizable technical selection decisions under clear trade-offs.