Long-Horizon Agent Evaluation: Balancing Task Completion and Cost
This article explains how to evaluate AI agents on long, multi-step tasks that depend on external tools and may change direction midway. It covers four signals (completion, instruction following, steerability, and tool reliability), how to read cost alongside effectiveness, how to interpret version-to-version gains, and a minimal workflow for teams building their own evaluations.
Long-horizon agent evaluation asks whether a model can finish a task, not whether it gives a polished single answer. The question is whether it completes multi-step work that can be corrected along the way and depends on external tools. Compared with standardized question banks, this kind of evaluation is closer to real engineering work. Goals change mid-task, context grows longer, and failures usually come from tool calls, lost state, or slow responses to correction instructions rather than from a single wrong multiple-choice answer.
Evaluate capabilities, not leaderboard positions
Mixing up “lab ranking” with “overall ranking of a single model” distorts conclusions. At the organization level, cumulative results reflect the overall level of several model lines and multiple version iterations over a period of time. A single model’s ranking reflects the relative position of one callable version on the current sample. The two answer different questions and cannot replace each other.
A more reliable approach is to fix the evaluation slice: the same task distribution, the same tool set, the same success criteria, and the same cost accounting. Then compare versions. Gains between versions should be decomposable. For example, a higher completion rate might come from better planning, or from a more aggressive retry strategy that also raises cost. A single overall score cannot support engineering decisions.
What long-horizon tasks actually measure
Long-horizon tasks usually include goal understanding, plan decomposition, tool selection, verification of intermediate results, failure recovery, and final delivery. An evaluation should cover at least four signals.
The first is task completion. Distinguish between “the system believes it has finished” and “the user explicitly confirms completion.” The former is easy to game through wording optimization. The latter is closer to business value, but samples are sparser and labeling is more expensive. Keep both, and avoid relying only on model self-assessment.
The second is instruction following. This covers whether constraints are violated, whether the output format stays stable, and whether prohibited actions are triggered. In long conversations, early constraints are often overridden by later content. Short static questions rarely expose this problem.
The third is user feedback polarity. Praise and complaints cannot replace completion rates, but they reflect interaction friction. A model might eventually get the task right while repeatedly misunderstanding the user, asking for irrelevant information, or giving steps that cannot be executed.
The fourth is steerability, meaning whether the model corrects itself in later steps after a user points out an error, instead of repeating the same plan. A system with poor steerability pushes the cost of manual correction onto the user, so its nominal automatic completion rate is overstated.
Measure tool reliability separately
Agent failures often happen at the tool boundary rather than the language boundary. Take command execution tools as an example. Record whether each call respects permission and parameter constraints, whether the return value is interpreted correctly, whether the agent retries blindly after a failure, and whether it writes temporary state to an unrecoverable location. Reliability metrics should be based on observable logs, not impressions after the fact.
At minimum, record: call success rate, share of tasks succeeding on the first attempt, number of invalid retries, misreading rate for erroneous outputs, and task abort rate caused by tool failures. If a preview-to-release change improves language metrics while tool metrics stay flat, you cannot claim that the agent’s overall capability has improved.
A summary schema can be simple. The key is keeping the definitions stable:
task_id, model_version, confirmed_success,instruction_violations, steer_events, steer_fixed,tool_calls, tool_failures, invalid_retries,latency_ms, token_in, token_out, cost
For aggregation, use medians to describe cost and latency, so a few extremely long sessions do not skew averages. For completion rates, report confidence intervals or at least sample sizes. Over 10,000 sessions do not automatically make results statistically valid. If task types are highly concentrated, generalization remains limited.
Plot cost and effectiveness together
Per-task cost is usually determined by input and output tokens, tool calls, and retries together. Comparing accuracy alone favors strategies with unlimited retries and unlimited context. Comparing unit price alone favors short answers that barely call tools. A sounder approach is to observe the Pareto frontier on an effectiveness-cost plot. A point that no other point beats across the board without higher cost sits on the frontier. It represents a representative trade-off within the current sample, not an absolute optimum.
When comparing cost, lock pricing, caching policy, retry limits, and whether human takeover is included. A median cost that rises slightly might only mean tasks got harder or tool outputs got longer. Do not read that directly as “the model got more expensive” or “the model got stronger.” For cross-model comparisons, publish the cost breakdown, not just a total.
Reading version iterations without overinterpreting
For a preview-to-release change, break results down by metric: confirmed success, feedback polarity, steerability, command or tool reliability, and unit cost. If one metric improves substantially while another stalls, the optimization is local. Rankings shift with the set of models evaluated, user sessions, and sampling window. A ranking at one point in time should not be extrapolated to every business scenario.
Open-weight and proprietary deployments also change comparability. When weights can be inspected, the community can check for data contamination, reproduce experiments, and replace tool adaptation layers. Proprietary interfaces depend more on the provider’s version freezing, log fields, and billing transparency. Either way, evaluation reports should state version identifiers, tool permissions, the task time range, and exclusion rules.
A minimal closed loop for engineering teams
Teams can build their own agent evaluation in the following order without waiting for external leaderboards.
- Sample long-horizon tasks from real tickets. Keep the goal, constraints, available tools, and acceptance owner.
- Define success in advance: how much weight goes to user confirmation, automated checks, and manual spot checks.
- Freeze the model version, temperature, retry limit, and tool allowlist. Do not change them mid-evaluation.
- Track completion, instruction following, steerability, tool reliability, and median cost separately.
- Use an effectiveness-cost scatter plot to find the frontier, rather than sorting by a single composite score.
- For each release, claim only improvements verified repeatedly on the same distribution, and keep failure cases.
These conclusions can guide routing. High-risk, low-tolerance tasks should use configurations with higher completion and tool reliability. Batch tasks that can be rolled back should use configurations with lower unit cost and acceptable steerability. The value of evaluation lies in turning “looks stronger” into auditable trade-offs, not in producing a permanent ranking detached from the task distribution.
