Long-Horizon Agent Evaluation: Metrics Design and Practical Methods
As LLM agents move from single-turn question answering to continuous task execution, evaluation must shift from answer correctness to process reliability. This article presents a practical framework for long-horizon agent evaluation, covering scope definition, real-session and blind-test data design, five core metric dimensions, net improvement rate calculation, common pitfalls, and an implementation workflow.
Long-Horizon Agent Evaluation: Metrics Design and Practical Methods
I. From Single-Turn Q&A to Long-Horizon Tasks
When a model moves from answering questions to continuously executing tasks—such as reading files, calling command-line tools, modifying code, or collaborating with a user across multiple turns—the object of evaluation changes. In the past, the focus was on whether a single-turn answer was correct. Now the focus must be on whether the entire process is reliable.
Single-turn benchmarks mainly reflect knowledge and reasoning ability. They struggle to expose problems such as errors accumulating across multiple steps, context drifting during long conversations, and the model failing to adjust in time when the user changes requirements mid-task. Evaluation therefore needs to cover the full task lifecycle, observing the process segment by segment from task start to acceptance.
II. Define the Evaluation Scope First
Before formal evaluation, three things need to be agreed upon.
- Task types: For example, document processing, software development, data analysis, and daily office assistance. Different types have very different requirements for tool calling and state management, so they should be counted separately.
- Comparison targets: Usually the old and new versions of the same product, or similar solutions compared under the same conditions. The comparison target must be fixed; otherwise, the conclusions cannot be traced.
- Judgment criteria: Who judges, on what basis, and how disagreements are handled.
When the scope is unclear, conclusions are easily misread as valid for all scenarios. Writing down the scope is itself part of the evaluation report.
III. Data Sources: Real Sessions and Blind Testing
The advantage of real sessions is that the task distribution is close to actual usage, and task chains are often longer, exposing problems that synthetic benchmarks frequently overlook. For example, a constraint casually added by the user may be violated more than a dozen turns later.
Real data also introduces bias risks, which need to be controlled through the following design choices:
- Blind testing: Hide the identity of the tested version and randomize left/right positions to avoid evaluator preconceptions based on brand or version number.
- Multiple independent evaluators: Have at least two evaluators judge the same task independently, then aggregate the results.
- Consistency checks: Calculate inter-rater agreement, for example using Cohen’s kappa to measure agreement between two evaluators. If consistency is low, revise the scoring rubric first before collecting more data.
- Sample representativeness: Use stratified sampling by task type to avoid overrepresentation of simple tasks.
- Report confidence intervals: When sample size is limited, point estimates fluctuate. Report intervals as well, rather than giving only a single number.
IV. Decomposing Core Metrics
The quality of long-horizon tasks is usually determined by multiple dimensions, and a single metric is hard to summarize. The following five dimensions are a commonly used analytical framework.
1. Command Execution Reliability
This metric measures the agent’s performance when calling the command line or other execution environments, including whether command syntax is correct, whether parameters are reasonable, and whether the agent can locate the cause and recover after a failure.
Quantifiable submetrics include:
- First-attempt success rate: The proportion of commands that succeed in one attempt.
- Retry rate: The proportion that requires retry due to syntax errors, path errors, or permission issues.
- Success rate after retry: The proportion that ultimately succeeds after retry, used to distinguish “recoverable” from “unrecoverable.”
- High-risk operation rate: The number of times destructive parameters such as delete, overwrite, or force push are misused.
When evaluating this item, commands must be executed in an isolated sandbox, and an operation allowlist should be predefined to prevent accidental damage to the evaluation environment.
2. Direction Adjustment and Correction Capability
It is common for users to change requirements mid-task. This metric measures whether the model can accurately understand new constraints and withdraw or correct previous changes that no longer meet the requirements.
Key observations:
- After a correction request, does the next output meet the new requirements?
- Does the model repeat the same type of error?
- After correction, does the model still retain residual content from the old constraints?
Judgment should be based on the user’s latest instruction, and the turn in which the correction occurs should be recorded. The later the correction happens, the higher the rework cost is usually.
3. State Consistency
Multi-step tasks rely on persistent memory of files, variables, and confirmed information. State consistency measures whether the model keeps this information correct and stable throughout the process.
Typical problems include: a filename confirmed earlier is changed later, an option the user has already excluded is recommended again, or numerical values contradict one another after multiple calculations. Reviewers can compare the agreed items at the start of the session and check subsequent outputs one by one.
4. Task Completion Quality
This item focuses on whether the final output meets the acceptance conditions, while also examining whether there are redundant or destructive operations during the process. The two should be scored separately: a correct result with a messy process and a correct result with a clean process represent different levels of usability.
Acceptance conditions should be written before the task starts, for example, “the generated script passes the specified tests” or “the document contains all required sections.” When acceptance conditions are vague, evaluator disagreement increases significantly.
5. Efficiency and Cost
Efficiency metrics include the number of tool calls, total number of steps, time consumed, and token consumption. There is often a tradeoff between efficiency and quality: reducing the number of calls may lower verification frequency, increasing the risk of missed errors. Therefore, efficiency metrics should only be interpreted together with quality metrics and should not be used alone as a ranking basis.
V. Net Improvement Rate: Definition and Calculation
In version iteration comparisons, the “net improvement rate” is commonly used to summarize the overall change of a new version relative to a baseline.
Definition: For the same batch of tasks, perform pairwise comparisons, count the number of wins and losses for the new version, and divide the difference by the total number of tasks. Ties count in the denominator but not as wins or losses. Values range from -1 to +1; multiply by 100 to get a percentage.
This metric reflects direction better than win rate alone. If ties account for a high share, the win rate can remain near 50% even when there is a noticeable win-loss difference, while the net improvement rate will be closer to 0, which better matches the actual situation.
The calculation logic is illustrated below (pseudocode):
function calculateNetImprovementRate(judgmentResults):total = length(judgmentResults)if total == 0: throw error "Results are empty"wins = count judgmentResults == "new version"losses = count judgmentResults == "baseline"return (wins - losses) / total
Example: Among 100 tasks, the new version wins 35, loses 20, and ties 45. The net improvement rate is (35 − 20) / 100 = 15%. This number only describes the net change relative to the baseline; it does not represent the model’s absolute capability.
When using the net improvement rate, note the following limitations:
- It is a relative metric and depends on baseline selection. If the baseline changes, values are not directly comparable.
- Different task-set compositions produce different results. Before cross-table comparison, confirm that the task sets are consistent.
- Report component metrics as well, so that a composite score does not hide regression in a particular dimension.
VI. Common Pitfalls in Iterative Comparisons
- Sample leakage: Tasks used during tuning or training should not be reused for evaluation. Otherwise, the results reflect memorization rather than generalization ability.
- Evaluator bias: If evaluators know the version identity, they tend to favor the new version. Blind testing is necessary.
- Metric gaming: Optimizing only one metric may harm other capabilities. For example, overemphasizing first-attempt success rate may cause the model to avoid necessary verification steps.
- Environment drift: Changes in tool versions, dependency libraries, or network conditions all affect results. The evaluation environment should be fixed, and version information should be recorded.
- Long-tail neglect: Reporting only average performance hides a small number of costly failures. High-risk failure cases should be listed separately and analyzed one by one.
VII. Implementation Workflow
The following steps are recommended:
- Define the task set and scoring rubric, and clarify acceptance conditions and judgment criteria.
- Build an isolated sandbox and save complete execution traces for replay and review.
- Organize blind tests, arrange multiple evaluators to score independently, and calculate consistency.
- Calculate the net improvement rate and component metrics, and provide confidence intervals.
- Attribute failure samples by distinguishing model capability issues, environment issues, and scoring issues, and form an improvement list.
- Freeze the task set before the next iteration begins to ensure that different versions are compared on the same baseline.
VIII. Conclusion
The difficulty of evaluating long-horizon agents lies in complex processes and frequent disagreements. An evaluation system with clearly defined metrics, reliable data sources, and a reproducible judgment process can provide a more stable basis for version decisions and help teams identify which areas should be optimized next.
