Long-Horizon Agent Evaluation: From Leaderboards to Capability Diagnostics

    Long-horizon agent evaluation is moving beyond single-turn question accuracy toward the quality of multi-step task completion. This article explains why leaderboard rankings alone can be misleading and how teams can build a diagnostic evaluation system based on stratified task sets, blind testing, net improvement rates, granular capability metrics, failure attribution, and reproducible regression pipelines. It covers key challenges such as trajectory-level correctness, tool-environment uncertainty, long-context constraint retention, and cost-latency trade-offs, and it provides a practical pseudocode framework for task execution, metric collection, and net improvement calculation. Readers will learn how to turn agent evaluation results into actionable engineering signals rather than relying on a single ranking.

    Agent evaluation is shifting from single-turn question-answering accuracy to the quality of long-horizon task completion. Public benchmark rankings provide useful horizontal reference points, but the task distribution, inference budget, tool permissions, scoring criteria, and sample composition behind a ranking often affect conclusions more than the rank itself. For R&D teams, the valuable question is not “where does a model rank?” but “on which tasks is it stable, at which stage does it fail, whether failures are recoverable, and whether improvements are reproducible.” Evaluation therefore needs to move from consuming a single leaderboard to building a capability diagnostics system.

    1. Core Challenges in Long-Horizon Agent Evaluation

    Long-horizon tasks typically span dozens or more interaction steps and involve planning, tool calls, state maintenance, error recovery, and safety constraints. Compared with single-turn Q&A, their evaluation challenges fall into four main areas.

    First, task completion is not about single-point correctness but trajectory correctness. An agent must maintain goal consistency across multiple steps and avoid irreversible side effects introduced by intermediate actions. A task whose final answer appears correct may still involve unauthorized calls, duplicate execution, or state contamination. Scoring should therefore cover the process trajectory, not only the final state.

    Second, tool environments introduce uncertainty. Command execution, file operations, and external queries may time out, return exceptions, or produce partial success. Evaluation needs to distinguish model decision errors from environment fluctuation, and improve reproducibility through sandboxing, idempotent design, retry strategies, and failure injection.

    Third, long contexts and long-term constraints are easily diluted. When users change goals mid-task, add constraints, or revoke earlier instructions, the agent must identify conflicts and correct its plan. This capability is difficult to measure with final accuracy alone and requires dedicated controlled tasks and fine-grained metrics.

    Fourth, cost and latency cannot be ignored. If a long-horizon task achieves a high score through unlimited retries or extremely long reasoning, it may not be practical in engineering terms. Evaluation should also record steps, tool-call counts, token consumption, end-to-end latency, and failure retry rates to form a joint profile of quality and efficiency.

    2. Benchmark Design: Stratified Sampling and Blind Testing

    An interpretable benchmark must first define task stratification. Common strata include office and business workflows, coding and engineering tasks, and dialogue and knowledge collaboration. Each category should be further divided by difficulty, tool dependency, constraint complexity, and recoverability so that one task class does not overrepresent overall capability. For open-weight and closed models, results should be reported by group: open-weight categories should note quantization method, inference runtime, context length, hardware budget, and toolset; closed categories should note call budget and rate-limit conditions. Cross-group comparisons are meaningful only under equivalent budgets and permissions.

    Second, samples should come from anonymized trajectories of real long-horizon task sessions and should undergo desensitization, deduplication, and difficulty calibration. Real sessions can expose ambiguous instructions, temporary changes, and abnormal recovery scenarios that lab tasks rarely cover, but they also require protection against privacy leakage and data contamination. A combination of a “fixed golden set + dynamic holdout set” is recommended: the golden set is used for version regression, while the dynamic holdout set reduces overfitting and memorization risk.

    Blind testing is key to improving credibility. Evaluation should hide model identity, randomize task order, freeze task versions, and hide candidate sources from raters or automated graders. Automated scoring is suitable for structured checks such as command execution results, file state, and constraint satisfaction; human scoring is suitable for judging plan reasonableness, explanation quality, and correction behavior. The two should be cross-validated, and agreement metrics should be reported.

    3. Metric System: Composite Net Improvement Rate and Granular Capabilities

    Composite metrics are suitable for judging version trends, but they should not replace granular diagnostics. The net improvement rate can use the formulation “number of winning samples minus number of losing samples divided by number of valid comparison samples,” with confidence intervals and sample sizes reported alongside it. Its advantage is that it directly reflects relative improvement; its disadvantage is that it can hide differences across task categories, so it must be interpreted together with stratified metrics.

    Granular metrics should cover at least the following dimensions:

    • Task completion rate: whether the final state meets goals and constraints and whether unacceptable side effects occur.
    • Planning consistency: whether intermediate steps serve the final goal and whether there is repeated oscillation or ineffective looping.
    • Command execution reliability: for command-line tools, focus on execution success rate, output parseability, failure recoverability, idempotency, and side-effect control.
    • Controllable correction capability: whether the agent can identify conflicts and adjust its plan after a mid-task goal change, added constraint, or revoked instruction.
    • Error recovery capability: whether the agent can locate the problem and adopt safe retries or alternative paths when timeouts, abnormal returns, or partial success occur.
    • Context retention capability: whether long-term constraints, user preferences, and confirmed facts are still correctly referenced after multiple rounds of interaction.
    • Safety and compliance: whether unauthorized operations are refused and whether least-privilege and sensitive-data protection requirements are followed.
    • Efficiency cost: steps, tool-call counts, token consumption, and latency required to complete equivalent tasks.

    Metric design should avoid a “completion-rate-only” approach. For example, a strategy may achieve a higher completion rate through many retries, but if command execution reliability is low and side-effect risk is high, it should not be considered superior in engineering terms. Different task categories should be weighted differently, and reports should show means, percentiles, and failure distributions together.

    4. From Ranking to Diagnosis: Result Interpretation and Engineering Adoption

    Ranking results need interpretation before they can enter model selection and R&D loops. The first step is attribution: classify failures into planning errors, tool selection errors, parameter errors, state loss, recovery failure, safety refusal, and budget exhaustion, and calculate the share of each category. The second step is stratification: observe performance separately on office workflows, coding engineering, and dialogue collaboration so that an overall rank does not mask local weaknesses. The third step is regression: establish controlled experiments for each model, prompt, tool description, or runtime change, and use golden task sets and confidence intervals to determine whether an improvement is significant.

    In engineering terms, the evaluation pipeline should be versioned, replayable, and auditable. Task sets, graders, tool sandboxes, and model configurations should all have version numbers; each evaluation should retain full trajectories, tool returns, and scoring details; anomalous samples should support replay for localization. Infrastructure can be assembled from general categories such as container platforms, object storage, logging services, and monitoring and alerting. On Baidu AI Cloud, these general capabilities can also be organized into a pipeline; the focus is interface abstraction and data auditability rather than binding to proprietary interfaces.

    5. Example of an Actionable Evaluation Pipeline

    The following product-agnostic pseudocode framework illustrates the basic structure for task execution, metric collection, and net improvement calculation:

    1. def evaluate_task(task, policy, budget):
    2. state = init_state(task)
    3. trajectory = []
    4. for step in range(budget.max_steps):
    5. action = policy.plan(state)
    6. result = env.execute(action) # Execute tool actions in a sandbox
    7. trajectory.append((state, action, result))
    8. state = update_state(state, result)
    9. if task.is_done(state) or task.is_failed(state):
    10. break
    11. metrics = score(trajectory, task)
    12. return metrics, trajectory
    13. def net_improvement(wins, losses):
    14. valid = wins + losses
    15. if valid == 0:
    16. return 0.0
    17. return (wins - losses) / valid

    In this framework, score should return granular items such as completion rate, constraint satisfaction, command execution reliability, correction success rate, and safety violations; net_improvement should be used only as a summary metric and must be accompanied by sample size and confidence intervals. For long-horizon tasks, per-step budget consumption and failure recovery paths should also be recorded so that “insufficient capability” and “insufficient budget” can be distinguished.

    6. Conclusion

    The essence of long-horizon agent evaluation is to use reproducible tasks, controllable environments, and interpretable metrics to answer whether capability is stable, whether improvement is real, and whether risk is controllable. Public rankings can serve as an observation window, but they cannot replace a diagnostic system. R&D teams should build an evaluation framework that combines stratified task sets, blind testing, net improvement rates, and granular metrics, and should incorporate failure attribution and regression monitoring into daily engineering workflows. Only then can teams avoid being misled by a single rank and turn evaluation results into actionable optimization directions.