Evaluating AI Agents on Long-Horizon Tasks: Metrics, Rankings, and Engineering Practices

    This article explains why agent evaluation is shifting from single-turn Q&A to long-horizon tasks, and how to build a sound evaluation framework. It covers grouping strategies for open-weight and closed-source models, blind testing and unified sandbox environments, sampling and de-identification of real task sessions, and a practical metric system including net improvement rate, shell command reliability, steerability, efficiency, and safety. It also explains how to correctly interpret leaderboards and provides a reproducible evaluation pipeline with configuration and pseudocode examples, plus actionable recommendations for combining public leaderboards with internal regression benchmarks.

    Evaluating AI Agents on Long-Horizon Tasks: Metrics, Rankings, and Engineering Practices

    1. Why Agent Evaluation Is Moving from Q&A to Long-Horizon Tasks

    Traditional model evaluation focuses on single-turn Q&A, short text generation, or fixed question banks. These methods are useful for comparing language understanding and knowledge coverage, but they fail to reflect an agent’s overall capability in real-world environments. An agent must understand multi-turn goals, invoke external tools, maintain intermediate state, handle exceptions, comply with constraints added mid-task, and complete tasks within limited resources. A single task may involve dozens of environment interactions, and any failed command execution, state loss, or drift in direction can cause the entire task to fail.

    Evaluation systems therefore need to shift from “point-in-time capability” to “long-horizon task sessions.” Long-horizon does not simply mean many conversation turns; it means the task has persistent state, tool-call chains, error-recovery paths, and staged acceptance criteria. For example, a workflow task may require the agent to read data, run a script, validate the output, fix errors, and generate a report; a coding task may involve locating an issue, modifying code, running tests, and fixing based on failure messages; chat-style tasks emphasize constraint retention, context consistency, and responsiveness to user corrections.

    Public blind-test leaderboards and group rankings provide valuable horizontal reference points, but they cannot replace an organization’s own evaluations. A more robust path is to treat public leaderboards as an “external signal,” treat business task sets as the “internal benchmark,” and build an interpretable, reproducible evaluation loop through unified metrics, a unified sandbox, and continuous regression.

    2. Evaluation Framework Design: Grouping, Blind Testing, and Real-Session Sampling

    2.1 Open-Weight Models Need Their Own Group

    Open-weight models generally refer to models whose weights are accessible and can be deployed locally or fine-tuned privately; closed-source models are mostly offered as service APIs. The two are not fully comparable in terms of resource investment, deployment method, context length, and tool-calling strategy. A single overall leaderboard mixes “downloadable weights” with “served API calls,” which distorts the interpretation of rankings.

    A better approach is to set up layered groups: all participating models, open-weight models, regional model groups, and category groups. Group rankings answer “where a model stands among its peers,” while overall rankings answer “where it stands in absolute terms.” For example, an open-weight model might rank first within the open-weight group while sitting in the upper-middle range overall, and rank first across work, coding, and chat categories within the open-weight group. This kind of information is far more interpretable than a single overall ranking, because it shows both relative advantage and absolute gap at the same time.

    2.2 Blind Testing and a Unified Environment

    The core of blind testing is hiding model identity, randomizing task order, and unifying tool permissions, timeouts, and the sandbox environment. Otherwise, scorers may be influenced by brand expectations, and models may gain unfair advantages from differences in tool versions, network conditions, or resource quotas. Real-time blind-test leaderboards should also carry a snapshot timestamp, because new model releases, task-set updates, and scorer revisions all change the rankings.

    2.3 Sampling and De-identification of Real Long-Horizon Sessions

    Real user task sessions are more representative, but they introduce privacy, bias, and annotation-quality issues. Sampling should cover task types, difficulty levels, turn counts, tool-call density, and failure modes. De-identification should remove identity information, business secrets, and traceable identifiers. Annotation should preserve task goals, constraints, acceptance criteria, and key intermediate states. In terms of scale, more than 17,500 real long-horizon task sessions can serve as an observation sample, but you still need to check whether the task distribution is balanced to prevent one task category from dominating the total score.

    3. The Metric System: Net Improvement Rate and Fine-Grained Capabilities

    3.1 Task Success Rate and Net Improvement Rate

    Task success rate is the most intuitive metric, but score levels alone cannot tell whether a version upgrade truly helps. Net improvement rate is better suited for version comparison: run the old and new versions on paired tasks, count improved tasks, regressed tasks, and total tasks, then compute (improved − regressed) / total. It penalizes upgrades that “fix one problem while breaking two.”

    In an anonymized case, the net improvement rate for a generational upgrade rose from +0.9% to +5.1%, a gain of 4.2 percentage points. This signal indicates the new version delivers clearer net gains on paired tasks, but the conclusion should still be validated with confidence intervals, difficulty stratification, and fine-grained metrics—never on a single percentage point alone.

    3.2 Shell Command Reliability

    Shell command reliability measures an agent’s success rate, idempotency, error handling, and safety boundaries when executing shell commands in an isolated environment. Evaluation should go beyond exit codes: does the agent understand error messages, avoid repeating destructive operations, and take reasonable fallback actions after failures? For high-risk commands, confirmation or refusal should be required, with audit logs recorded.

    3.3 Steerability and Course Correction

    Steerability refers to whether the agent can retain historical state, detect conflicts, and switch smoothly when the user adjusts goals, adds constraints, or corrects direction mid-task. A highly steerable model neither mechanically repeats the old plan nor discards valid work already completed. Evaluation scenarios can include “injecting a constraint mid-task,” “changing acceptance criteria,” and “requiring an explanation for deviation,” while observing compliance rate, state-retention rate, and post-correction task success rate.

    3.4 Efficiency, Cost, and Safety

    Beyond correctness, record interaction turns, tool-call counts, latency, resource consumption, and safety-violation rates. A solution that completes the task but with excessive calls and unacceptable latency has limited engineering value. Safety metrics should cover privilege escalation, sensitive data leakage, destructive commands, and unauthorized external calls.

    4. How to Correctly Interpret Rankings and Leaderboards

    Rankings should be read together with grouping, timestamps, sample size, and confidence intervals. Ranking first in the open-weight group does not mean first overall; ranking 5th on a real-time blind-test leaderboard does not mean the model beats the 6th on every business task. Release-time snapshots and continuously updated leaderboards can also differ: a newly released model often stands out in a specific group at launch, but rankings shift dynamically as more models join and task sets are updated.

    Therefore, model selection should prioritize alignment with your business task distribution: if you mainly build coding agents, weight coding tasks more heavily; if you build workflow automation, focus on long-chain state retention and shell reliability; if you build chat assistants, focus on constraint retention and steerability. Public rankings are a reference; your internal regression set is the decision basis.

    5. A Reproducible Evaluation Pipeline with Code Examples

    A complete evaluation pipeline typically includes: task collection and de-identification, task orchestration, sandboxed isolated execution, automated scoring, human review, log aggregation, statistical reporting, and continuous regression. When building your own platform, you can use general-purpose cloud capabilities such as object storage, container platforms, log services, and monitoring/alerting—for example, the corresponding services on Baidu AI Cloud—to carry task data, isolated execution, log collection, and metric observability respectively. The specific architecture should follow your organization’s security requirements.

    Example configuration:

    1. evaluation:
    2. groups: [all_models, open_weights, region_group, category_group]
    3. sampling:
    4. source: real_long_horizon_sessions
    5. deidentify: true
    6. min_turns: 8
    7. blind:
    8. hide_model_identity: true
    9. randomize_order: true
    10. unify_tool_permissions: true
    11. metrics:
    12. - task_success
    13. - shell_reliability
    14. - steerability
    15. - net_improvement
    16. - safety_violation

    Net improvement rate can be expressed in pseudocode:

    1. function net_improvement(old_version, new_version, paired_tasks):
    2. improved = 0
    3. regressed = 0
    4. total = 0
    5. for task in paired_tasks:
    6. old_score = evaluate(task, old_version)
    7. new_score = evaluate(task, new_version)
    8. total += 1
    9. if new_score > old_score + epsilon:
    10. improved += 1
    11. else if new_score < old_score - epsilon:
    12. regressed += 1
    13. return (improved - regressed) / total * 100

    Shell command reliability can be tracked via success rate and interception of high-risk operations:

    1. function shell_reliability(session):
    2. total = 0
    3. success = 0
    4. unsafe = 0
    5. for step in session.tool_calls:
    6. if step.kind == "shell":
    7. total += 1
    8. result = execute_in_sandbox(step.command, timeout=step.timeout)
    9. if result.exit_code == 0 and result.completed:
    10. success += 1
    11. if is_destructive(step.command) and not step.confirmed:
    12. unsafe += 1
    13. return {success_rate: success / total, unsafe_count: unsafe}

    6. Practical Recommendations

    First, establish a dual-track mechanism of “public leaderboards + internal benchmarks” to avoid being swayed by single ranking fluctuations. Second, compute group statistics for open-weight and closed-source models separately while retaining a combined view. Third, use net improvement rate as a gate for version upgrades, and pair it with fine-grained metrics to locate the source of regressions. Fourth, long-horizon tasks must run inside an isolated sandbox, with full trajectory logging that supports replay and human review. Fifth, continuous regression should cover categories such as work, coding, and chat, weighted by business importance, and report confidence intervals. Sixth, every ranking conclusion should be annotated with snapshot time, sample size, and scorer version to ensure it is interpretable, reproducible, and auditable.

    Agent evaluation is not a one-time leaderboard—it is a continuously evolving engineering capability. Only by combining real long-horizon tasks, blind-testing mechanisms, fine-grained metrics, and a reproducible pipeline can developers accurately understand a model’s capability boundaries and turn evaluation results into reliable product and engineering decisions.