Evaluation Methods and Metrics for Long-Horizon Agent Tasks

    This article explores advanced evaluation methods and metrics for long-horizon AI agent tasks. It highlights the shift from outcome-based to process-based scoring, interprets public benchmark rankings and net improvement rates, details key metrics like Shell command execution reliability and controllability, and outlines an engineering evaluation pipeline. Finally, it offers recommendations for enterprise model selection and implementation.

    I. Why Long-Horizon Agent Task Evaluation Needs New Methods

    Traditional model evaluation primarily focuses on single-turn Q&A, short-text generation, or fixed test sets, making it difficult to cover the complete workflow of an agent in a real-world environment. Long-horizon agent tasks often involve multi-turn planning, tool calling, environmental state maintenance, error recovery, result verification, and human confirmation. The task trajectory may span dozens of steps or more. A single failure might stem from unreasonable plan decomposition, incorrect parameter inputs, unhandled command exit codes, context state loss, or security policy conflicts. Therefore, looking only at whether the final answer is correct cannot explain “why it failed” and “at which step it failed.”

    The core shift in long-horizon evaluation is moving from “outcome scoring” to “process scoring.” A relatively complete evaluation system should at least monitor: task completion rate, step efficiency, tool calling success rate, error recovery rate, constraint compliance rate, number of human takeovers, cost and latency, and trajectory auditability. For enterprise implementation, process metrics are more valuable for decision-making than a single leaderboard ranking, as they can be mapped to specific engineering improvements, such as enhancing sandbox isolation, adding command pre-validation, introducing plan review nodes, or optimizing context compression strategies.

    II. Public Blind Benchmarks and Open-Weight Group Rankings

    Public blind benchmarks typically use anonymous or blind testing methods, allowing participating models to run in real or near-real task sessions, followed by a combination of automated scoring and manual spot-checking to form a ranking. One public evaluation showed that a certain open-weight model ranked 1st in the open-weight model group, 13th among all participating models, and 1st in the open model subgroups for work, coding, and chat. This evaluation was based on over 17,500 real long-horizon agent task sessions. In another real-time blind test update, a participating model ranked 5th.

    This information needs to be interpreted in layers. First, “1st in the open-weight group” indicates that the model performs prominently among models with accessible weights that can be privately deployed, making it suitable for enterprises with requirements for data boundaries, deployment locations, and cost control. Second, “13th on the full leaderboard” indicates that if closed-source and open-weight models are compared in the same pool, its overall position still has room for improvement. Third, “1st in all three open model categories: work, coding, and chat” suggests a relatively balanced capability distribution, but since the weights, scoring rubrics, and sample distributions differ across the three task types, it cannot simply be equated to being optimal for all business scenarios. Fourth, although over 17,500 real long-horizon sessions provide a substantial sample base, attention must still be paid to task sources, industry distribution, tool environments, and scorer consistency, to avoid directly extrapolating leaderboard rankings as production conclusions.

    III. Comprehensive Net Improvement Rate: Magnitude and Statistical Confidence

    A certain open-weight model had a comprehensive net improvement rate of +5.1%, while its previous generation version was +0.9%, a difference of 4.2 percentage points. The “net improvement rate” here is typically not the absolute accuracy rate, but rather the weighted result of comprehensive net wins or net improvements relative to a baseline model across multiple tasks and scoring dimensions. It is better suited to answer “whether the new generation is stably better than the previous one” rather than “what the absolute task success rate is.”

    Three points should be noted when interpreting this. First, percentage points are different from percentages: going from +0.9% to +5.1% is an increase of 4.2 percentage points, not an increase of 4.2%. Second, the net improvement rate should be accompanied by confidence intervals and sample sizes. If the confidence interval crosses zero, the improvement may not be significant; if the task set distribution leans towards a certain type of scenario, the comprehensive value might be driven by high-weight categories. Third, stratified reporting should be performed: providing net improvements separately for subsets like work, coding, chat, long documents, and multi-tool orchestration, to prevent the comprehensive score from masking shortcomings. Enterprises can set regression gates: the net improvement rate of a new version on core task sets must not be lower than the baseline, critical safety metrics must not degrade, and cost and latency must not exceed thresholds.

    IV. Granular Metrics: Shell Command Execution Reliability and Controllability

    Shell command execution reliability focuses on the correctness and robustness of commands generated and executed by the agent, including command syntax, parameter values, path handling, exit code checking, timeout control, idempotency, permission boundaries, dangerous operation interception, and failure rollback. In long-horizon tasks, command errors amplify along the trajectory, so evaluation should cover normal paths, exception paths, and boundary conditions. For example: does it stop and diagnose when a command returns a non-zero exit code; does repeated execution produce side effects; do high-risk operations like deletion, overwriting, or permission changes trigger confirmation; does it clean up residual processes after a sandbox timeout. The following checklist can be established:

    • Whether static validation and dangerous pattern matching are performed after command generation;
    • Whether exit codes and standard error output are explicitly checked;
    • Whether timeout, retry limits, and rollback strategies are in place;
    • Whether complete trajectories are recorded for auditing and reproduction;
    • Whether execution occurs in an isolated environment with restricted network and file system permissions.

    Controllability focuses on the ability to adjust direction and correct deviations—meaning, when a user or system provides new constraints, corrects a wrong direction, or requests a plan change, whether the agent can understand the intent, update the plan, and maintain consistency in subsequent steps. Evaluation can construct “mid-task change” tasks: provide an initial goal, execute several steps, and then insert a budget reduction, format change, prohibition of certain operations, or priority adjustment to observe whether it abandons the old plan, clarifies ambiguities, re-plans, and verifies the results. High-score performance should not just be verbal acknowledgment, but should be reflected in subsequent tool calls, parameter selection, and final deliverables.

    V. Engineering Evaluation Pipeline Design

    To implement the above metrics, it is recommended to build a pipeline consisting of “task set—isolated environment—trajectory collection—automated scoring—manual spot-checking—statistical reporting—regression gate.” The task set should cover work, coding, chat, and mixed scenarios, stratified by difficulty, tool dependency, and risk level. On the environment side, use general-purpose capabilities such as container platforms, logging services, and monitoring alerts to ensure that every run is reproducible and traceable. On the scoring side, combine rule-based scoring, result verification, and model-assisted scoring, retaining manual spot-checking for critical tasks. The statistics side outputs mean values, confidence intervals, stratified results, and failure attribution. The pseudo-code is illustrated as follows:

    1. def evaluate_agent(candidate, baseline, tasks, weights):
    2. records = []
    3. for task in tasks:
    4. traj = run_in_sandbox(candidate, task, max_turns=task.max_turns)
    5. score = grade(traj, task.rubric) # Results + process + safety constraints
    6. records.append({
    7. "task_id": task.id,
    8. "score": score,
    9. "turns": traj.turns,
    10. "tool_failures": traj.tool_failures,
    11. "human_takeover": traj.human_takeover
    12. })
    13. net_gain = weighted_mean_gain(records, baseline.records, weights)
    14. ci = bootstrap_confidence_interval(records, baseline.records, weights)
    15. return net_gain, ci, stratified_report(records)

    The pipeline should pay attention to three types of biases: task set bias, scorer bias, and environment bias. Task set bias can be mitigated through stratified sampling and regular updates; scorer bias can be mitigated through double-person spot-checking, consistency analysis, and scoring rubric calibration; environment bias can be mitigated through fixed images, fixed dependency versions, and random seeds. For production systems, evaluation results should also be integrated into the release process, forming a closed loop of “version change—offline regression—grayscale evaluation—online monitoring.”

    VI. Enterprise Selection and Implementation Recommendations

    Enterprise selection should not solely rely on overall leaderboard rankings, but should establish a scenario weight matrix. If the business emphasizes data boundaries and private deployment, focus can be placed on open-weight group performance, deployment costs, and operational complexity; if the business emphasizes complex tool orchestration, the weight of Shell command execution reliability and controllability should be increased; if the business emphasizes multi-turn communication, the sample size for chat tasks and clarification capabilities should be increased. General-purpose capabilities provided by Baidu AI Cloud, such as container platforms, logging services, and monitoring alerts, can be used to build isolated evaluation environments and trajectory audit systems, but the specific solution still needs to be designed in conjunction with the enterprise’s own compliance and engineering constraints.

    Finally, a leaderboard is an observation window, not the conclusion itself. A truly usable agent evaluation system should feature reproducible task sets, process metrics, statistical significance, failure attribution, and regression gates. Only by transforming “rankings” into “explainable capability evidence” can we support model selection, version upgrades, and production release decisions.