Long-Horizon Agent Evaluation: Metrics, Blind Testing, and Engineering Practices
Long-horizon agents must perceive environments, call tools, maintain state, and revise plans across many steps, so single-turn accuracy is not enough to measure their capability. This article explains why long-horizon agent evaluation is harder, discusses result, process, and stability metrics, and shows how to interpret blind tests and rankings with sample size, task distribution, confidence intervals, and version timing in mind. It also outlines a reproducible evaluation pipeline covering environment isolation, trajectory logging, automated scoring, and result aggregation, and offers governance practices for using public leaderboards as external references while building internal regression suites for model selection.
Long-Horizon Agent Evaluation: Metrics, Blind Testing, and Engineering Practices
1. Why Long-Horizon Agent Evaluation Is Harder
The core difference between an agent and a conventional conversational model is that an agent must continuously perceive the environment, call tools, maintain state, and revise plans across multiple rounds of interaction. A single task may contain dozens of steps, involving file reads and writes, command execution, API calls, error recovery, and intermediate result validation. Single-turn question-answering accuracy cannot reflect this long-horizon capability: a model may make a correct judgment in the first step but later cause the overall task to fail because of state loss, command failures, or delayed correction.
Long-horizon agent evaluation therefore faces at least three challenges. First, task chains are long and failure points are dispersed, so a single overall score can easily hide defects in key stages. Second, evaluation is strongly environment-dependent: tool versions, permission configurations, network conditions, and initial data all affect results, so evaluation must be reproducible. Third, real user needs vary widely; workflows, coding tasks, and everyday chat place different demands on agents, requiring category-level statistics rather than reliance on an overall leaderboard alone. Public blind tests can cover more than 17,500 real long-horizon agent task sessions. Such large-scale samples can improve statistical stability, but they still need to be interpreted together with granular metrics.
2. Metric System: From Overall Scores to Diagnostic Metrics
A usable agent evaluation system should include result metrics, process metrics, and stability metrics at the same time. Result metrics answer whether a task was completed; process metrics answer why it was completed or failed; stability metrics answer whether the agent is reliable across repeated runs. The composite net improvement rate, command execution reliability, and steerability (direction adjustability) mentioned in public materials characterize agent capability from different angles.
The composite net improvement rate can be understood as the change in task pass rate or composite score relative to a baseline version, usually expressed as a percentage. For example, the composite net improvement rate of an open-weight model increased from about +0.9% in the previous generation to about +5.1%, an increase of about 4.2 percentage points. This shows that the gain from iteration must be interpreted in the context of the baseline, task distribution, and scoring criteria. The net improvement rate should not be used in isolation; otherwise, a high share of simple tasks may be misread as a large overall capability gain.
Command execution reliability focuses on an agent’s ability to execute commands in a real environment, read outputs, handle non-zero exit codes, and recover through retries. It is not only about whether a command is written correctly; it also includes whether the agent understands error messages, avoids destructive operations, and chooses reasonable alternative paths after failures. Steerability focuses on whether an agent can adjust its plan in time when a user changes the goal, adds constraints, or corrects an error mid-task, rather than mechanically continuing along the old path. For coding tasks, these two types of metrics are especially critical. For workflow tasks, additional dimensions such as state consistency, permission boundaries, and result auditability are also needed.
3. Blind Testing and Ranking Statistics: Credibility Comes from Methodology, Not Rank
The value of real-time blind testing lies in reducing the influence of brand priors and manual tuning on results. The evaluator assigns tasks while hiding model identities, uses a unified scorer or human review to produce results, and then aggregates them by category and overall. When interpreting rankings, focus on sample size, task distribution, confidence intervals, and version timing. A model leading in the open-weight category does not necessarily lead across all models; ranking in the upper middle of the overall leaderboard does not mean it has no advantage in coding or workflow categories. Public materials show that open-weight models are already competitive in several categories, but the overall leaderboard is still affected by competition among multiple model types. Such conclusions should be read together with granular metrics.
Rankings should also distinguish between “ranking at release time” and “ongoing leaderboard ranking.” The former reflects the performance of a specific version in a specific time window; the latter is affected by later versions, task updates, and changes in scoring criteria. For enterprise model selection, a more reliable approach is to build a small internal regression set and use public blind test results as external references rather than as the sole basis. If a public leaderboard shows that a certain model type performs strongly among open models in work, coding, and chat, you should further examine its failure sample types, long-task completion rate, and variance across repeated runs.
4. Engineering Implementation: Building a Reproducible Evaluation Pipeline
A reproducible evaluation pipeline usually consists of six stages: task set management, environment isolation, agent execution, trajectory logging, automated scoring, and result aggregation. Dependencies should be isolated through a container platform, task data and artifacts should be placed in object storage, execution logs should be written uniformly to a logging service, and key metrics should be connected to monitoring and alerting. If deployed on Baidu AI Cloud, general-purpose capabilities such as container platform, object storage, logging, and monitoring/alerting can be used to build the infrastructure, but the evaluation framework itself should remain cloud-neutral and avoid binding to a specific runtime environment.
Below is simplified pseudocode for metric calculation, focusing on measurement definitions rather than specific APIs:
function evaluate_tasks(task_set, agent, baseline):results = []for each task in task_set:trajectory = agent.execute(task.input, task.environment)completed = scorer.judge_completion(task.expected, trajectory.final_state)command_reliability = successful_commands / max(1, total_commands)correction_success = judge_whether_goal_recovered_after_user_correction(trajectory)results.append({"category": task.category,"completed": completed,"command_reliability": command_reliability,"correction_success": correction_success,"steps": trajectory.steps})for each category:pass_rate = average(results.completed)baseline_pass_rate = baseline.category_pass_rate(category)net_improvement_rate = pass_rate - baseline_pass_rateoutput category, pass_rate, net_improvement_rate,average(results.command_reliability),average(results.correction_success)output overall_net_improvement_rate = overall_pass_rate - baseline_overall_pass_rate
This pseudocode emphasizes three points: aggregation by category, retention of process metrics, and comparison with a baseline. In actual deployment, repeated runs, random seed logging, timeout handling, and failure-cause classification should also be added so that environment failures are not misclassified as model failures.
5. Result Interpretation and Governance Recommendations
When interpreting evaluation results, follow the order of “first look at task distribution, then granular metrics, and finally the overall leaderboard.” If a model shows clear improvement in command execution reliability but a decline in steerability, it is better at following a fixed script but may become rigid when requirements change. If the composite net improvement rate rises by about 4 percentage points, check which categories drive the gain, whether it comes at the cost of more steps or more calls, and whether failure samples cluster around permissions, timeouts, or error recovery scenarios.
For governance, a three-layer mechanism is recommended. First, use public blind tests as an external reference and record versions and timing regularly. Second, build an enterprise regression set that covers core business paths and run it whenever the model or prompt strategy changes. Third, retain manual spot checks and audit logs for high-risk tasks. Through the combination of metrics, blind testing, and engineering pipelines, developers can turn “rankings” into an explainable, reproducible, and actionable basis for agent selection.
