Evaluating AI Agents: Core Design Principles and Key Metrics

Why Agent Evaluation Cannot Rely on Conversation Voting Alone

Traditional model evaluation often depends on users subjectively voting between two responses. While this approach has some value in conversational scenarios, it becomes increasingly inadequate as large models are deployed in real task execution environments. Agents need to complete multi-step operations, invoke external tools, execute commands, and adjust strategies based on feedback. The effectiveness of these behaviors is not directly reflected in a single text response.

An agent evaluation system therefore needs to answer several core questions:

1. Was the task actually completed, rather than merely appearing reasonable?
2. Did the model encounter unrecoverable errors during execution?
3. Did the user need to correct the model’s behavior through follow-up instructions?
4. Did the model remain stable over long-horizon tasks?

These questions cannot be addressed through traditional human voting. Instead, a more realistic evaluation framework is required—one that extracts observable outcome signals from real user sessions and uses sound statistical inference to estimate capability differences between models, rather than simply comparing win rates or average scores.

Core Design Principles of an Agent Evaluation Framework

An agent evaluation leaderboard typically requires three design layers: data sources, signal extraction, and statistical inference.

Data Sources: Real Task Sessions, Not Synthetic Prompts

High-quality evaluation starts with high-quality data. Unlike fixed test sets, evaluation based on real user sessions covers a more diverse task distribution, including:

• Long-horizon multi-turn interactive tasks
• Engineering tasks involving external tool calls
• Operations tasks that require reading environment state and making judgments
• Open-ended tasks where user goals and constraints change dynamically

The advantage of this data is that tasks come from genuine needs, with a more natural distribution, and they expose problems that rarely appear in fixed benchmarks—such as error accumulation, context loss, and recovery after tool call failures.

In practice, the system continuously collects complete session records between users and agents, including user inputs, model outputs, tool calls, command execution results, environment state changes, and subsequent user feedback. To ensure evaluation validity, only sessions reaching a minimum number of interaction turns or containing clear task completion signals are retained, avoiding interference from short chit-chat data.

Signal Extraction: From What Users Say to Whether Tasks Succeed

Extracting structured signals from raw sessions is a critical component of the evaluation system. Common key signals include:

Explicit confirmation signals: The user clearly states that the task is complete or the result is correct, such as “that works,” “no problem,” or “it ran successfully.” These signals indicate task achievement, though they are not necessarily the only endpoint.

Correction instructions: The user asks the model to modify its output, such as “No, I want it grouped by date” or “Regenerate it in JSON format.” The frequency and timing of correction instructions reflect the accuracy of the model’s initial response and its subsequent correction efficiency.

Praise and complaints: Statements expressing user satisfaction or dissatisfaction. These signals carry emotional sentiment but must be interpreted in context and cannot serve as standalone capability indicators.

Tool call errors: Including Shell command execution failures, API parameter errors, nonexistent paths, and insufficient permissions. These signals directly reflect the agent’s reliability at the execution level and are especially important for programming and operations tasks.

Task interruption and retry: The user actively terminates the current session and restarts the same or a similar task, typically indicating that the previous execution did not meet expectations.

In practice, signal extraction requires a combination of rules and model-based judgment. Rules can identify explicit command error codes and specific user phrasing, while semantic-level confirmation, correction, and sentiment judgment are better handled by trained classifiers. After extraction, each session is converted into multiple structured labels for subsequent statistical computation.

Causal Inference: Estimating Net Improvement Rate, Not Absolute Scores

Different models are often assigned tasks of varying difficulty and type, so directly comparing success rates is susceptible to task distribution bias. To estimate relative capability differences between models more accurately, randomization-based inference can be applied.

The basic idea is as follows: when multiple models serve in parallel, user tasks are randomly assigned to different models. Because assignment is random, the task difficulty received by each model tends to be statistically equivalent, significantly reducing the impact of task difficulty on capability estimation. The system then compares each model’s task completion against the overall average across all models to compute a net improvement rate.

The net improvement rate represents how much a model’s task completion improves relative to the average of all participating models under the same task distribution. This metric can be positive, negative, or near zero. A positive value indicates above-average performance, a negative value indicates below-average performance, and zero indicates performance at the average level.

This method does not rely on users comparing outputs between two models, nor does it require users to have evaluation expertise. Instead, it infers capability differences from real task outcomes, providing a more objective reflection of agent performance in real-world scenarios.

Key Metrics and Engineering Implementation

In agent evaluation, beyond the overall net improvement rate, several capability sub-metrics are needed to identify specific strengths and weaknesses.

Shell Command Execution Reliability

In programming and operations tasks, models frequently need to generate Shell commands, read and write files, manage processes, or invoke system tools. Shell command execution reliability measures the correctness and executability of model-generated commands, including:

• Syntactic correctness
• Parameter validity
• Whether paths and file references exist
• Whether the current environment state is fully considered

A command that appears correct may still fail if it ignores directory context or environment variables. This metric therefore evaluates not only the model’s ability to generate commands but also its grasp of the execution context.

Direction Adjustment and Error Correction

This metric measures whether the model can quickly understand the source of deviation and adjust its behavior after receiving user correction. A high-level agent does not necessarily need to execute perfectly every time, but when a user identifies an issue, it should be able to:

• Accurately locate the source of the deviation
• Preserve previously correct work
• Avoid introducing new errors during correction
• Return to the correct path with fewer interaction turns

This metric is particularly important for long-horizon tasks, since real tasks are rarely completed in a single attempt. Models must possess the ability to continuously correct and self-calibrate.

Task Scenario Segmentation

General agent evaluation typically also breaks down performance by task type, such as work scenarios, programming scenarios, and everyday conversation scenarios. This is because the same model may perform noticeably differently across scenarios. For example, some models excel at creative writing but have higher error rates in code generation tasks with strict syntax constraints, while others show the opposite pattern.

Scenario segmentation allows developers to select models more precisely based on their application type, rather than relying solely on overall rankings.

How Evaluation Results Guide Engineering Decisions

For developers, the value of an evaluation leaderboard is not about which model ranks first, but whether it supports sound engineering decisions. In practical applications, several dimensions matter:

Task Fit Takes Priority Over Absolute Scores

Different business scenarios impose different capability requirements. If the primary use case is automated script generation, Shell command execution reliability and code task sub-metrics deserve priority. If the use case is customer support conversations, correction instruction response and user satisfaction signals matter more.

When selecting a model, teams should first clarify their task types and acceptable failure costs, then evaluate sub-metrics accordingly, rather than defaulting to the highest-ranked model overall.

Pay Attention to Long-Horizon Stability

Evaluation data based on real sessions often contains a large number of long-horizon tasks, which place higher demands on a model’s context management, state tracking, and error recovery. Strong short-conversation performance does not guarantee good long-task performance. Developers should observe model behavior in evaluation subsets with task lengths similar to their own.

Assess Infrastructure Costs

In real deployments, model capability is only one dimension of the decision. Inference cost, response latency, and deployment complexity are equally important. Evaluation metrics help developers shortlist models that meet capability requirements; the final choice still requires balancing against infrastructure constraints.

General Steps for Building an Internal Agent Evaluation System

Teams that want to build their own agent evaluation capability can follow this process:

1. Define the task scope: Clarify which task types the agent needs to handle and define success criteria for each type.
2. Collect real sessions: Record complete sessions during service operation and apply data anonymization to protect user privacy.
3. Label outcome signals: Use a combination of rules and classification models to extract key events such as task completion, corrections, errors, and interruptions from sessions.
4. Establish a statistical baseline: Randomly assign tasks across multiple candidate models under equivalent task distributions and compute each model’s net improvement rate or similar metrics.
5. Continuously update and monitor: An evaluation system is not a one-time project. As task types and user behavior evolve, signal definitions and baseline data must be updated regularly to keep evaluation results valid.

In implementation, logging systems, session storage, asynchronous statistical jobs, and visualization dashboards are essential foundational components. For large-scale deployments, message queues can be used to stream raw session data into storage in real time, with offline jobs handling batch metric computation.

Conclusion

Agent evaluation is shifting from “does the response sound human” to “was the task actually completed.” A well-constructed evaluation framework requires real task data, fine-grained outcome signals, and rigorous statistical inference to provide reliable guidance for model selection and iteration. For developers, understanding the design logic behind such evaluation systems—and building an internal evaluation process suited to their own business—is a critical step toward improving agent application quality.