Agent Evaluation Beyond Vote Ranking: Signals, Randomization, and Causal Inference

Why Agent Evaluation Needs a New Approach

Most traditional LLM evaluation relies on pairwise answer voting or static benchmark questions. These methods have value for measuring single-turn QA quality, but they expose clear limitations in agent scenarios. An agent task is rarely completed in a single request; it typically involves multi-step reasoning, tool calls, environment interaction, and error recovery. Users may issue corrective instructions mid-task, and they may offer praise or complaints after final confirmation. Preference votes between two answers can hardly capture these critical signals that occur across long sessions.

The more fundamental issue is that traditional leaderboards focus on “which answer looks better,” while agent engineering cares about “which model fails less often on real tasks, is easier to correct, and completes goals more reliably.” An agent-oriented evaluation system therefore needs to answer three questions: first, how to obtain sufficiently realistic agent task sessions; second, how to automatically extract valuable outcome signals from those sessions; third, how to use rigorous statistical methods to estimate the relative performance of different models.

Extracting Outcome Signals from Real Sessions

One viable design approach is to require no additional user annotation and instead mine feedback from real agent usage records that have already occurred. When users complete agent tasks, their behavior leaves behind a large number of parseable clues. These clues can be grouped into several categories of key signals.

The first category is task completion confirmation. After an agent finishes a task, users often give explicit confirmation such as “that works,” “this is fine,” or “problem solved.” These signals usually appear near the end of a session and are the most direct positive outcome indicator.

The second category is praise and complaints. When users are satisfied with agent performance, expressions such as “very useful” or “this helped a lot” may appear; conversely, when the agent misinterprets requirements, makes operational mistakes, or produces unexpected results, users may give direct negative feedback. These signals are subjective but reflect user recognition of the overall experience.

The third category is steering instructions. If an agent deviates from the goal during execution, users often insert new instructions to correct direction. For example, if the original request was to clean data but the agent starts generating charts, the user might say “don’t draw charts yet, clean the data first.” Detecting these steering signals can measure a model’s ability to maintain goal consistency over long tasks.

The fourth category is command execution errors. When an agent interacts with the environment through command lines or scripts, execution failures, non-zero exit codes, permission errors, and missing commands can all occur. These errors can be automatically recorded and structured, forming objective reliability indicators. Compared with manual annotation, such signals are easier to standardize and better reflect model stability in real execution environments.

Estimating Net Improvement Rate with Randomized Inference

Signals alone are not enough; the key is how to turn signals into comparable evaluation results. Simply counting each model’s success rate would be affected by differences in user populations, uneven task difficulty distributions, and varying session lengths. A more robust approach introduces randomized inference.

The core idea is this: when real user requests arrive, assign different models randomly to different sessions. Because assignment is random, the user populations and task types each model faces converge in expectation. Differences observed in outcomes can therefore be attributed more plausibly to differences in model capability rather than sample bias.

On this basis, a “net improvement rate relative to the average level” can be computed. Specifically, define an average baseline, which may be the aggregated performance of all participating models on the same signal dimension. Then, for a given model, calculate its positive or negative deviation from that baseline across various signals in real sessions. For example, if a model’s task confirmation rate is above the average baseline, its execution error rate is below the average baseline, and its recovery success rate after steering is also higher, then its net improvement rate will be positive.

The advantage of this method is that it avoids the problem of absolute scores being easily affected by data distribution. Absolute values may be inflated overall because simple tasks are more common in a given period, but improvement rates relative to the average level more stably reflect whether a model is genuinely better on comparable tasks.

Why Granular Metrics Matter More Than a Total Score

When an agent’s overall ranking looks good, engineering teams still need to break down its capability structure. Different business scenarios place very different demands on agents: some prioritize script execution reliability, while others prioritize whether the agent can quickly get back on track after being corrected.

A reasonably complete agent evaluation system should cover at least the following dimensions:

Task completion rate: the proportion of sessions in which the agent advances the task to explicit user confirmation within a limited number of steps. This metric directly corresponds to “whether the job ultimately got done.”

Command execution reliability: the probability that commands or scripts generated by the agent execute successfully in the real environment. Execution failure does not necessarily mean complete task failure, but it significantly increases user waiting time and intervention cost.

Direction adjustment and steering capability: whether the model can quickly understand new requirements and adjust subsequent behavior after a user gives a steering instruction. This capability is especially critical in long-horizon tasks, because real tasks rarely align perfectly on the first attempt.

Session efficiency: the number of interaction turns and tool calls required to complete a task without sacrificing completeness of the result. Efficiency that is too low wears down user patience, while efficiency that is too high at the expense of result fidelity is meaningless.

Error recovery rate: whether the model can perform a local fix after an execution error or misunderstanding without requiring the user to restate the entire requirement.

Only by observing these dimensions together can teams avoid being misled by a single total score. For example, a model may rank high overall but perform only moderately on command execution reliability, making it unsuitable for production environments with a high degree of automation.

Boundaries of Interpretation

Any leaderboard should clearly state its capability boundaries, and agent evaluation is no exception. Real agent sessions are closer to actual usage than pairwise voting, but several issues deserve attention.

First, task type distribution is affected by the user population. If a certain type of task accounts for a significantly larger share in a given period while samples for another type are insufficient, the generalizability of the corresponding conclusions will decline. Evaluators need to continuously accumulate samples and document task distribution in their reports.

Second, user expressions are ambiguous. Not all task confirmations appear at the end of a session, and not all praise means the task was truly completed. Signal extraction needs to incorporate context modeling, for example by jointly considering session termination position, subsequent follow-up questions, and changes in environment state.

Third, statistical improvement does not equal business improvement. A model with a net improvement rate above baseline does not mean it can replace humans in every vertical scenario. Business teams should still run small-scale online validation against their own task characteristics.

Practical Implementation Guidance

For teams seeking to build internal agent evaluation capabilities, a phased implementation is advisable.

Phase 1: Unify session logging. Regardless of which entry point the agent is deployed at, it should fully record user inputs, intermediate model steps, tool call parameters, command execution return codes, and final result states. Without structured logs, all subsequent evaluation is impossible.

Phase 2: Build a signal extraction pipeline. Apply rule-based pre-filtering to existing logs to identify explicit confirmation, steering, and error events; then introduce classification models to handle more ambiguous expressions. Signal outputs should be unified into structured fields, for example:

{
  "session_id": "s_9f21a",
  "signals": [
    {"type": "confirm", "turn": 14, "confidence": 0.91},
    {"type": "command_error", "turn": 6, "exit_code": 127},
    {"type": "steer", "turn": 8, "confidence": 0.84}
  ]
}

This structure facilitates subsequent aggregation and cross-comparison.

Phase 3: Introduce randomized controls. Randomly assign models at the traffic entry point to ensure that the samples each model receives are statistically comparable. Record the incidence of each signal type and compute differences against the overall mean.

Phase 4: Periodic reporting and regression alerts. Publish weekly or monthly trends for each model on key metrics. When the relative improvement rate of a metric drops noticeably, trigger a regression check rather than waiting for a surge of user complaints.

Future Directions

Agent evaluation is evolving from a “static leaderboard” toward a “continuous observation system.” As agents take on increasingly complex tasks, evaluation signals will expand from the current focus on text feedback and command states to additional dimensions such as environment state validation, resource consumption measurement, and security and compliance checks. At the same time, randomized inference can be combined with stratified sampling, covariate adjustment, and other methods to further reduce sample bias.

For developers, the most valuable takeaway is not a single ranking of any model, but an understanding of the measurement logic behind those rankings. Only by seeing that logic clearly can teams design evaluations tailored to their own business environments, interpret the data, and ultimately make more reliable agent technology choices.