Agent Evaluation: Moving from Preference Voting to Causal Inference
Preference voting and leaderboard rankings work well for single-turn LLM chat, but agents need something different. This article explains why pairwise human preferences break down for multi-turn, stateful, open-ended agent tasks, how to mine quantifiable signals from real sessions (task completion, sentiment, correction rate, command error rate), how to estimate a net improvement rate with randomized experiments, and how to wire it all into an engineering pipeline. It also covers the core metrics — command execution reliability, controllability, and composite net improvement — and the conditions under which the approach holds.
Why agent evaluation has to move beyond preference voting
Bottom line first: agent evaluation is shifting from “human pairwise voting” toward “mining real session signals plus randomized causal inference.” This article covers four things: why agent evaluation can’t rely on preference alone, how to extract quantifiable signals from real sessions, how to estimate the net improvement rate with randomized experiments, and how to plug evaluation into an engineering pipeline.
When evaluating ordinary large language models, the common approach is to have annotators or users compare two answers pairwise, then convert the results into rankings using methods such as Bradley-Terry or Elo. This works for single-turn Q&A because what’s being compared is usually one response, and the dimensions are relatively narrow. Agents are different. An agent can plan autonomously, call tools, execute across multiple turns, and act on a real environment. Its behavior has three characteristics:
- Multi-turn and stateful: a task usually spans several interactions, and the intermediate state influences the final outcome.
- Open-ended: the goal isn’t necessarily “pick A or pick B,” but “was the task actually accomplished.”
- Long-horizon: one real task can run for a while, so short spot checks struggle to reflect stable performance.
So the focus of agent evaluation has to move from “which answer is more pleasing” to “which agent reliably completes the real task.”
Three bottlenecks of traditional preference-based evaluation
Once transplanted into the agent setting, traditional preference evaluation exposes three problems.
First, sampling bias. Preference voting naturally suits short, easily comparable Q&A, while long-chain tasks with tool calls are often underrepresented. Complex tasks are exactly the ones that reveal agent capability, and they’re also the hardest to get into a preference sample.
Second, signal monotony. “Which answer a user prefers” and “whether the task actually got done” are not the same thing. A response that reads beautifully but never got the task working can win a human vote; conversely, a terse response that successfully completed the task may not be likable.
Third, attribution difficulty. Even if you observe that one model has a higher session success rate, it’s hard to tell whether that comes from model capability or from differences in user population, task difficulty, time distribution, and similar factors. Correlation is not causation.
Breaking through these bottlenecks requires a different set of data and methods: real sessions as the primary source, observable behavioral signals as the measure, and randomized experiments as the attribution mechanism.
How to evaluate agents: extracting multiple signals from real sessions
Every interaction an agent has in a real environment leaves a trace. Compared with post-hoc subjective scoring, these behavioral footprints are closer to the objective outcome of the task. A typical signal extraction framework can focus on the following categories:
- Task completion confirmation: when the user explicitly says things like “that’s done,” “that works,” or “that’s exactly what I wanted,” it’s a strong positive signal.
- Praise and complaints: these reflect the sentiment polarity of the user toward the result.
- Correction turns: when the user says “no, it should be…,” earlier steps may have gone off track. If the corrections are mainly caused by agent deviation, more corrections usually means worse controllability.
- Command execution errors: whether the agent errors out or fails and retries when invoking command lines, scripts, or tools directly reflects reliability at the execution layer.
These signals have different scales, so normalize them first, then weight them into a comparable “session effectiveness” score. Sample code:
def weighted_normalize(raw, weights):# raw: raw signals# task_confirmed: whether an explicit completion confirmation appeared, 0 or 1# sentiment: praise counts as +1, complaints count as -1# correction_ratio: number of correction turns / total turns# exec_error_rate: number of failed commands / total commandstask_confirmed = float(raw["task_confirmed"])# map the [-1, 1] sentiment polarity to [0, 1]sentiment_norm = (raw["sentiment"] + 1) / 2# lower correction and error rates are better, so use 1 - ratiocorrection_good = 1.0 - raw["correction_ratio"]exec_good = 1.0 - raw["exec_error_rate"]features = [task_confirmed, sentiment_norm, correction_good, exec_good]return sum(w * x for w, x in zip(weights, features))
The weights shouldn’t be picked by guesswork. A safer approach: first build a human-labeled ground-truth set where annotators judge whether each session succeeded; then fit logistic regression or another interpretable model on the human success labels to get initial weights; after launch, keep recalibrating with ongoing human spot checks.
A sample session
Illustrative data, to show the calculation only.
Suppose a session has 8 turns: the user first asks the agent to look up an order, the agent runs a command that fails; the user corrects it with “no, check today instead of yesterday”; the agent retries and succeeds; the user finally says “that works.” In this session:
- explicit confirmations: 1;
- corrections: 1;
- total commands: 4;
- failed commands: 1;
- praise: 0;
- complaints: 0.
So:
- task_confirmed = 1;
- sentiment = 0, which maps to sentiment_norm = 0.5;
- correction_ratio = 1 / 8 = 0.125;
- exec_error_rate = 1 / 4 = 0.25.
With weights of [0.45, 0.15, 0.20, 0.20]:
score = 0.45 * 1.0+ 0.15 * 0.5+ 0.20 * (1 - 0.125)+ 0.20 * (1 - 0.25)= 0.45 + 0.075 + 0.175 + 0.15= 0.85
The score indicates relatively high overall effectiveness for this session, but there was still one command failure and one correction, so it isn’t a perfect score.
Proxy signals must be validated
Keep in mind that confirmation, sentiment, correction rate, and execution error rate are only proxy signals for task success, not ground truth. It’s worth validating rules and classifiers against a human-labeled ground-truth set for precision and recall, and computing the agreement between the composite effectiveness score and human judgments — for example, Cohen’s κ. If agreement between the proxy signals and human judgments is too low, they can’t be used directly for version comparison.
From correlation to causation: randomization and the net improvement rate
Even with large-scale behavioral signals, one key question remains: is the observed difference caused by model capability, or by something else?
The answer is to introduce randomization. In real traffic, randomly assign the policies or model versions being compared to similar tasks and user groups. Because the assignment is random, the two groups are expected to be balanced on confounders such as task difficulty, user habits, and time distribution. The difference in session effectiveness between the two groups can then be attributed much more cleanly to the thing being evaluated.
On that basis, the net improvement rate is defined as the net effect increment of the target model relative to a control group. The control group is the set of sessions that didn’t hit the target policy. It isn’t an isolated absolute score, but a causal estimate relative to a baseline, used to answer “how much better is this version than the control, really.”
Sample code:
import numpy as npfrom scipy.stats import tdef estimate_net_improvement(sessions, alpha=0.05):"""sessions: each session contains at least- treatment: whether it hit the target policy or version- success: normalized session effectiveness, 0 to 1"""treated = np.array([s["success"] for s in sessions if s["treatment"]], dtype=float)control = np.array([s["success"] for s in sessions if not s["treatment"]], dtype=float)n_t = len(treated)n_c = len(control)net_gain = treated.mean() - control.mean()# for small samples, use sample variance and the t quantile; for large samples, 1.96 is a fine approximationvar_t = treated.std(ddof=1) ** 2 if n_t > 1 else 0.0var_c = control.std(ddof=1) ** 2 if n_c > 1 else 0.0se = (var_t / n_t + var_c / n_c) ** 0.5df = n_t + n_c - 2crit = t.ppf(1 - alpha / 2, df) if df > 0 else np.infreturn net_gain, (net_gain - crit * se, net_gain + crit * se)
A few engineering points need to be added here.
First, the randomization unit and the analysis unit must match. If you randomly assign a policy to users or tasks, one user or task may contribute several sessions, and computing standard errors as if sessions were independent will make them too small. A more appropriate approach is cluster-robust standard errors clustered by user or task, or a block bootstrap. You also need to state the SUTVA interference assumption: one user’s session outcome shouldn’t significantly affect another user’s session outcome. If there’s strong cross-user interference, simple randomized attribution breaks down.
Second, 1.96 is only a large-sample normal approximation. Small samples should use the t quantile, or bootstrap directly. Any randomized inference must report a confidence interval and a sample size. A point estimate of “+X%” detached from any measure of uncertainty has limited engineering value — it may just be random fluctuation.
Third, when real traffic is limited, you can hash users for bucketing so that the same user always lands in the same group. Before a formal A/B test goes live, run an A/A test first to verify that the bucketing mechanism and the metric definitions behave as expected.
Which metrics should agent evaluation look at?
Within the framework above, a few core metrics deserve separate explanation.
Command execution reliability
Command execution reliability measures the success rate and stability of an agent invoking command lines, scripts, or tools. One workable definition is:
Command execution reliability = successful commands / total commands
You can also track failure retry rate, tool call timeout rate, and environment adaptation failure rate at the same time. Command failures directly drag down task completion rate and reliability scores, which makes this a foundational metric for assessing agent capability.
Controllability and correctability
Controllability measures whether an agent can return to the right direction after the user issues a correction. Two definitions work here:
Correction rate = number of correction turns / total turnsTurns to recover after correction = turns between the first explicit correction and the user's confirmation
A lower correction rate and fewer recovery turns usually indicate that the agent better understands user intent and self-corrects more effectively. The premise, though, is that those corrections are mainly caused by agent deviation; if they come from the user changing requirements or making a mistake, they should be counted separately.
Composite net improvement rate
The composite net improvement rate converts multiple signals into session effectiveness, then uses randomized experiments to estimate the net effect relative to the control group. It’s suited to cross-version and cross-policy comparison, rather than looking at an absolute score in isolation.
Applicability conditions
This approach fits the following scenarios:
- traffic can be randomly split;
- the task distribution is relatively stable;
- the correlation between behavioral signals and task success has been validated by humans;
- there is no strong interference between users;
- there is enough sample size to support confidence interval estimation.
It does not fit scenarios where traffic can’t be randomly split, cross-user interference is very strong, tasks are highly customized, or the sample size is severely insufficient.
How to run agent evaluation: a five-step pipeline
To actually put the method to work, you need a complete evaluation pipeline.
1. Session collection and compliant de-identification
Real sessions can involve user data. Before collecting, you need a lawful basis — for example, informed consent or coverage by a privacy policy — and clarity on the data source, purpose of use, access permissions, and retention period. De-identification is not the same as anonymization, and you still need to prevent re-identification of users through combinations of fields. Collection should follow the data minimization principle, keeping only the structured behavioral signals relevant to evaluation.
2. Signal extraction and human validation
You can use rules, classifiers, or small models to label signals in sessions. For high-risk signals such as task confirmation, complaints, and command failures, human spot checks are warranted. Those checks should look beyond the labeling results and also compute precision, recall, and the agreement between the composite effectiveness score and human judgments.
3. Random assignment and experiment design
Before formally comparing versions, design the bucketing first. If traffic is limited, hash users for assignment. Before a formal A/B test, run an A/A test to confirm that the two groups show no significant drift when there is no real policy difference. Before the experiment, also estimate the minimum sample size and statistical power, specifying the minimum detectable effect, α, and power.
4. Statistical inference and release gating
A rollback decision can’t hinge on the word “significant” alone. A release gate should predefine α — for example, 0.05 — and set a minimum sample size. If one update looks at multiple metrics at once, or if tests are run repeatedly and often, the false positive risk goes up. In that case you can apply multiple comparison corrections such as Benjamini-Hochberg, or use sequential testing such as SPRT.
If a confidence interval indicates a regression, it’s better to have a human review it before blocking the release, so that sampling fluctuation doesn’t cause a false alarm.
5. Baseline maintenance and continuous monitoring
A fixed baseline session set helps comparability, but it can also be optimized against deliberately. So the baseline set should be refreshed periodically or supplemented with new samples, while a portion of core tasks is kept unchanged over the long term for trend observation. Evaluation results should land in a dashboard and be linked with the version release, rollback, and alerting systems.
Five best practices for agent evaluation
- Put privacy and compliance first: collecting, storing, and using real sessions must be compliant and traceable; don’t rely on technical de-identification alone.
- Don’t overfit to a single signal: confirmation, sentiment, corrections, and execution errors should be combined with weighting. Optimizing one metric alone makes it easy to game the evaluation.
- Always report sample size and uncertainty: small-sample rankings are highly unstable, so prioritize confidence intervals and effect sizes.
- Keep randomization clean: once random assignment is contaminated by business logic, causal attribution breaks.
- Fix your definitions and refresh the baseline periodically: version comparison needs stable definitions, but the baseline also needs periodic updates so the evaluation set isn’t over-optimized.
Conclusion
Agent evaluation is going through a methodological upgrade: from a “voting era” that depends on human preference, toward an “empirical era” built on real sessions as data, multiple signals as measurement, and randomized causal inference for attribution. For developers, understanding the logic behind randomized experiments, net improvement rate, reliability, and controllability is worth more in the long run than memorizing any single leaderboard rank. What really accumulates isn’t a score, but a trustworthy, reproducible, and continuously evolving agent evaluation capability.
