Agent Evaluation Beyond Voting: Measuring Causal Net Improvement Rate

1. Why Agent Evaluation Needs to Move Beyond Voting

As agent technology moves rapidly into production, evaluation systems often lag behind model capabilities. Traditional evaluation typically relies on two approaches: subjective user voting between two responses, or offline scoring against a fixed test set. Both have value, but each has clear limitations in real agent scenarios.

Voting-based evaluation is good at measuring which response is more popular, but it struggles to explain how an agent actually performs on long-horizon tasks. Agent tasks are usually not single-turn question answering but a continuous process of multiple tool calls, command execution, result verification, and course correction. An agent that sounds fluent may repeatedly produce command errors, misunderstand user intent, or fail to converge on the correct result in a real task.

Offline test sets are controllable and reproducible, but their coverage is limited. Real user needs are highly diverse, and task context, runtime environments, and recovery behavior after failure all differ substantially from any test set. Extracting reliable signals from real usage data and turning them into comparable, interpretable evaluation results is therefore a key challenge in agent engineering.

The evaluation approach discussed in this article does not rely on user voting. Instead, it automatically extracts task-outcome-related signals from real agent sessions and then uses randomized inference to estimate each model’s net improvement rate relative to the average. This is closer to outcome-based process evaluation and better reflects agent usability in complex tasks.

2. Extracting Key Signals from Real Sessions

Real agent sessions contain a wealth of usable signals. The key is identifying which signals correlate with task success or failure. Common signals include:

1. Explicit user confirmation of the task: When a user says things like “it worked,” “that’s exactly what I wanted,” or “that’s fine,” it usually indicates the task met expectations.
2. Praise and complaints: Emotional feedback from users often directly reflects agent performance, such as “well done” or “completely wrong.”
3. Correction instructions: When a user asks the agent to change direction, fix an error, or redo a step, it typically means the agent did not complete the task in one pass.
4. Shell command execution errors: In tasks involving command-line operations, whether a command executed successfully, was rejected by the environment, or produced unexpected output are all important process signals.
5. Task abandonment or restart: A user who stops interacting for a long time or terminates the session directly may indicate the agent did not meet their needs.

The advantage of these signals is that they come from real user behavior rather than simulated experimental environments. By automatically parsing session logs, large numbers of samples can be collected without adding burden on users. These signals also cover multiple stages of task execution, characterizing model behavior more granularly than a simple “was the final answer correct” measure.

Of course, automatic signal extraction has challenges. Natural language confirmations or negations may be ambiguous, and command execution failures may stem from environment issues. An evaluation system therefore needs context-aware parsing, threshold filtering, and exception handling to minimize noise interfering with final conclusions.

3. Estimating Net Improvement Rate with Randomized Inference

After extracting signals from real sessions, a sound statistical framework is needed to compare different models. Naively comparing success rates or average scores is often confounded by task difficulty, user habits, session length, and other factors.

One viable approach is randomized inference, also called causal tracing. The basic idea: randomly assign different models to handle similar tasks in real service, record session outcome signals, and then compute each model’s net improvement rate relative to the average across all models.

Net improvement rate is not simply a success rate lift but a relative effect value derived from combining multiple signals. For example, more task confirmations, fewer correction instructions, and fewer command errors together determine the net improvement rate. Randomization helps remove bias from task difficulty differences and user population differences.

The calculation is roughly as follows:

net improvement rate = model's actual performance - expected average performance under random assignment

A positive net improvement rate means the model outperforms the average on the same task distribution; a negative value means it underperforms. This is more robust than absolute scores, especially for continuous comparison across time and user populations.

Multi-signal fusion also requires weight design. Different signals have different impacts on task success; for example, a shell command execution error is usually more serious than a single ordinary correction. Weights can be learned from historical data or set manually based on business context. What matters is keeping the evaluation methodology transparent so it can be reproduced and calibrated.

4. Interpreting Key Metrics: Command Reliability and Course Correction

Two metrics are especially important for long-horizon tasks in agent evaluation.

Shell command execution reliability measures whether commands generated by the agent can be correctly executed by the environment and produce expected results. Low reliability means the agent frequently generates wrong arguments, incompatible commands, or ignores execution results. For scenarios such as automated operations, code generation, and data processing, this metric directly affects whether tasks can be completed without human intervention.

Direction adjustment and course-correction ability measures whether the agent can quickly understand a problem and adjust subsequent behavior after receiving user corrections. Real tasks rarely follow a single path from start to finish; users are likely to request changes mid-task. If the agent cannot change direction based on correction information, it will fall into repeated errors or ineffective operations.

These two metrics are not isolated. Command reliability correlates with task success rate, while course-correction ability affects task recovery capability and user collaboration experience. A high-quality agent must not only perform well on the initial path but also recover reliably when deviations occur.

5. Applying This Evaluation in Engineering Practice

For enterprises and development teams, this evaluation method can be applied at several key stages.

The first is model selection. Traditional leaderboards may focus on general question answering or fixed test sets, whereas evaluation based on real agent tasks provides references closer to business scenarios. Enterprises can focus on metrics such as command execution reliability and long-task completion rate according to their own workload types.

The second is version regression. After iterating a model or adjusting prompts and tool configurations, randomized evaluation can show whether net improvement rate has increased. Compared with looking at individual cases, this provides more quantitative decision support.

The third is online monitoring. Deploying signal extraction logic to the production environment enables continuous monitoring of agent performance changes and timely alerts when a metric drops abnormally. For example, a sudden rise in shell command error rate may be caused by an environment change or a model update.

The fourth is data feedback. Real sessions and error samples accumulated during evaluation are an important data source for subsequent fine-tuning and prompt optimization. Analyzing high-frequency error types enables targeted improvements to agent behavior.

In engineering implementation, the evaluation system needs the following capabilities:

session log collection -> automatic signal parsing -> randomized assignment records -> net improvement rate calculation -> visualization reports

Session logs need desensitization to avoid leaking sensitive information. Signal parsing can combine rules and models to balance efficiency and accuracy. Randomized assignment records are the basis for net improvement rate calculation and must be traceable and recomputable. Visualization reports help users quickly locate problematic models or scenarios.

6. Implications for Developers

The core of agent evaluation is not pursuing an absolute score but building a measurement system that reflects real task performance. When building their own agent applications, developers can follow these practical principles.

First, extend the evaluation focus from “response quality” to “task outcome.” No matter how fluent a response is, its business value is limited if it cannot complete the user’s task. Evaluation should therefore include process signals such as task completion confirmation, command execution results, and correction success rate.

Second, emphasize randomization and causal inference. Simply comparing the average scores of two models is easily affected by task difficulty differences; randomized assignment and net improvement rate reduce this bias and make comparisons more informative.

Third, pay attention to recovery capability in long-horizon tasks. An agent cannot do everything right the first time; what matters is how it recovers after errors. Compared with optimizing only initial accuracy, improving course correction and error recovery often delivers more noticeable user experience gains.

Finally, keep the evaluation system open and interpretable. Publishing metric definitions, signal definitions, and statistical methods helps align teams internally and accumulate experience across model version iterations.

7. Conclusion

Agent evaluation is evolving from “whose response is better” to “who can better complete real tasks.” Evaluation methods based on real session signals and randomized inference offer a path that is closer to engineering practice for assessing agent capability. Metrics such as net improvement rate, shell command execution reliability, and course-correction ability help developers and enterprises understand model performance on long-horizon tasks more accurately.

As agents take on more responsibilities in office work, programming, data analysis, and automated operations, evaluation systems also need to become richer and more granular. Future agent capability assessment will not only focus on final results but will gradually extend to finer dimensions such as task planning, tool use, error recovery, and collaborative interaction, providing a more reliable data foundation for continuous agent optimization.