Evaluating AI Agents in Practice: From Real Sessions to Net Improvement Rate
Evaluating AI agents is fundamentally different from benchmarking chat models. This article explains why traditional pairwise human voting breaks down for agents, and presents a practical alternative: mining signals directly from real user sessions. It covers automatic extraction of multiple complementary signals (explicit task confirmation, praise and complaints, corrective instructions, command execution errors), and shows how randomized causal inference produces a Net Improvement Rate that isolates a model's true contribution from confounds like task difficulty. It also defines actionable sub-metrics such as shell reliability, steerability, and long-horizon task completion, plus engineering guidance for building your own evaluation pipeline. AI engineers, LLM researchers, and teams shipping agentic systems will find a rigorous, comparable, and explainable measurement framework they can adopt.
1. Why Agent Evaluation Is a New Problem
Traditional evaluation of conversational models is essentially “given a question, compare which answer is better” — it measures preference over single-turn text generation. An agent’s capability boundary is entirely different: it must autonomously plan, call tools, read and write to its environment, and execute commands across multiple turns, driving a real task forward over a long period until the user’s goal is met. The evaluation criterion is no longer “which response looks nicer,” but whether the task is actually completed, how it is completed, and whether it can be brought back on track after something goes wrong.
This introduces three fundamental shifts:
- The unit of evaluation moves from “the response” to “the task trajectory.” A single complete session may contain dozens of instructions, tool calls, and state changes; a single point’s quality cannot represent the overall outcome.
- Success signals are sparse and delayed. Task success often shows up as a single “that works” at the end of the session, not at any intermediate step.
- Failure modes are more subtle. Command execution errors, repeated course corrections, and off-target answers can all hide inside a conversation that looks smooth.
Evaluation methods therefore have to evolve from “human preference voting” to “structured observation of real tasks.”
2. The Structural Limits of Pairwise Human Voting
In pairwise voting evaluations, a reviewer looks at answers from model A and model B and picks the better one. This paradigm is simple and effective for single-turn dialogue, but when carried over to agent scenarios it produces clear structural biases:
- Fragmented perspective: The reviewer only sees truncated snippets and cannot perceive the global progress of a long-horizon task, making them prone to overrating polished phrasing.
- Mismatch between preference and completion: A more detailed, longer, or more polite answer is not necessarily closer to the task goal, and may even introduce noise.
- Systematic bias: Position bias, length bias, and style bias contaminate rankings and are hard to correct.
- High sampling cost: Accumulating enough pairwise votes for each model requires continuous human effort and struggles to cover the long tail of complex tasks.
The conclusion is that agent evaluation needs a new paradigm that does not rely on human voting and can extract signals directly from traces of real usage.
3. Turning Real Sessions into an Evaluation Data Source
The first step of the new paradigm is to acknowledge that “users have already voted with their feet.” When large numbers of real users collaborate with agents, they leave behind many high-value behavioral traces in their sessions. Rather than constructing extra votes, we can mine signals directly from these raw sessions. This offers two advantages: first, the data scale is naturally massive and can cover real long-horizon tasks; second, the signals come from users’ spontaneous behavior and are therefore closer to real satisfaction than controlled voting.
The key is converting “unstructured natural-language conversations” into “structured, evaluable signals.” The diagram below shows the conversion pipeline from session to metric:
Real session logs│├─ Signal extraction layer: identify task confirmation / praise & complaints / corrective instructions / command execution errors│├─ Attribution layer: align signals to a specific model, specific task, specific time slice│└─ Inference layer: randomized causal inference → Net Improvement Rate / sub-metrics
4. Automatic Multi-Signal Extraction
The core engineering of evaluation is automatically identifying trustworthy success/failure signals from session text. In practice, several complementary signal types are usually combined to reduce the bias of any single signal:
- Explicit task confirmation: When users say things like “done,” “exactly what I wanted,” or “that works” at the end of a session, this is the strongest positive signal.
- Praise and complaints: Explicit positive or negative feedback, used to correct the subjectivity of confirmation signals.
- Corrective instructions: When users interrupt midway and re-point toward the goal (“No, I meant…”, “Don’t do that yet”), this is direct evidence of task drift and can be counted negatively.
- Command execution errors: Shell command errors, non-zero exit codes, and timeouts are objective, low-cost failure signals that do not depend on subjective user expression.
A schematic signal extraction flow is shown below (pseudocode, illustrating the idea only):
def extract_signals(session):signals = {"confirmed": False, # task explicitly confirmed"praise": 0, # praise"complaint": 0, # complaint"corrections": 0, # number of corrections"cmd_errors": 0, # number of command execution errors}for turn in session.turns:if is_explicit_confirmation(turn.user_text):signals["confirmed"] = Truesignals["praise"] += count_praise(turn.user_text)signals["complaint"] += count_complaint(turn.user_text)signals["corrections"] += is_redirection(turn.user_text)signals["cmd_errors"] += turn.tool_exit_code_not_zeroreturn signals
Three points to watch in the engineering implementation: first, subjective signals (praise, confirmation) should have thresholds and deduplication to prevent a single exaggerated expression from being counted repeatedly; second, command errors should distinguish “expected exploratory errors” from “real failures”; third, corrective instructions should be decoupled from task difficulty to avoid misjudging “hard tasks necessarily need many corrections” as poor model capability.
5. From Correlation to Causality: The Net Improvement Rate
Even after extracting the signals above, simply averaging signals across models is still not rigorous enough, because there are serious confounders: different models are assigned to different tasks, different users, and different difficulties, and differences in a task’s own completability will systematically distort results. This is exactly why randomized causal inference needs to be introduced.
The core idea of this “causal tracking” style of randomized inference is using randomization to break confounding: in a controlled or approximately controlled setting, tasks and models are randomly paired so that the comparability conditions across models converge. Then, based on the “potential outcomes” framework, the causal contribution of a model relative to the global average is estimated — the “Net Improvement Rate.” It answers not “how many questions this model got right,” but “under the same conditions, how much net benefit did using this model bring compared to using the average.”
def net_improvement_rate(model_signal, global_baseline, task_controls):# After controlling for confounders such as task difficulty,# estimate the causal gain relative to the average leveladjusted = adjust_for_covariates(model_signal, task_controls)return (adjusted - global_baseline) / global_baseline
What makes this method rigorous is that it separates “looks better” from “actually better because of the model.” A model might have stronger signals simply because it happened to be assigned easier tasks, and causal inference strips out this windfall, keeping only the net effect attributable to the model itself. Across generations, the same methodology can also compare the improvement in Net Improvement Rate, quantifying the real progress brought by a version iteration rather than an illusion distorted by fluctuations in task distribution.
6. Designing Key Sub-Metrics
A single aggregate score cannot guide optimization, so the evaluation system should drill down to explainable, actionable sub-metrics. In agent scenarios, the following categories are especially important:
- Command execution reliability (Shell/Bash Reliability): Measures the accuracy and robustness of an agent when generating and executing system commands, including whether non-zero exit codes are frequently triggered and whether repeated retries are needed. This is the most basic trust threshold for coding and ops agents.
- Steerability: Measures the success rate and cost (number of corrections, time spent) of pulling an agent back on track via short instructions when it drifts off target. High steerability means the task “can still be saved even when it goes wrong,” which directly determines the practical usability of long-horizon tasks.
- Long-horizon task completion: In multi-turn tasks spanning many steps, evaluate the final completion rate rather than the local accuracy of intermediate steps.
- Scenario classification performance: Compute separately for work, coding, chat, and other scenarios, to avoid different capabilities being masked by an average score.
These metrics should form a two-tier “overall—sub-metric” structure with the Net Improvement Rate: the overall score gives a comparable overall position, while sub-metrics reveal the specific weak points that can be improved.
7. Engineering Recommendations
To build a similar agent evaluation system within your own team, you can start from the following points:
- Prioritize signal collection: First ensure session logs are complete and command exit codes are observable — this is the foundation for all subsequent inference.
- Fuse multiple signal sources: Let subjective confirmation and objective execution errors corroborate each other, avoiding single-signal manipulation or misreading.
- Stick to randomized control: Even at a modest scale, randomize task-to-model assignment as much as possible; otherwise causal conclusions are unreliable.
- Make metrics action-oriented: Sub-metrics should map directly to optimization directions — for example, if reliability is low, focus on tool calling and command generation.
- Continuously calibrate and debias: Periodically inspect potential biases such as position, length, and difficulty to keep the evaluation methodology stable.
8. Conclusion
Agent evaluation is moving from “human preference comparison” to “causal measurement based on real sessions.” Using real user sessions as the data source, automatically extracting multi-dimensional signals such as task confirmation and correction, and command execution errors, then applying randomized causal inference to obtain the Net Improvement Rate, this method is not only closer to real task performance but also provides a rigorous, comparable, and explainable measurement standard for model iteration. Understanding and adopting this paradigm is a key step in building trustworthy agents and continuously quantifying their real value.
