Evaluating AI Agents on Real-World Tasks: From Session Signals to Causal Inference

1. Background: Why Voting-Based Evaluation Is No Longer Enough

As AI agents move from demos to production, evaluating them has become harder than ever. Traditional evaluation of conversational models typically relies on two approaches: static benchmark sets and human preference voting. Static benchmarks suffer from fixed question sets that models can easily overfit, and they struggle to cover long-horizon, multi-step tasks that involve tool calls. Preference voting is expensive, produces sparse samples, and voter subjectivity tends to mask differences in whether a task was actually completed.

The core value of an agent lies in getting things done, not just sounding good. An agent may produce a perfectly fluent response while repeatedly failing to execute shell commands, failing to converge after the user pushes back, or omitting an explicit user requirement at a critical confirmation step. None of this shows up clearly in pairwise voting, yet it is exactly what shapes the real user experience.

This points to an evaluation approach that is closer to production reality: instead of asking users “which response is better,” go back to real sessions and observe what users actually did—whether they confirmed the task was complete, asked for corrections, expressed dissatisfaction, or retried repeatedly because a command kept failing. These behavioral signals occur naturally, require no extra annotation, and correlate strongly with task success.

2. Method Overview: Signal Extraction + Causal Tracing

The evaluation method consists of two sequential stages.

2.1 Stage 1: Automatically Extracting Key Signals from Real Sessions

The evaluation system mines real agent sessions from actual users and automatically identifies several high-value signal types, including:

• Explicit confirmation: the user acknowledges the task result, for example confirming the problem is solved or accepting the output;
• Praise and complaints: explicit positive or negative feedback;
• Correction requests: the user asks the agent to change direction, fix a mistake, or try a different approach;
• Command execution errors: tool calls (especially command-line operations) return failures, errors, or unexpected results.

What these signals have in common is that they are not labels constructed by evaluators. They are behavioral traces users leave naturally during real usage. Compared with asking users to pick between two options, these signals are sparser but more authentic, and much harder for a model to game against a fixed evaluation set.

On the engineering side, signal extraction has to handle several practical issues: sessions can be long and multi-turn with interleaved topics; a single correction request may contain multiple intents; and command execution errors must be aligned with the specific tool-call context. The extraction layer therefore typically performs session segmentation, intent classification, and context binding, so that every signal can be traced back to a specific task segment.

2.2 Stage 2: Estimating the Net Improvement Rate with Randomized Causal Tracing

Signals alone are not enough, because sessions differ widely in task difficulty, user habits, and domain mix. Directly counting “complaint rates” introduces serious selection bias: some models may simply have been assigned harder tasks.

To address this, the method introduces a randomized inference technique called causal tracing. The core idea is to introduce controlled randomization into real traffic so that the task distributions faced by different models are statistically comparable. On that basis, it estimates each model’s net improvement rate relative to the overall average—that is, an average-level baseline.

The net improvement rate can be understood as follows: after controlling for task difficulty and user distribution, the net benefit a model delivers on key signals compared with average performance. It is a relative quantity rather than an absolute score, which makes it better suited for cross-model comparison and more robust to task distribution drift. The value of randomized inference is that it pushes correlation toward causality—not simply “sessions using this model had fewer complaints,” but “under comparable conditions, this model reduced complaints.”

It is worth emphasizing that the reliability of this approach depends on two prerequisites: the quality of the randomized assignment itself, and the stability of signal definitions. If the signal extraction criteria change frequently between versions, cross-version comparability of net improvement rates weakens. In practice, teams therefore usually maintain versioned signal definitions together with regression checks.

3. Metric Design: Composite Net Improvement Rate and Sub-Dimensions

The output of an evaluation should not be a single aggregate score but a set of interpretable metrics. A typical design looks like this:

Metric	Meaning	What It Reveals
Composite net improvement rate	Overall net gain relative to the average level	Global cross-model comparison
Command execution reliability	Share of tool/CLI calls that succeed with expected results	Execution-layer stability
Steerability	Ability to converge to the right direction after user correction	Interaction robustness
Task confirmation rate	Share of tasks the user explicitly confirms as complete	End-to-end completion

Steerability deserves particular attention. In real tasks, users will almost inevitably step in with corrections, and whether a model can quickly adjust after a correction says more about production readiness than getting something right on the first try. Command execution reliability, meanwhile, maps directly to the capability that separates agents from pure conversational models: an agent must operate tools reliably, not merely describe how to operate them.

The point of sub-dimension metrics is diagnosis. The composite net improvement rate tells you who is better; the sub-dimensions tell you where a model is better or worse. That is far more actionable for deciding what to iterate on next.

4. A Reference Evaluation Workflow

The following abstracted workflow (pseudocode, for logic only) shows how the pieces fit together:

# 1. Session collection and segmentation
sessions = collect_real_sessions(min_turns=..., min_duration=...)
tasks = [split_into_task_segments(s) for s in sessions]
# 2. Randomized assignment (prerequisite for causal tracing)
assignments = randomized_assign(tasks, models, ratio=...)
# 3. Signal extraction
signals = []
for task, model in assignments:
    signals += extract_signals(task, model)
    # Signal types: confirm / praise / complaint / correction / tool_error
# 4. Net improvement rate estimation
baseline = estimate_average_level(signals)
for model in models:
    net_gain[model] = causal_estimate(signals, model, baseline)
# 5. Sub-dimension aggregation and confidence intervals
report = aggregate_by_dimension(net_gain, dims=["reliability", "steerability", ...])

Three engineering points are critical in this workflow: validating the balance of the randomized assignment, measuring the precision and recall of signal extraction, and reporting confidence intervals alongside net improvement estimates. A ranking without confidence intervals is easily misread as a definitive conclusion.

5. Practical Considerations

First, sample size and latency. Collecting long-horizon real sessions and extracting signals from them both take time, so evaluation results naturally lag behind. Define the statistical window explicitly and avoid drawing conclusions from short-term fluctuations.

Second, signal noise. A user complaint does not necessarily mean the model made a mistake (the task itself may be infeasible), and a user confirmation does not necessarily mean high quality (the user may simply have given up). Signals therefore need context-aware filtering, plus periodic human spot checks for calibration.

Third, maintaining comparability. When the set of models being evaluated changes, the average-level baseline drifts, and historical net improvement rates cannot be compared directly across batches. It is advisable to fix a baseline version or report differences relative to the baseline.

Fourth, avoid overfitting the evaluation. Once evaluation signals become public and stable, they invite targeted optimization. Keeping part of the signal definitions private and rotating them periodically is a common way to preserve the validity of the evaluation.

6. Conclusion

The hard part of agent evaluation is not assigning scores—it is producing a score that holds up on real tasks. An evaluation paradigm that uses real sessions as its data source, behavioral signals as its observations, and randomized inference as its estimation method essentially moves evaluation from subjective preference comparison toward attributable causal estimation. For developers, the value of understanding this methodology goes beyond reading a leaderboard: it offers a transferable approach. When you need to evaluate how your own agent performs in a real business, you can likewise start from user behavior signals and use controlled experiments to estimate net gains—rather than relying on gut feeling or scattered anecdotes.