Evaluating AI Agents with Real-Session Signals and Causal Inference

Why Real-Task-Driven Evaluation Matters for AI Agents

Traditional LLM evaluation follows two main paths: static benchmark question sets and human pairwise voting. Static benchmarks are prone to “benchmark hacking” and often diverge from real-world usage. Human voting captures user preference but has clear limitations: samples skew toward short Q&A, scores depend on subjective opinion, and both approaches fail to cover the long-horizon, multi-turn, tool-augmented tasks that define modern AI agents.

Agentic tasks differ fundamentally from ordinary chat. A single session can span dozens of turns, involving file reads and writes, command execution, code modifications, and multi-step planning. Success is not determined by how well the model answers, but by whether the task gets done. Evaluation must therefore shift from “response quality” to “task outcome,” and from “user votes” to “behavioral signals.” Real-session-driven evaluation returns the judgment to the actual task itself.

Core Idea: Automatically Extracting Signals from Real Sessions

Real sessions contain rich supervision signals. By automatically mining production conversation logs, four high-value signal types can be extracted:

1. Explicit confirmation: Positive user confirmation of results (e.g., “that works,” “exactly what I needed”) — strong evidence of task completion.
2. Praise and complaints: Emotionally charged positive and negative feedback, reflecting experience quality and issue severity.
3. Correction instructions: User requests to change direction, switch approaches, or redo work — indicating the previous path deviated from expectations.
4. Command execution errors: Non-zero exit codes, command-not-found errors, permission denials — directly measuring execution reliability.

In practice, a “rules + model” dual-channel approach is used. High-frequency, pattern-stable signals are captured with regex or template matching. Semantically ambiguous feedback is passed to an LLM for structured extraction, which outputs a unified schema. Periodic sampling with human annotation is then used for quality calibration.

# Simplified signal extraction schema
SCHEMA = {
  "confirm":    "bool",   # user explicitly confirms task completion
  "praise":     "bool",   # positive feedback
  "complaint":  "bool",   # negative feedback
  "correction": "bool",   # correction or redirection instruction
  "cmd_error":  "list"    # list of command execution error types
}

The quality of the extraction layer determines the ceiling of the entire evaluation system. Extraction accuracy should be monitored periodically, and low-confidence samples should be routed through manual review.

Causal Tracking: Using Randomized Inference to Estimate Net Improvement

Signals alone are not enough. Directly comparing “task completion rates” across models introduces two confounding factors: different task difficulty and different user demographics. The solution is to randomly assign live traffic to different models at the session level, treating “model choice” as a randomized treatment variable, and using randomized inference to estimate the average treatment effect.

“Net improvement rate” can be understood as: the amount of task-outcome improvement for a given model’s sessions, relative to the average across all evaluated models. Illustration:

# Simplified randomized inference
def net_improvement(sessions, model_id, B=10000):
    y_t = [s.outcome for s in sessions if s.model == model_id]
    y_c = [s.outcome for s in sessions if s.model != model_id]
    point = mean(y_t) - mean(y_c)              # net effect relative to average
    ci = permutation_ci(sessions, model_id, B) # permutation test confidence interval
    return point, ci

The outcome can be a weighted composite of signals, for example: confirmation +1, complaint -1, correction -0.5, and command errors deducted per occurrence, then normalized at the session level. Because assignment is random, the confidence interval of the net improvement rate covers sampling noise. If the interval does not cross zero, the difference can be considered statistically significant.

Example (fictional numbers for illustration): if a model shows a net improvement rate of +3.2% with a confidence interval of [+1.8%, +4.6%], its session outcomes are significantly better than average. If another model shows -1.5% but the interval crosses zero, we cannot assert that it is worse — only that it is roughly on par with the average.

Metric System: From Composite Score to Breakdown Dimensions

The composite net improvement rate answers “who is better overall,” but troubleshooting and model selection require finer granularity:

• Command execution reliability: first-pass success rate of commands, self-repair rate after errors, retry convergence speed.
• Correction handling: after receiving a correction instruction, how many turns it takes to return to the right direction, and whether the same mistake recurs.
• Instruction following: how well constraints are maintained across long multi-turn tasks.
• Scenario breakdown: aggregate separately for work, coding, chat, and other scenarios to avoid “one strong metric masking other weaknesses.”

For long-horizon tasks, session-level aggregation is crucial: use “session” rather than “single turn” as the statistical unit to avoid duplicate counting in multi-turn tasks. Also stratify by session duration and tool usage intensity to control for task-difficulty confounding.

Practical Implementation: Five Steps to Build Your Own Evaluation Pipeline

1. Data collection and compliance: Before collecting production session logs, complete de-identification (removing personal information, secrets, internal addresses). Apply tiered storage and access control according to compliance requirements.
2. Signal extraction and quality control: Establish a unified schema. Run rule-based and model-based channels in parallel. Align with manual annotation on samples, and continuously monitor extraction accuracy.
3. Randomized traffic splitting: Randomly assign models at the session level and log metadata. Avoid user-level assignment, which introduces individual-level confounding.
4. Statistical inference and dashboards: Regularly compute net improvement rates and breakdown metrics for each model, output confidence intervals, and set up monitoring alerts to catch abnormal regressions.
5. Iteration and regression testing: Before releasing a new version, replay historical sessions for regression evaluation. After release, validate with real traffic. This forms a closed loop combining “offline replay + online randomization.”

At the infrastructure level, you can assemble this pipeline from general-purpose cloud components. For example, on Baidu AI Cloud: model serving handles multi-version inference and gradual traffic splitting, log service aggregates session and command execution events, object storage retains raw logs, monitoring and alerting set thresholds on net improvement rates, and a data warehouse supports cross-version metric comparison. The entire solution relies on no proprietary interfaces and can be assembled from standard building blocks.

Common Pitfalls and Best Practices

• Selection bias: Only counting sessions that reach N turns or more systematically underestimates problems in scenarios where users give up easily. Preserve the full funnel.
• Signal overfitting: Evaluation signals must not be fed back to filter training prompts; otherwise metrics will inflate artificially.
• Data drift: User demographics and task types change over time. Re-estimate using sliding windows and run periodic significance tests.
• Small-sample noise: When samples are scarce in a breakdown dimension, confidence intervals widen considerably. Never draw conclusions from point estimates alone.
• Privacy and audit: Command execution logs may contain sensitive information. De-identification must happen before extraction, and all access should be logged and auditable.

Summary

Real-session-driven evaluation returns the question of “is this model good” to real tasks: replace human voting with automatic signal extraction, replace naive comparison with randomized inference, and use net improvement rates paired with breakdown dimensions to characterize capability differences. For engineering teams, the value of this methodology goes beyond ranking models — it establishes a sustainable, explainable, regression-ready capability measurement system, where every model upgrade is validated by real business outcomes.