Agent Evaluation: From Signal Extraction to Causal Inference in Practice

Introduction

Large language models are evolving from single-turn text generation into agent-based systems, shifting evaluation toward real-world task execution. The traditional “side-by-side comparison plus user voting” approach suffers from subjective bias and context fragmentation. This article focuses on a new automated evaluation paradigm built on real-world session mining and causal tracking. It breaks down the core mechanisms, examines measured performance of open-weight models, and outlines an enterprise adoption path, offering developers a practical guide for agent evaluation and model selection.

Why Agent Evaluation Needs a New Paradigm

When a model is wrapped into an agent and connected to business systems, its capability surface expands significantly. It must not only understand language, but also handle intent recognition, tool planning, error recovery, and direction correction across dozens of conversational turns. Traditional evaluation faces three major pain points:

Context fragmentation — Static test sets cannot reproduce long-running sessions, making it difficult to assess memory and state management.

Subjective voting bias — Ordinary users often cannot accurately judge technical details such as shell errors or API parameters.

Lack of causal inference — Comparing raw success rates cannot isolate confounding variables like task difficulty or environment differences.

Evaluation therefore needs to move from “subjective perception” to a new paradigm based on “objective signals from real sessions and causal inference.”

How to Extract Key Signals from Real Sessions: Mechanism and Code

The new evaluation framework mines agent session logs from real business scenarios, converting natural user feedback and system runtime status into quantifiable signals. Three core signal categories are used:

• Explicit user confirmation: Directly accepting a result or expressing thanks, reflecting intent alignment and delivery effectiveness.
• User correction commands: Statements like “that’s wrong” or “redo it,” serving as the key basis for measuring steerability — the ability to adjust direction and correct course.
• Shell execution errors: Non-zero exit codes or abnormal stack traces reflecting command reliability, though false positives caused by missing environment packages need to be filtered out.

Note: In the pseudocode below, intent_classifier requires a pretrained NLU model (such as a fine-tuned BERT), with clearly defined training data and classification thresholds. The shell_error signal must filter out non-model factors such as environment configuration anomalies.

def extract_agent_signals(session_log):
    """Extract evaluation signals from session logs after deduplication and cleaning."""
    # Pre-processing: data cleaning and deduplication
    cleaned_logs = deduplicate_and_clean(session_log)
    signals = {
        "explicit_confirm": 0,
        "complaint_correct": 0,
        "shell_error": 0
    }
    for turn in cleaned_logs.turns:
        # 1. Identify user correction or complaint intent (requires NLU classifier, not keyword matching)
        if turn.role == "user" and nlu_classifier(turn.text) in ["complaint", "correction"]:
            signals["complaint_correct"] += 1
        # 2. Identify explicit confirmation
        elif turn.role == "user" and "thanks" in turn.text.lower():
            signals["explicit_confirm"] += 1
        # 3. Identify shell execution errors (excluding environment issues)
        elif turn.role == "tool" and turn.tool_name == "shell_executor":
            if turn.exit_code != 0 or "error" in turn.stderr.lower():
                # Secondary validation: exclude non-model causes such as missing environment packages
                if not is_environment_issue(turn):
                    signals["shell_error"] += 1
    return signals

By aggregating such signals globally, a large and realistic objective evaluation base can be constructed. Steerability can be quantified using the formula: Steerability Score = 1 - (number of corrections / total interaction turns). A threshold of ≥0.85 is recommended for excellent performance.

How Causal Tracking Estimates Net Improvement: Steps and Limitations

After extracting multidimensional signals, causal tracking converts them into comparable scores. The core logic simulates A/B testing by using propensity score matching to control for confounding variables and estimate a model’s net improvement over the global baseline.

The main steps:

1. Task difficulty stratification: Cluster tasks into layers based on historical data, including step count and tool invocation depth.
2. Baseline expectation calculation: Compute the average performance of all evaluated models within each difficulty layer.
3. Causal effect estimation: Observe the target model’s actual performance and use a causal graph model to remove noise such as user operational errors.
4. Net improvement output: Derive the net percentage improvement over the global average baseline.

Note: Causal inference relies on sufficient sample size and randomization quality. In low-traffic scenarios, confidence intervals may be wide, so Bootstrap validation is recommended. Residual confounding variables always exist, so results should be treated as reference rather than definitive conclusions.

Open-Weight Model Performance: What the Data Shows

According to publicly available agent evaluation leaderboard data based on more than 17,500 real user sessions globally, a certain open-source model (such as the Qwen3 series and other public models) ranks first among open-weight models with approximately 2.8 trillion parameters and optimized alignment strategies. It places 13th among all models overall, with a net improvement rate of +5.1%, a 4.2 percentage point jump from the previous generation’s +0.9%.

The table below summarizes its ranking among open models and net improvement rate across three core scenarios:

Scenario	Open Model Rank	Net Improvement
Work-related tasks (documents, retrieval, reasoning)	1st	+4.8%
Programming tasks (generation, debugging, testing)	1st	+5.3%
Chat tasks (long conversations, role consistency)	1st	+4.6%

The model also ranks near the top on metrics such as “shell command execution reliability” and “steerability and correction capability,” indicating strength in generating executable code and recovering quickly after user corrections.

How Enterprises Can Select and Deploy: A Four-Step Path

1. Build an internal evaluation base: Do not rely solely on external leaderboards. Set up a session log collection pipeline tailored to your business, and use object storage to structure and extract key signals.
2. Prioritize steerability: Simulate frequent anomaly scenarios to test model recovery speed. Steerability Score should be a core selection metric.
3. Introduce causal inference for A/B testing: Randomly split identical traffic pools, remove task difficulty fluctuations, and accurately calculate the net benefit of a new model.
4. Leverage cloud-native infrastructure: Use mainstream cloud-native AI service platforms (such as Baidu AI Cloud, AWS, and others) to achieve high-concurrency scheduling and log aggregation, building an automated evaluation flywheel.

Conclusion

Agent evaluation is moving toward objective quantification: extracting multidimensional signals from real sessions, using causal tracking to strip away noise, and ultimately producing a credible net improvement rate. The rapid progress of open-weight models offers enterprises a cost-effective option. Developers should pay attention to the boundary conditions of signal extraction and the sample requirements of causal inference, and build an iterable evaluation system aligned with their own business needs. That is the core metric for successful agent deployment.