Evaluating AI Agents on Real-World Tasks: Methodologies and Practices

1. Why Traditional Evaluation Fails in the Agent Era

The fundamental difference between AI agents and standard large language model (LLM) Q&A lies in their operational lifecycle. Rather than generating a single text response, agents engage in multi-turn interactions, invoke tools, execute commands, observe environmental feedback, and continuously correct their behavior. This long-term, highly interactive nature subjects traditional evaluation paradigms to three major challenges:

First, rapid saturation of static benchmarks. Evaluation sets based on fixed question banks are susceptible to targeted optimization once published. An increase in scores does not necessarily correspond to an improvement in real-world capabilities. Furthermore, the distribution of questions is often narrower than what is found in production environments, making it difficult to assess critical dimensions like tool invocation and error recovery.

Second, subjective noise in human voting. Asking users to choose between two responses via preference voting works adequately for short Q&A scenarios. However, in long-term tasks, users typically only see the final result and cannot judge the rationality of intermediate steps. Additionally, voting behavior is easily influenced by irrelevant factors like UI layout and phrasing style, resulting in a low signal-to-noise ratio.

Third, unsustainable cost and scale. The marginal cost of high-quality human annotation is extremely high. A single agent task session might span dozens of turns and contain extensive tool invocation logs, making manual, turn-by-turn review engineering infeasible.

Therefore, the focus of evaluation must shift from “constructing test questions” to “observing real-world usage,” and from “subjective preferences” to “observable behavioral signals.”

2. Mining Real Conversation Signals: The Data Foundation

When collaborating with agents, real users inadvertently leave behind a wealth of high-value feedback. Because this feedback is not explicitly generated for evaluation purposes, it is harder to optimize for and possesses higher ecological validity. The extractable signals can be broadly categorized into four types.

2.1 Explicit Confirmation and Positive/Negative Feedback

When a user says “Yes, exactly” or “No problem,” it constitutes an explicit confirmation of task completion. Conversely, phrases like “That’s wrong,” “You made a mistake,” or “Start over” serve as negative signals. The key to processing these signals is attribution alignment: it requires determining which specific model output turn or tool invocation result the feedback targets, rather than applying it broadly to the entire session.

2.2 Correction Instructions

Corrective statements given by users after an agent deviates from the intended path (e.g., “Don’t modify the config file, only change the test cases”) serve as golden data for evaluating “steerability.” This data simultaneously contains two pieces of information: the model previously went in the wrong direction, and whether the model correctly followed the user’s intervention. The latter is especially crucial because, in real-world collaboration, users are willing to correct mistakes but unwilling to correct the same mistake repeatedly.

2.3 Tool Execution Errors

In agent scenarios equipped with command execution capabilities, hard signals such as non-zero shell exits, parameter spelling errors, and non-existent paths can be automatically collected. These signals are objective, require no human judgment, and directly correspond to the core capability dimension of “execution reliability.” It is important to distinguish between “failures caused by the environment itself” and “failures caused by the model generating incorrect commands.” The former should be filtered out or modeled separately in statistical analysis.

2.4 Session Structural Signals

Structural characteristics—such as whether a task was abandoned midway, abnormal inflation of session turns, or repeated attempts at the same failed operation—are also components of a capability profile. An efficient agent should converge to a solution in fewer turns rather than falling into an infinite loop.

3. Causal Tracing: From Correlation to Net Improvement Rate

Having signals is not enough; the key is estimating “how much better a specific model is compared to the average level.” Directly comparing the average approval rates of different models introduces severe confounding: different models are assigned tasks of varying difficulty, user demographics, and session lengths.

The core concept of Causal Tracing is introducing randomization. In real traffic, model assignment for the same type of task request is randomized, ensuring that confounding factors like task difficulty and user characteristics are balanced across models. Based on this, inferential methods are used to estimate each model’s Net Improvement Rate relative to the overall average level—essentially isolating the treatment effect from confounders.

Key engineering implementation points include:

• Randomization unit selection: Randomize assignment at the session or task level to prevent interference caused by assigning multiple models to the same user on the same task.
• Stratified randomization: Stratify by task type (e.g., workflow orchestration, code modification, conversational Q&A) to ensure each model has samples within every stratum, thereby supporting multi-dimensional rankings.
• Effect estimation and confidence intervals: The net improvement rate should be output with confidence intervals. Models with insufficient sample sizes should not enter the main leaderboard to prevent small-sample noise from misleading results.
• Covariate adjustment: Adjust for session-level covariates like turn count and task duration to further reduce variance.

The following pseudocode illustrates the process of signal aggregation and effect estimation:

# Pseudocode: Session-level signal aggregation and net improvement rate estimation
def aggregate_session_signals(session):
    signals = []
    for turn in session.turns:
        if turn.user_feedback == "confirm":
            signals.append(("task_success", +1.0))
        elif turn.user_feedback == "complaint":
            signals.append(("task_success", -1.0))
        if turn.is_correction:
            signals.append(("steerability", -1.0 if turn.followed else +0.0))
        if turn.tool_call and turn.tool_call.exit_code != 0:
            signals.append(("bash_reliability", -1.0))
    return normalize_by_category(signals)
def estimate_net_improvement(sessions, model_id, strata):
    # Compare the model with the stratum average, then aggregate with weights
    effects = []
    for stratum in strata:
        treated = [s for s in sessions if s.model == model_id and s.stratum == stratum]
        control = [s for s in sessions if s.model != model_id and s.stratum == stratum]
        effects.append(weighted_mean_diff(treated, control))
    return combine_with_ci(effects)

The above process emphasizes two points: signals must be aggregated separately by capability dimension (success rate, steerability, and execution reliability cannot be conflated), and effects must be estimated by stratum before aggregation, otherwise differing task proportions will skew the conclusions.

4. Metric Design: From a Single Score to a Capability Profile

The value of real-world task evaluation lies not in providing a single total score, but in characterizing the capability structure. It is recommended to cover at least the following dimensions:

1. Task Success Rate: Measured by the proportion of sessions ending with explicit confirmation and no complaints. The truncation issue of “no user feedback” must be addressed.
2. Execution Reliability: Tool invocation success rate, command exit code distribution, and retry counts.
3. Steerability: The proportion of times the model correctly follows user corrections, along with the distribution of correction counts.
4. Convergence Efficiency: The number of turns and tool invocations required to achieve the goal, normalized relative to a task baseline.
5. Stability: Variance estimates across multiple assignments of the same task, reflecting output consistency.

Multi-dimensional rankings are more instructive than overall rankings: one model might exhibit extremely strong execution reliability but average steerability, while another might be the exact opposite. Developers should use these profiles to choose a solution that best fits their specific scenario.

5. Engineering Practices for Long-Term Task Evaluation

Implementing the above methodology into a sustainable evaluation system requires focusing on the following engineering aspects.

5.1 Data Collection and Desensitization

Session data contains both user inputs and tool outputs. Desensitization and compliance reviews must be completed prior to collection to remove identity and sensitive information. The collection pipeline should perform integrity checks on sessions, discarding truncated or corrupted records to prevent statistical pollution.

5.2 Automation and Sampling for Signal Annotation

Automatic signal extraction rules require continuous calibration. It is recommended to conduct a small-scale manual sampling inspection of automated annotation results to estimate accuracy and retroactively adjust rules if accuracy drops. For highly ambiguous signals (e.g., statements implying implicit dissatisfaction), it is better to label them as “undetermined” than to force a categorization.

5.3 Sample Size and Leaderboard Admission

Set a minimum sample size threshold and report confidence intervals for each model. Newly added models can initially enter an observation zone before moving to the main leaderboard once sufficient samples are accumulated. Multi-dimensional rankings also require sample size constraints to prevent false superiority in a dimension due to sparse samples.

5.4 Preventing Overfitting and Maintaining Ecological Validity

Real conversation data itself can be targeted for optimization. Therefore, evaluators should retain an undisclosed subset of sessions for cross-validation and periodically rotate the statistical window to prevent models from locally optimizing for specific traffic characteristics over certain time periods.

6. Conclusion

AI agent evaluation is shifting from the static paradigm of “testing—answering—scoring” to a dynamic paradigm of “observing real-world usage—extracting behavioral signals—inferring causal effects.” The technical core of this transition relies on three points: using real long-term sessions as the data source to ensure ecological validity; replacing subjective preferences with multi-dimensional behavioral signals to ensure objectivity; and using randomized inference to strip away confounding factors to ensure credible conclusions. For developers, understanding this methodology not only aids in correctly interpreting public leaderboards but also helps in building an agent quality assessment system within their own businesses that closely mirrors production and iterates sustainably. Ultimately, an agent’s value is determined by its performance in real-world tasks, not by a single line of numbers on a benchmark set.