Evaluating AI Agents on Real-World Tasks: From Session Signals to Causal Attribution

    How do you measure whether an AI agent actually completes real work? This article covers extracting reliable signals from real user sessions, converting them into comparable metrics, applying randomized causal inference to control confounders, and breaking results down into actionable capability dimensions.

    Why Evaluate Agents on Real-World Tasks

    AI agents differ from traditional conversational models in that they need to read environment state, call tools, execute commands, adjust their plans based on intermediate results, and keep making progress toward a goal over long sessions. These capabilities are hard to measure with single-turn Q&A benchmarks. A question with a standard answer cannot tell you whether an agent still stays on track after executing a dozen steps.

    Evaluation therefore needs to shift its focus to real sessions. Two core questions arise: where does evaluation data come from, and how do you attribute results to the model itself?

    Replacing Human Voting with Session Signals

    Common comparison-based evaluations rely on users voting between two responses. This approach works well for judging the linguistic quality of answers, but it poorly reflects how agents perform on long tasks. Voting results are also easily skewed by interface design, response length, and user mood.

    An alternative is to extract signals directly from interaction traces in real sessions. The following signal types tend to be most valuable:

    • Explicit confirmation: the user states the task is complete, for example by confirming the result is correct or asking to move on to the next step.
    • Positive feedback: the user expresses satisfaction with the result.
    • Negative feedback: the user complains, reports failure, or asks for a redo.
    • Correction instructions: the user points out an error and specifies a fix, such as “the output path is wrong—it should go under the results directory.”
    • Execution errors: shell commands return non-zero exit codes, time out, or hit permission denials.

    These signals can be identified by a combination of rules and classification models. Rules capture high-confidence expressions, while classifiers handle colloquial or implicit feedback.

    Turning Signals into Comparable Metrics

    Signals on their own are just counts. Sessions vary widely in length and difficulty, so comparing raw counts directly is meaningless. Signals need to be converted into relative metrics.

    One intuitive definition is the net improvement rate relative to a baseline. Let p_model be the proportion of positive outcomes for a given model on a task category, and p_avg the average proportion across all models on the same category:

    Net improvement rate = (p_model − p_avg) / p_avg

    This definition captures deviation from the average level. For example, if the average is 40% and a model reaches 42%, the relative improvement is about 5%. This number only indicates relative position, not absolute capability.

    When designing metrics, keep the following in mind:

    1. Compute per task type first, then aggregate with weights, so that one high-volume task type does not drown out others.
    2. Weight signals differently. Explicit confirmations are generally more trustworthy than generic praise and should carry more weight.
    3. Exclude invalid sessions, such as those where the user leaves midway or chats about things unrelated to the task.

    Controlling Confounders with Randomized Causal Inference

    Session outcomes are influenced by many factors, including task difficulty, user experience, domain, and prompt quality. If you directly compare success rates of different models across different user groups, the difference may come from the user mix rather than the models themselves.

    The basic idea of randomized causal inference is to assign tasks to different models randomly, so that confounders become statistically balanced across groups. When direct random assignment is not possible, you can randomize user–task matching or apply weighting adjustments on known covariates.

    Below is illustrative pseudocode for estimating a model’s effect relative to the average:

    1. for session in sessions:
    2. task_type = classify(session.task)
    3. model = session.assigned_model
    4. outcome = extract_outcome(session.signals) # positive = 1, negative = 0, neutral ignored
    5. group_stats = aggregate(sessions, by=[model, task_type])
    6. baseline = weighted_mean(outcome over all models, weights=task_type_share)
    7. for model in models:
    8. effect = group_stats[model].mean - baseline
    9. ci = resample_confidence_interval(effect, data=sessions)
    10. report(model, effect, ci)

    This approach also produces interval estimates, avoiding reliance on a single point estimate. A wide interval indicates that the sample size is insufficient to support fine-grained ranking conclusions.

    Breaking Down Key Capability Dimensions

    Aggregate metrics are fine for rough ranking, but engineering decisions require decomposition into specific capabilities. The following four dimensions are most closely tied to real agent performance.

    1. Command Execution Reliability

    This measures whether commands generated by the agent actually execute correctly in real environments. Focus areas include syntax correctness, recognition of exit codes and retry behavior, judgment on permissions and paths, and timeout handling for long-running commands. A reliable agent reads error messages and adjusts parameters after a command fails, rather than rerunning the same command.

    2. Course Correction Ability

    This measures whether the agent, after receiving a mid-task correction, can quickly revise its plan and stay aligned with the user’s goal. Criteria include whether the user-specified action was executed, whether the fix introduced new errors, and whether subsequent steps carried the correction forward.

    3. Completion Verification Quality

    This measures whether the agent provides verifiable evidence when claiming a task is done. A reply that only says “done” without the location of deliverables, validation results, or key outputs should be considered low quality, even if the user never complains.

    4. Scenario Grouping

    Office document processing, code modification, and everyday Q&A each fail in different ways. Evaluating by scenario group prevents a single average from masking obvious weaknesses in a particular category.

    Building a Similar Evaluation System in Your Own Stack

    If your team wants to evaluate its own agents, proceed as follows.

    Step one: instrument data collection. Log every user turn, agent action, tool call result, and each command’s exit code and duration. Session logs may contain sensitive information, so sanitize at the collection point and restrict access.

    Step two: define a signal dictionary and rules. Organize expressions for “confirmation,” “complaint,” “correction,” and so on into a rule library. Sample and manually label by category on a regular basis, and periodically check the rules’ precision and recall.

    Step three: stratification and assignment. Stratify by task type, assign model versions as randomly as possible, and record assignment results completely to support later causal analysis.

    Step four: compute metrics and intervals. Use the net improvement rate and interval estimates described above, and when reporting conclusions always note the sample size, time range, and confidence intervals.

    Step five: replay and attribute. Replay negative samples to determine whether the problem stems from model capability, tool failure, or unclear user input. Attribution results guide improvement far better than a single overall score.

    Limitations of Evaluation Conclusions

    • Signals can be biased. Satisfied users are more likely to confirm success, while dissatisfied users may simply leave, causing negative signals to be underestimated.
    • Rankings are sensitive to sample distribution. The same model may perform noticeably differently across user groups.
    • Relative metrics cannot be directly converted into absolute experience. A top-ranked model may still fail frequently in certain specific scenarios.
    • Public leaderboards are useful for initial screening but cannot replace acceptance testing against your own business needs.

    Practical Recommendations

    First identify the three to five task types your team cares about most, then collect session signals around them. Metrics should be reviewed periodically and cross-checked against manual sampling. The goal of evaluation is to provide evidence for model selection and system improvement, so every conclusion should come with its data source, sample size, and confidence interval.