Agent Evaluation for Real-World Tasks: Task Boundaries, Multi-Dimensional Metrics, and Production Practices

    Traditional question-answering evaluation focuses on the quality of a single response. Agents, however, must act toward a goal over time: understanding requirements, planning, calling tools, checking results, and adjusting execution based on user feedback. This article outlines an evaluation framework for real-world agent tasks, covering task-boundary definition, signal extraction from real interactions, multi-dimensional metrics, randomized experiments, interpretation of long-horizon tasks, a five-layer production architecture, and an enterprise adoption path. The goal is to help teams measure task completion quality with verifiable outcomes, explain failures with process data, and distinguish genuine improvements from random fluctuation through well-designed experiments.

    1. From Answer Quality to Task-Process Evaluation

    The main difference between an agent and a traditional question-answering system is that an agent must act continuously around a goal: understand the requirement, make a plan, call tools, inspect results, and adjust its execution path based on user feedback. A single response that looks complete is therefore not enough to show whether the agent can reliably complete work.

    For example, in a code-modification task, an agent may provide a clear explanation without actually making the change. It may also successfully run a local check while breaking other functionality. Conversely, a failed tool call in the middle of a task does not necessarily mean the whole task failed: if the agent can identify the cause, adjust its approach, and ultimately deliver an acceptable result, that recovery capability also has value.

    Evaluation for real-world tasks needs to answer three questions at the same time:

    1. Is the result correct? Does the deliverable meet the user’s requirements, and does it pass the necessary validation?
    2. Is the process reliable? Are tool calls reasonable? Can the agent recover from failures? Are there dangerous operations?
    3. Is collaboration effective? Can the agent understand user corrections, and does it require repeated explanation and human takeover?

    The unit of evaluation should therefore expand from “a single response” to “a task process with clear boundaries.” Human comparison can still serve as a supplement, but it cannot replace observation of the complete task.

    2. Define Task Boundaries Before Defining Evaluation Metrics

    A real conversation does not naturally equal one task. A user may raise multiple goals in the same session, or handle the same project across multiple sessions. Calculating success rates directly by session count can easily introduce statistical bias.

    2.1 Create Task-Level Records

    Each task should contain at least the following information:

    • The initial goal and constraints explicitly stated by the user;
    • Input materials, environment state, and available tools;
    • Agent execution steps and key intermediate results;
    • User supplements, corrections, and confirmations;
    • The final deliverable and validation results;
    • The termination reason, such as completion, cancellation, timeout, or human takeover.

    Task boundaries should follow pre-defined rules. When the goal materially changes, the task can be split. When the user provides additional explanation around the same goal, it should usually remain within the same task.

    2.2 Distinguish Completion, Failure, and Unknown Outcome

    No complaint does not mean the task was completed, and a user leaving does not necessarily mean dissatisfaction. At minimum, three states should be defined: “completed,” “failed,” and “unknown outcome.”

    For samples with unknown outcomes, the proportion should be reported separately, and the impact of different handling methods on conclusions should be analyzed. Simply deleting this data may make a solution that is more likely to cause user churn appear better.

    3. Extract Effective Signals from Real Interactions

    Real user behavior provides rich evaluation signals, but no single signal should be mechanically interpreted as success or failure.

    3.1 Explicit Confirmation and Praise

    Statements such as “this has been verified” or “the result meets the requirements” are relatively strong positive signals, but their scope must be identified. “This explanation is good” only indicates that the explanation was recognized; it does not prove that subsequent operations were correct.

    Signal extraction should use context to determine whether the user is evaluating the presentation, a local step, or the final deliverable. For tasks that can be objectively verified, artifact checks should take priority over scoring based only on emotional expression.

    3.2 Complaints and Correction Instructions

    When a user asks the agent to “run it again,” “do not modify this part,” or says “you did not follow the original constraint,” it may indicate that the agent deviated from the task, or it may simply mean that the requirement has changed.

    Three cases should therefore be distinguished:

    • The agent missed an already explicit requirement;
    • The user supplied information that was not previously available;
    • The user actively changed the goal or preference.

    Only the first case is direct evidence of insufficient constraint following. Treating all subsequent modifications as errors would penalize normal collaboration.

    3.3 Command Execution and Tool Feedback

    Command execution errors are important process signals, but the source may be improper operation, insufficient permissions, missing dependencies, or environment anomalies. Some validation tasks also intentionally trigger failures to check whether the system handles exceptions correctly.

    The evaluation should therefore record the error category, whether it can be attributed to the agent, the recovery process, and its final impact. Counting error occurrences alone cannot accurately measure execution capability.

    4. Describe Capabilities with Multi-Dimensional Metrics, Not Just a Total Score

    A composite score is convenient for quick comparison, but it can easily hide critical weaknesses. Enterprise solution selection needs to know which tasks an agent is reliable on and what consequences failures may cause.

    4.1 Task Completion Quality

    Task completion quality should be built around verifiable acceptance conditions. For code tasks, behavior can be checked against requirements. For documentation tasks, required content, factual consistency, and constraint adherence can be checked. For analysis tasks, the reasoning basis and conclusions should correspond.

    Parts that cannot be automatically verified can be manually reviewed using unified criteria, with sampling checks on consistency between reviewers.

    4.2 Command Execution Reliability

    This dimension should focus on whether operations comply with environment constraints, whether failure causes are accurately identified, and whether final execution results can be reproduced.

    Teams can report first-attempt success rate, recovery success rate for recoverable errors, and the proportion of repeated ineffective execution separately. Different metrics correspond to different problems and should not be combined into an unexplainable single number.

    4.3 Direction Adjustment and Correction Capability

    Effective correction is not simply restating the user’s requirement; it must produce the right change in subsequent behavior. Evaluation should check whether the agent:

    • Stops executing a plan that has already been rejected;
    • Retains context and constraints that are still valid;
    • Modifies the relevant plan and deliverables;
    • Avoids repeating the same type of error.

    For newly added constraints, evaluation should also observe whether the agent can identify conflicts with the original plan and ask for necessary clarification.

    4.4 Cost, Latency, and Risk

    High-quality completion in real business contexts is also constrained by resource cost and risk boundaries. Evaluation should cover completion latency, number of user interventions, tool-call overhead, and human takeover.

    For irreversible operations, authorization confirmation and risk control should be independent check items. A high completion rate cannot offset the risk caused by unauthorized operations.

    5. Reduce Comparison Bias Through Randomized Experiments

    Different models may receive tasks of different difficulty, user groups, and environment conditions. Directly comparing observed success rates can easily mistake differences in task distribution for differences in model capability.

    The value of randomized experiments is that they allow different solutions to face comparable task conditions on average, so the impact of a solution change can be estimated more reliably.

    5.1 Choose an Appropriate Randomization Unit

    The randomization unit can be the user, session, or task. The choice should consider context continuity and interference between groups.

    For long-horizon tasks, it is usually necessary to keep the same task on a stable solution to avoid state inconsistency caused by switching mid-task. If multiple tasks from the same user are clearly correlated, user-level correlation should also be considered in statistical analysis.

    5.2 Keep Comparison Conditions Consistent

    To compare the impact of the model itself, tool permissions, execution environment, context configuration, and resource budget should be fixed as much as possible. If these conditions change at the same time, the result is the effect of the overall solution and cannot be fully attributed to the model.

    Task type, difficulty, and environment state should also be recorded to check whether randomized groups are balanced.

    5.3 Clearly Define Net Improvement

    “Net improvement rate” does not have a universal meaning independent of metric definitions. One interpretable approach is to first define a task utility score and then calculate the average difference between a solution and the reference group:

    Net improvement = average task utility of the solution group − average task utility of the reference group

    Task utility can combine completion outcome, interaction burden, and risk, but the weights must be determined in advance and disclosed. If a success-rate difference is used, it should be reported in percentage points and not mixed with relative improvement percentages.

    Randomization provides a basis for causal identification, but it does not automatically solve missing feedback, non-compliant execution, or sample contamination. Experimental conclusions still need to be interpreted together with data quality.

    6. Statistical and Interpretation Boundaries for Long-Horizon Tasks

    Long tasks usually contain multiple interdependent steps. An early error may affect all subsequent operations, while effective correction later may eliminate the early problem.

    Evaluation should therefore preserve two levels at the same time:

    • Task-level metrics: used to judge the overall result and as the primary basis for comparison;
    • Step-level metrics: used to locate failures and explain performance differences.

    Multiple messages within the same task cannot be treated as independent samples. When reporting effective sample size, the statistical unit should be explicit, and analysis methods should handle repeated observations or group correlation.

    In addition, rankings should include sample size, statistical time window, and uncertainty notes. A slight difference in point estimates between two solutions does not necessarily mean there is a stable practical difference. Repeatedly checking real-time results and stopping the experiment when results look favorable can also amplify random fluctuation, so observation and stopping rules should be agreed in advance.

    7. Production Architecture: From Logs to Trustworthy Conclusions

    A maintainable evaluation system can be divided into the following five layers.

    7.1 Data Collection Layer

    Associate tasks, interactions, tool execution, and result records in a unified way, and preserve the necessary temporal order. Collection should follow the principle of minimal necessity and should not store sensitive content unrelated to evaluation by default.

    7.2 Cleaning and De-Identification Layer

    Remove duplicate records, mark missing logs, and handle accounts, keys, and personal information. For samples that cannot be safely de-identified, restrict their use or retain only reviewed structured metrics.

    7.3 Signal Extraction Layer

    Identify confirmations, corrections, complaints, and execution anomalies from interactions, while storing evidence location and confidence level. Automated extraction results need periodic manual spot checks, especially for misjudgments involving implicit confirmation, sarcasm, and complex context.

    7.4 Statistical Analysis Layer

    Generate overall results and grouped results according to pre-defined definitions, and show uncertainty. Analysis should avoid showing only better-performing subsets or arbitrarily changing metric weights to fit results.

    7.5 Issue Attribution Layer

    Classify failures into categories such as requirement understanding, planning, tool use, environment adaptation, missing validation, and risk control, and then verify the causes through reproduction experiments.

    For example, a tool-call failure may come from permission configuration rather than insufficient reasoning capability. Only after attribution is complete can optimization measures become truly effective.

    8. Implementation Path for Enterprise Selection and Continuous Iteration

    Enterprises can first build an internal sample set covering high-frequency tasks, complex tasks, and high-risk tasks, define acceptance criteria, and then conduct controlled offline evaluation. The offline stage should focus on identifying basic capability defects and safety issues, and should not be directly extended to all business operations.

    After initial screening, small-traffic, reversible online randomized experiments can be conducted. Before the experiment, primary metrics, observation period, abnormal stopping conditions, and human takeover mechanisms should be defined to avoid changing rules arbitrarily during operation.

    For tasks involving data deletion, permission changes, or external submissions, validation should first be performed in an isolated environment, with an approval process matched to the risk level. After launch, task distribution changes should also be monitored continuously, because changes in user needs and runtime environments may invalidate historical evaluation conclusions.

    Ultimately, the value of real-task evaluation is not to produce a permanently valid ranking, but to establish continuous calibration capability: measure completion quality with verifiable results, explain failure causes with process data, and use reasonable experiments to distinguish real improvement from random fluctuation. Such an evaluation system can provide a trustworthy basis for agent selection, deployment, and iteration.