How to Evaluate AI Agent Models: Key Metrics Beyond Accuracy
A practical guide to evaluating AI agent models across task completion, latency, instruction following, reasoning, tool use, reliability, and cost—rather than relying on traditional accuracy scores alone. Internal benchmark figures are discussed with appropriate limitations because the test design and statistical methods were not disclosed.
1. How Agent Tasks Differ from Traditional Question Answering
The focus of large language model applications is shifting from answering questions to completing tasks. A question-answering task is usually evaluated by whether a response is accurate and complete. An agent task, by contrast, requires the model to understand the objective, break it into steps, select tools, interpret returned results, revise its plan, and ultimately deliver a verifiable outcome.
The industry sometimes refers to these scenarios as “lobster” scenarios. The term has no standardized definition and generally describes AI agent systems that support tool calling, long-running execution, scheduled tasks, and multi-step collaboration. In this article, “lobster” refers only to this category of tasks, not to any specific product.
There are two common approaches to building these systems:
General-purpose models with post-training adaptation.
Prompts, workflow orchestration, function-calling protocols, or limited fine-tuning are used to give a model basic agent capabilities. This approach is fast to deploy and offers good compatibility, but the model’s underlying behavior was not designed specifically for agent tasks.Models trained with agent capabilities as a core objective.
Training focuses on complex instruction understanding, tool selection and parameter generation, multi-turn state management, failure recovery, persistent task execution, multi-agent collaboration, and stability across long execution chains.
As a result, these models cannot be evaluated using traditional question-answering accuracy alone. Real-world assessment should also consider task completion rate, execution efficiency, and resource consumption.
2. How to Interpret Internal Benchmark Results
The figures in this section come from an internal Chinese comprehensive evaluation containing approximately 15,000 questions. The test composition, specific model versions, statistical methodology—such as whether the figures represent means or medians and whether retries were included—and measurement period were not disclosed. The numbers should therefore be used only as directional references and should not be compared directly with results from other vendors or evaluation frameworks.
In this evaluation, a commercial model enhanced for agent tasks achieved an average accuracy of approximately 71.5%, compared with about 71.0% for the previous general-purpose version—a difference of roughly 0.5 percentage points. Without the sample size and confidence intervals, this gap does not establish a statistically meaningful improvement. Describing the results as broadly unchanged is more appropriate.
The same evaluation reported an average latency of about 52 seconds per call, average consumption of approximately 2,934 tokens, and an average cost of about RMB 60.8 per 1,000 calls. All three figures depend on the measurement methodology. For example, latency may include queueing time, while cost may or may not include cache-related billing. Anyone using these figures for budgeting should first verify the definitions and accounting scope.
1. Accuracy Measures Baseline Capability
An accuracy score of 71.5% indicates stable baseline capability in Chinese language understanding, knowledge-based question answering, reasoning, and instruction execution. It does not describe how the model performs on dynamic tasks.
Traditional test sets mainly evaluate static responses. Agent capability is more visible in areas such as:
- Whether the model can determine which tool to use next;
- Whether it can adjust its plan based on tool results;
- Whether it can maintain the original objective after multiple calls;
- Whether it can continue after an intermediate failure;
- Whether it can reduce unnecessary calls and redundant reasoning.
If a benchmark does not include these elements, the benefits of agent-specific training may not be reflected in its accuracy score.
2. Evaluate Latency Using Percentiles
The internal evaluation reported changes in response speed between the new and previous versions, but the specific before-and-after figures were not disclosed. The scale of the improvement therefore cannot be quantified here.
Speed is important for agent systems. If a task requires ten model calls and each call is slow, the waiting time accumulates across the entire chain. This is particularly noticeable in automated office workflows, code generation, data processing, and real-time customer service.
Latency improvements may come from more efficient reasoning strategies, output-length control, optimized context processing, or changes to the execution path for tool calls. However, speed should not be evaluated independently of accuracy. If a model finishes too early to reduce latency and consequently skips critical steps, the overall task completion rate may decline.
When measuring speed, consider recording the following metrics together:
- Time to first token;
- Full-response latency at P50 and P95;
- Total execution time per task;
- Task success rate;
- Number of retries after failure;
- Cost per completed task.
3. Evaluate Instruction Following Against Task Constraints
In the internal evaluation, the language understanding and instruction-following score increased from approximately 67.6% to 69.3%. This direction is consistent with the training goals of an agent model, but the lack of sample size and confidence intervals means the result should be treated only as a reference.
Complex tasks often contain multiple constraints. A user may ask the model to extract information first, classify it next, and then produce the result in a specified format. The model may also be required not to omit exceptional cases and to identify uncertain information.
In such scenarios, the model must distinguish between:
- The final objective;
- Conditions that must be satisfied;
- Steps that have to be performed in a specific order;
- Information that requires tool-based confirmation;
- Information that must not be guessed.
In production, structured prompts, clearly defined output formats, and result-validation mechanisms are still recommended to reduce the risk of the model drifting away from the intended objective.
4. Use Programmatic Checks to Backstop Reasoning
The reasoning and mathematical calculation score increased from approximately 73.6% to 74.8% in the internal evaluation. This is also an internal benchmark result.
For agents, reasoning is not limited to solving mathematics problems. It also affects task planning, conditional decisions, and exception handling.
For example, in a data-processing task, the model may first need to determine whether the input is complete and then select an appropriate cleaning strategy. If the number of fields does not match expectations, it must decide whether to terminate the task, request additional information, or use a fallback approach. These are all forms of conditional reasoning.
However, calculations produced by a model should still be validated programmatically. For amounts, statistical values, time ranges, and batch data, a practical division of responsibility is: the model handles interpretation, while software handles calculation.
3. Why Traditional Agent Metrics May Change Very Little
In the internal evaluation, the score for agents and tool calling increased from approximately 64.2% to 64.5%, a difference of only 0.3 percentage points. The result was essentially unchanged. Because the task definition, scoring rules, sample size, and confidence intervals for this category were not disclosed, it is not possible to determine whether the difference is meaningful. Nor should the result be interpreted as proof that tool capabilities did not improve.
At least three factors may explain the limited change.
First, different evaluation frameworks define agent…
