How to Scientifically Evaluate LLMs for AI Agent Scenarios: Accuracy, Latency, Token Usage, and Cost
This article explains why AI agent scenarios require a dedicated evaluation perspective that goes beyond traditional accuracy rankings. It covers the core capabilities that agent-oriented models optimize (tool calling, instruction following, long-running and scheduled tasks, long-horizon execution, and multi-agent collaboration) and proposes a four-dimension evaluation framework built on accuracy, response latency, token usage, and overall cost, illustrated with a Python evaluation loop. It also shows how to interpret version-to-version results as capability trade-offs rather than across-the-board gains, and lists best practices for building a repeatable, production-ready evaluation process.
$$# Why Agent Scenarios Need a Dedicated Evaluation Perspective
Over the past two years, the way large language models are applied has shifted from “one question, one answer” toward “multi-step execution.” When a model has to break down a task on its own, call external tools, and keep making progress across multiple turns until the goal is met, it has entered the agent scenario. This kind of scenario places noticeably different demands on a model than traditional “text generation” or “single-turn Q&A”: the model must not only answer correctly, but also call tools accurately, stay on track, and hold up over long runs.
Because of this, evaluating a model meant for agent scenarios cannot stop at traditional accuracy rankings. A complete evaluation system should answer at least four questions: How often does it get things right (accuracy)? How fast does it respond (response latency)? How many resources does it consume (token usage)? And all told, how much does it cost (cost)? Only by considering these dimensions together can you form a reliable judgment of a model’s true capabilities.
Core Optimization Directions for Agent-Oriented Models
Unlike traditional general-purpose models, which follow a “general pre-training first, scenario adaptation later” path, models built for agent scenarios often reinforce several key capabilities directly during training. Understanding these directions is the prerequisite for designing an evaluation plan.
- Tool-calling capability: Whether a model can correctly grasp when to call a tool, which tool to call, and with what parameters is the first gate to making an agent work in practice. It determines whether the model stays at “chatting” or can actually “get things done.”
- Instruction-following capability: Faced with complex instructions containing multiple constraints, priorities, or branching conditions, the model must accurately break them down and execute step by step, rather than dropping or misinterpreting any single requirement.
- Scheduled and persistent tasks: In tasks that need to stay suspended for a long time, trigger periodically, or maintain state, the model must preserve contextual consistency and avoid “losing its memory midway.”
- Long-horizon execution: When a task requires a dozen or more steps chained together, the model must keep every step correct and prevent errors from accumulating and amplifying along the chain.
- Multi-agent collaboration: When several models divide up the work, deciding how to allocate responsibilities, pass along intermediate results, and aggregate the final output is equally a test of the system’s overall orchestration design.
It is worth emphasizing that these capabilities are largely a matter of “systems engineering” outcomes rather than simple improvements to a single point metric.
Designing the Evaluation Framework: Four Core Dimensions
A rigorous evaluation framework should turn subjective impressions into quantifiable, reproducible metrics. The following four dimensions form a minimal, usable evaluation skeleton.
Accuracy: The Baseline of Capability
Accuracy measures the correctness of a model’s output, typically through statistics over a large, diverse test set. Two things matter when evaluating: first, the coverage of the test set—whether it spans different capability types such as tool calling, instruction following, and reasoning/computation; second, whether the scoring is objective—prefer automatically judgeable reference answers or structured validation to reduce the bias introduced by manual scoring.
Response Latency: A Hard Constraint on Experience
In interactive scenarios, the average time of a single call directly affects the user experience. Latency depends not only on the model itself but also on how inference services are deployed, how concurrency is scheduled, and the network path involved. Latency should be measured under controlled, reproducible conditions, with the test environment noted, otherwise the data lacks comparability.
Token Usage: The Intermediate Measure of Cost
Token usage is the key intermediate metric for “how much computation a model expends to do the same thing.” The higher the token consumption, the higher not only the cost but often the response time as well. Folding average token consumption per call into the evaluation helps identify models that “trade resources for results,” leading to a more rational selection.
Overall Cost: Where Decisions Land
What ultimately lands on the business is cost. Only by converting accuracy, latency, and token usage into “cost per 1,000 calls” can you compare the economics of different models side by side. It is worth noting that a low-cost model whose accuracy falls short may actually push up overall cost through retries and manual fallback.
The following illustrative code shows how to collect all four of these metrics within a single evaluation loop:
import timedef run_eval(model_client, dataset, price_per_token):total_correct = 0total_latency = 0.0total_tokens = 0total_cost = 0.0for case in dataset:t0 = time.perf_counter()resp = model_client.chat(case["prompt"])latency = time.perf_counter() - t0tokens = resp["usage"]["total_tokens"]cost = tokens * price_per_token # Unit price is configured per each provider's billing rulestotal_latency += latencytotal_tokens += tokenstotal_cost += costif evaluate(resp["text"], case["expected"]):total_correct += 1n = len(dataset)return {"accuracy": total_correct / n,"avg_latency": total_latency / n,"avg_tokens": total_tokens / n,"cost_per_1k_calls": total_cost / n * 1000,}
Evaluating Version Iterations: Trade-offs Rather Than Linear Gains
When a model releases a new version, a common misconception is to “assume every metric will improve across the board.” In reality, what usually happens is a set of capability trade-offs. Take a sample set of evaluation data as an example (the data is used only to illustrate the analysis method): a new-generation version’s overall accuracy edged up from 71.0% to 71.5%, moving its ranking slightly forward; language and instruction following rose from 67.6% to 69.3%; reasoning and math improved from 73.6% to 74.8%; while the agent and tool-calling dimension only ticked up from 64.2% to 64.5%, essentially flat.
This set of data offers two takeaways:
First, if the new version’s core optimization direction is complex instruction decomposition, then a more noticeable gain in “language and instruction following” (+1.7%) is exactly what you would expect, while limited change in dimensions unrelated to that direction is equally reasonable.
Second, a dimension staying “flat” does not mean the capability has not progressed. The agent dimension in a general Chinese evaluation framework often does not fully align with the “complex tool-chain calling and multi-step execution” emphasized in real workflows. In other words, the focus of the evaluation framework determines which facet of capability it can reflect. Talking about “how much a given dimension improved” without reference to the specific evaluation criteria can easily be misleading.
This is precisely the biggest trap in evaluating “specialized-optimization” models: the real gains a model achieves in its headline scenarios (such as multi-step tool calling, long-horizon execution, and multi-agent collaboration) may not be fully captured by a general-purpose evaluation framework.
Best Practices for Putting Evaluation into Practice
To do evaluation well, beyond metric design you also need to get the engineering and process right:
- Fix the evaluation environment and criteria: Specify the test set version, scoring method, concurrency, and network conditions so that results across multiple runs are comparable and reproducible.
- Design test sets by dimension: Build dedicated datasets separately for tool calling, instruction following, long-horizon execution, multi-agent collaboration, and so on, rather than using one general set that masks differences in specialized capabilities.
- Account for real workflows: Beyond offline evaluation, add end-to-end tests that closely mirror real business processes, and observe whether errors accumulate over long chains.
- Close the loop on cost accounting: Convert token consumption into actual cost, and factor in the hidden costs of retries and fallback.
- Establish continuous monitoring: After a model goes live, use logging services and monitoring/alerting capabilities to continuously track accuracy, latency, and token usage, so that performance drift can be caught promptly.
On the engineering side, the call chain of the evaluation service can be deployed on a container platform to gain elastic scaling and concurrency, evaluation results can be archived to object storage for traceability, and quantitative metrics can be tracked through a monitoring and alerting system. These general-purpose infrastructure capabilities—such as the container, storage, logging, and monitoring offerings provided by Baidu AI Cloud—can turn an evaluation process from a “one-off script” into a “sustainably operated quality system.”
Conclusion
Evaluating large language models for agent scenarios is, at its core, an engineering practice of “moving from impression to quantification.” The four dimensions—accuracy, response latency, token usage, and overall cost—are all indispensable; when evaluating version iterations, you must understand the optimization direction and make judgments about capability trade-offs; and for specialized-optimization models, you must be especially wary of general-purpose evaluation frameworks “failing to see their strengths.” Only by designing an evaluation system that is comprehensive enough and clear enough in its criteria can you make rational, reliable choices in model selection and iteration decisions.
