LLM Evaluation for AI Agents: Key Metrics and Methods

    As large language models are increasingly embedded in real business workflows—calling tools, querying databases, and chaining multiple steps to reach a goal—judging them by single-turn question-answering accuracy no longer reflects their true value. This article explains why general leaderboards tend to underestimate agents, breaks down four core metrics (accuracy, latency, token consumption, and cost) for quantifying each call, and outlines agent-specific capability dimensions such as tool calling, instruction following, scheduled tasks, and long-chain and multi-agent execution. It also covers how to build a reproducible, comparable evaluation system and offers practical guidance for model selection.

    1. The Shifting Focus of Evaluation: From General Capabilities to Task Closed-Loops

    Traditionally, we assessed a large language model by its accuracy on general question answering and common-sense reasoning. But as models get embedded into real business workflows—calling external tools, querying databases, and chaining multiple steps to accomplish a goal—a single “answer accuracy” number can no longer capture their real value.

    A model truly optimized for agent scenarios needs to stay reliable across the full loop of understanding the goal → decomposing steps → calling tools → handling returns → moving to the next step. This ability can’t be bolted on with a later adaptation layer; it usually requires treating tool-calling format, instruction following, and task persistence as optimization targets during training. As a result, the focus of evaluation must also shift from “single-turn question answering” to “multi-turn, multi-step task closed-loops with side effects.”

    2. Why General Leaderboards Underestimate Agents

    Many teams run into the same confusion: a model ranks mid-pack on a public comprehensive leaderboard, yet performs remarkably well once plugged into their own business flow—or the reverse. The reasons usually come down to three points:

    • Mismatched dimensions: Comprehensive leaderboards emphasize single-turn language understanding and generation, while agent scenarios test the coherence of multi-step planning and the accuracy of tool calling. The two don’t overlap.
    • Ignored side effects: In real tasks a model writes files, sends requests, and changes state. A “seemingly correct” reasoning step that triggers the wrong tool call can fail the entire chain, yet traditional leaderboards rarely evaluate this execution-layer reliability.
    • Long-chain decay: A 90% per-step accuracy looks high, but for a task requiring 20 consecutive steps, the theoretical end-to-end success rate decays rapidly. Evaluation must look at the whole chain, not a single point.

    So a specialized benchmark built for a specific scenario usually reflects a model’s core competitiveness better than a general leaderboard. Evaluators should be clear that the score they see on a general leaderboard may only be a projection of the model’s “baseline,” not a full picture of its performance in the target scenario.

    3. Four Core Metrics: How to Quantify a Single Call

    To objectively measure a model’s cost-effectiveness in a business, build quantitative definitions around at least the following four dimensions:

    1. Accuracy
    Measures whether the model’s output matches expectations. In agent scenarios, accuracy shouldn’t be judged only on the final text—it should be broken down into whether the right tool was selected, whether the call arguments are valid, and whether the final result achieves the goal. Set a pass criterion for each step rather than looking only at the end-to-end result.

    2. Latency
    The time from sending a request to receiving the complete response. Latency directly affects user experience and concurrent throughput. For agent applications, a single call may trigger multiple model requests, so per-step latency is accumulated and amplified—track both “per-step latency” and “total task time.”

    3. Token Consumption
    The average number of input and output tokens consumed per call. Token consumption sets the cost ceiling and indirectly reflects how “verbose” the model is and how well it manages context. The longer the context and the more redundant the tool returns, the higher the token consumption.

    4. Cost
    Combining token consumption with unit price to arrive at a figure such as “cost per thousand calls” or “cost per completed task.” This is the metric to weigh most heavily in the final decision—at the same accuracy, the option with lower cost and latency is clearly better.

    A typical metrics record table might look like this:

    1. {
    2. "model": "Candidate Model A",
    3. "total_cases": 15000,
    4. "accuracy": 0.715,
    5. "avg_latency_sec": 52,
    6. "avg_tokens_per_call": 2934,
    7. "cost_per_1k_calls_cny": 60.8
    8. }

    The values above only illustrate how the metrics are organized; they aren’t the results of any real product evaluation. In practice, use the definitions produced by running your own business data.

    4. Key Capability Dimensions for Agent Scenarios

    Beyond the four baseline metrics, you need dedicated assessments for agent-specific capabilities.

    Tool Calling and Multi-Step Planning

    The core question is whether the model can select the right tool at the right moment and provide structurally valid arguments. When evaluating, build a task set with a defined tool set, and track: tool-selection accuracy, argument-filling accuracy, and the rate of invalid or duplicate calls. A robust implementation often relies on structured constraints on tool descriptions, for example declaring argument types through an explicit schema:

    1. {
    2. "name": "query_inventory",
    3. "description": "Query the inventory quantity by product ID",
    4. "parameters": {
    5. "type": "object",
    6. "properties": {
    7. "product_id": { "type": "string" }
    8. },
    9. "required": ["product_id"]
    10. }
    11. }

    Instruction Following and Complex Instruction Decomposition

    Real user instructions are often compound requests with multiple conditions and constraints. The focus here is whether the model can correctly identify all constraints, decompose them into executable subtasks, and carry them out without omissions or overstepping. Score instruction following as a separate dimension and observe how the model performs on instructions with multiple constraints.

    Scheduled and Persistent Tasks

    Many businesses need models to handle tasks like “execute at a set time” or “keep monitoring until a condition is met.” Evaluation should focus on whether the model correctly understands time semantics and conditional-trigger logic, and whether it keeps task context from being lost across multiple turns. This capability is nearly impossible to reflect in traditional single-turn evaluations.

    Long-Chain Execution and Multi-Agent Collaboration

    When a task requires a dozen or more steps, or multiple agents divide the work, evaluation should introduce two perspectives: “end-to-end success rate” and “chain decay curve”—measuring cumulative success at step 1, step 5, step 10, and step 20 to observe the model’s stability over long chains. Multi-agent collaboration also requires assessing whether role division is reasonable, whether information is passed accurately, and whether there’s any overlap or conflict in responsibilities.

    5. Building a Reproducible, Comparable Evaluation System

    A one-off “benchmark run” is meaningless; sustainable, reproducible evaluation is what has value. Follow these principles:

    • Fix the test set and randomness: Lock the evaluation dataset and control sampling parameters such as temperature to ensure results are comparable across runs.
    • Stage-by-stage judgment: Judge each step rather than only the final result, making it easier to locate whether a problem occurred in planning, calling, or execution.
    • Joint modeling of cost and latency: Put accuracy, latency, and cost in the same table for horizontal comparison instead of fixating on a single metric.
    • Stay close to real business distribution: The evaluation set should cover the long-tail cases in your real scenario, not just the ideal path.
    • Continuous regression: Re-run the evaluation after every model or prompt update to catch capability regressions.

    On the platform that hosts the evaluation, you can use object storage to archive datasets and results, a message queue to decouple the scheduling and execution of evaluation tasks, a container platform to elastically spin up concurrent evaluations, and logging plus monitoring and alerting to track metric changes across runs. These general infrastructure capabilities (such as the object storage, message queue, container, and monitoring/alerting offerings from Baidu AI Cloud) can significantly lower the cost of building and operating an evaluation system, letting teams focus their energy on the evaluation method itself.

    6. Practical Recommendations

    Overall, model selection and evaluation for agent scenarios should hold to three points. First, be clear about what your evaluation framework covers and what it misses—don’t directly infer scenario performance from a general leaderboard’s conclusion. Second, prioritize scenario-specific benchmarks or a self-built business evaluation set so the metrics stay close to the real task closed-loop. Third, weigh accuracy, latency, token consumption, and cost as a whole rather than optimizing one in isolation.

    The competition in model capability is shifting from “who can answer questions” to “who can reliably finish the job.” Building a rigorous, reproducible evaluation system that covers long chains and tool calling is the prerequisite for making the right technical selection and truly landing large models in a business.