How to Evaluate and Select Large Language Models Optimized for AI Agents

    As large language models move from generating answers to completing end-to-end tasks, model selection for AI agents requires more than a single accuracy score. This article explains how to evaluate agent-oriented models across instruction following, tool use, long-running workflows, latency, token consumption, cost, task completion, exception recovery, and system-level reliability. It also presents a layered architecture and a practical framework for designing more meaningful evaluations.

    In recent years, competition among large language models has shifted from “can the model generate an answer?” to “can the model complete a task?” In scenarios such as intelligent office automation, operations automation, data analysis, content production, and business workflow orchestration, a model is often expected to understand a goal, break it into steps, call tools, handle exceptions, maintain intermediate state, and deliver a verifiable result.

    Against this background, models specifically optimized for AI agent workflows are receiving increasing attention. Their differences from general-purpose models may not be obvious in traditional knowledge-based question answering or single-turn writing. Instead, their strengths are more likely to appear in complex instruction execution, tool-chain coordination, long-running workflow stability, and continuous task processing.

    As a result, these models should not be evaluated using a single overall score. Accuracy, latency, token consumption, tool-call success rate, and task completion rate must be considered together.

    1. Why Models Need to Be Optimized for AI Agent Workflows

    A conventional question-answering model typically works as follows: the user submits a question, and the model generates a response. The interaction usually ends within one or a few turns. AI agent tasks have substantially different characteristics.

    First, agent tasks are often multi-step. For example, “organize sales data and generate a weekly report” may require the system to read a file, identify fields, calculate metrics, detect anomalies, create charts, write conclusions, and save the output to a specified location. The model must maintain consistent context across these steps. A deviation at any stage can affect the final result.

    Second, these tasks depend on external tools. A model does not inherently have real-time access to databases, search systems, file systems, or business APIs. It must use structured tool calls to access those capabilities. The model must therefore understand the user’s intent, select the correct tool, provide valid parameters, parse the returned data, and determine what to do next.

    Third, tasks may run for a long time. Scheduled execution, periodic reporting, continuous monitoring, and multi-round approval workflows are not suitable for a simple question-and-answer pattern. The model must understand task state, distinguish completed, pending, and failed items, and avoid duplicate execution or missed steps.

    Fourth, the cost of errors is higher. A wording mistake in a conventional response may only affect readability. In an automated workflow, incorrect tool parameters, faulty permission decisions, or an incomplete execution plan can lead to data corruption, duplicate operations, or business risk.

    Therefore, optimizing a model for agent scenarios is not primarily about making it “sound more human.” The goal is to make it more reliable, controllable, and efficient when handling complex tasks.

    2. How to Interpret Comprehensive Evaluation Results

    One Chinese comprehensive evaluation covered approximately 15,000 test questions. The results showed that a model optimized for agent tasks achieved an overall accuracy of about 71.5%, with an average single-call latency of approximately 52 seconds, average consumption of around 2,934 tokens, and an average cost of about CNY 60.8 per 1,000 calls.

    These figures can serve as references, but they should not be interpreted as the model’s performance in every real-world business scenario. Comprehensive evaluations usually combine knowledge understanding, language expression, mathematical reasoning, instruction following, and basic tool calling. The actual advantages of an agent-oriented model may instead be concentrated in long-horizon tasks and complex tool coordination.

    Compared with the previous generation, the new version achieved a similar overall accuracy but responded significantly faster. Accuracy increased from approximately 71.0% to 71.5%, and its ranking also improved. Although the increase was limited, it suggests that the specialized optimization did not significantly sacrifice baseline capabilities.

    In language understanding and instruction following, performance increased from approximately 67.6% to 69.3%, a larger improvement than the increase in overall accuracy. This generally indicates more stable handling of complex constraints, formatting requirements, task boundaries, and multi-condition instructions. For enterprise workflow automation, this type of improvement may be more valuable than a small change in general knowledge question answering.

    Performance in reasoning and mathematical calculation improved from approximately 73.6% to 74.8%, indicating progress in structured analysis, conditional deduction, and basic computation. However, performance on mathematical questions does not fully represent reasoning ability in real business environments. Production systems must also determine whether the model uses the correct data, recognizes uncertainty, and validates its conclusions.

    The basic agent and tool-calling score increased only slightly, from approximately 64.2% to 64.5%. This limited change does not necessarily mean that the specialized optimization was ineffective. It may reflect a gap between the evaluation framework and real workflows. If a test only measures a single tool call or simple parameter generation, it is unlikely to capture the model’s ability to perform multi-step planning, recover from failures, maintain state, and execute continuously.

    3. Faster Responses Usually Reflect Changes in Reasoning Strategy

    In agent scenarios, speed is not merely a user-experience metric. It directly affects system throughput, concurrency, and operational cost.

    A complex task may require multiple model calls. If each call has high latency, the entire workflow becomes slow. For example, even if each step in a six-step task takes only several dozen seconds, the total waiting time can reach several minutes. Improvements in model latency are therefore magnified across multi-step workflows.

    Speed improvements typically come from several areas:

    1. Reducing unnecessary reasoning

      The model can allocate reasoning resources according to task difficulty instead of expanding every task in detail. Clear format conversions and simple queries can be completed quickly, while high-risk tasks can retain a more thorough analysis process.

    2. Shortening the context

      Long-running agents accumulate substantial historical information. Sending the complete context with every call increases latency and wastes tokens. In practice, systems can use summaries, state compression, and layered memory so that each call retains only the information required for the current step.

    3. Optimizing tool-call workflows

      Frequent tool calls, duplicate queries, and repeated confirmations can significantly slow down a task. A better approach is to plan the tool chain first, execute operations in parallel when possible, and organize tool results into structured information.

    4. Differentiating task levels

      Not every task requires the same level of reasoning. Enterprises can classify tasks into high-frequency, low-risk tasks and low-frequency, high-risk tasks, then apply different model configurations and review mechanisms to balance quality and efficiency.

    However, faster is not always better. If lower latency results from reduced verification, lower tool-call accuracy, or inadequate exception handling, the overall task completion rate may decline. Latency must therefore be evaluated together with final success rate and error-recovery capability.

    4. Accuracy Should Not Be the Only Core Metric

    Accuracy is important for traditional text tasks. In real workflows, however, organizations need a broader evaluation framework.

    1. Task Completion Rate

    Task completion rate is closer to business value than single-step answer accuracy. Even if one step is imperfect, a task may still be successful if the system detects the problem, calls the tool again, and delivers a usable result.

    2. Tool-Call Success Rate

    Organizations should separately measure whether the model selected the correct tool, supplied complete parameters, used valid parameter types, and correctly parsed the tool’s response. These stages should not be combined into a single metric that records only whether the final task was completed.

    3. Long-Chain Stability

    Errors accumulate as the number of steps increases. Evaluation should include tasks with multiple tools, multiple decision rounds, and exception branches. These tests should examine whether the model maintains the original objective, avoids duplicate execution, and stays on task throughout the workflow.

    4. Exception-Recovery Capability

    Real systems inevitably encounter timeouts, insufficient permissions, empty data, and malformed responses. A capable model should not simply continue executing. It should identify the type of failure and choose an appropriate action, such as retrying, using an alternative approach, requesting human confirmation, or terminating safely.

    5. Token and Cost Efficiency

    Cost should not be measured only by the price of a single call. Organizations should calculate the total number of calls and total token consumption required to complete an entire task. A model with a low per-call price but frequent retries may cost more in practice than a more expensive model with a higher one-pass completion rate.

    Task cost can be estimated as follows:

    1. Cost per task =
    2. Input token cost
    3. + Output token cost
    4. + Additional context cost generated by tool calls
    5. + Retry call cost
    6. + Human review and exception-handling cost

    This approach is closer to actual enterprise operating costs.

    5. Use a Layered Architecture for Production Deployment

    Model capability is only one component of an AI agent system. The model should not be given complete responsibility for every operation. A more reliable system usually adopts a layered design.

    The first layer is task understanding. It identifies the user’s goal, constraints, and permission boundaries.

    The second layer is planning. It breaks a complex goal into executable steps and defines the inputs, outputs, and completion conditions for each step.

    The third layer is tool execution. It handles calls to databases, file systems, message queues, object storage, monitoring systems, and alerting infrastructure. Tool interfaces should enforce strict parameter validation to prevent invalid requests generated by the model.

    The fourth layer is state management. It records task progress, tool results, retry counts, and human intervention. State should not rely only on the model context. It should be stored independently in an auditable system.

    The fifth layer is security control. It performs permission checks, approval workflows, and activity logging for sensitive operations. Actions such as deletion, fund transfers, and bulk updates should include explicit human confirmation points.

    The sixth layer is observability and evaluation. It continuously records latency, token usage, failure reasons, tool-call traces, and final completion status, providing data for model and workflow improvements.

    In this architecture, the model handles understanding and decision-making, while software enforces constraints and performs execution. Separating these responsibilities can significantly reduce operational risk.

    6. How to Design More Valuable Evaluation Sets

    If the goal is to select a model for AI agents, evaluations should not rely only on conventional question-answering tasks. At a minimum, the test set should cover:

    • Multi-condition instruction execution;
    • Structured data extraction;
    • Sequential calls to multiple tools;
    • Retries after tool-call failures;
    • Information filtering across long contexts;
    • Scheduled and continuous tasks;
    • Multi-role collaboration;
    • Insufficient permissions and missing data;
    • Confirmation before high-risk operations;
    • Validation of the final result’s format and factual accuracy.

    Each test case should define clear success criteria. These may include whether the correct tool was called, whether parameters were complete, whether permission constraints were respected, and whether the final output was verifiable. Evaluation should not rely solely on subjective human reading.

    It is also important to distinguish between offline and online evaluation. Offline evaluation is suitable for quickly comparing model capabilities. Online evaluation should examine concurrency, stability, and cost under real traffic. Combining both approaches helps avoid situations in which a model performs well in testing but poorly after deployment.

    7. Overall Assessment: Specialized Models Are Not Universal Replacements

    A model optimized for AI agent scenarios is not necessarily superior to a general-purpose model for every task. Its value is more likely to appear in specific workflows, including complex instruction decomposition, tool-chain scheduling, continuous task execution, and long-running process management.

    Based on the available data, this type of model maintains relatively stable baseline Chinese-language capabilities while showing improvements in instruction understanding, reasoning and calculation, and response speed. Its gains in tool calling are limited in general-purpose evaluations, which indicates that its strengths should be validated in task environments that more closely resemble real business workflows.

    When selecting a model, enterprises can follow these principles:

    1. Define the business task before choosing the model;
    2. Use end-to-end task completion rate instead of a single accuracy score;
    3. Evaluate latency, token usage, retries, and human labor costs together;
    4. Retain software validation and human approval for high-risk operations;
    5. Conduct stress tests with real tool chains and exception scenarios;
    6. Coordinate model capabilities with cloud computing, storage, messaging, logging, and monitoring infrastructure.

    Ultimately, whether a model is worth adopting depends neither on its launch messaging nor on its position on a single leaderboard. The key question is whether it can reliably complete target business tasks, control errors, reduce total operating costs, and provide an observable, auditable, and continuously improvable foundation for production systems.