Evaluating and Selecting Models for AI Agent Workflows

    As large language models move from answering questions to completing tasks, model evaluation must extend beyond accuracy and response quality. This article presents a practical framework for assessing agent-oriented models across task completion, tool use, state management, latency, token consumption, long-chain reliability, and operational cost, while also outlining system design principles and a business-focused model selection method.

    As large language models move from answering questions to completing tasks, the criteria used to evaluate them are changing. Developers have traditionally focused on knowledge accuracy, text generation quality, and single-turn response speed. In AI agent applications, however, a model must also understand complex goals, break tasks into steps, select tools, maintain context, handle failures, and continue working over an extended period.

    As a result, models optimized for agent workflows from the training stage are receiving increasing attention.

    These models are not simply general-purpose models wrapped in an agent framework. They aim to improve several capabilities, including complex instruction understanding, multi-step planning, structured tool calling, scheduled task execution, and strategy adjustment based on feedback after failures. For enterprise users, the key question is not whether a model ranks highly on a particular benchmark, but whether it can complete a specific business workflow reliably and economically.

    1. Agent Models vs. General-Purpose Models

    A general-purpose model typically follows an “input question to generated answer” interaction pattern. The task boundary is usually clear. An agent workflow is closer to a continuous loop of:

    Goal → Plan → Execute → Observe → Revise

    For example, when a user asks the system to “compile sales data every week and generate a business summary,” the model may need to:

    1. Read the task configuration and time range.
    2. Query business data.
    3. Determine whether the data is complete.
    4. Perform aggregation, filtering, and anomaly detection.
    5. Generate a structured summary.
    6. Write the result to a document or send a notification.
    7. Record the execution status and wait for the next scheduled run.

    In this workflow, language generation is only a basic capability. The model must also determine the correct next action. If tool parameters are incorrectly formatted, context is lost, or the model repeats an operation midway through execution, the result may include data errors or unnecessary resource consumption.

    Agent-oriented models therefore tend to emphasize four capabilities:

    • Task decomposition: Converting a natural-language goal into executable steps.
    • Tool use: Selecting the correct tool and generating parameters that satisfy its constraints.
    • State management: Understanding previous actions, current progress, and pending tasks.
    • Continuous execution: Maintaining goal consistency throughout a long workflow.

    2. Metrics for Comprehensive Agent Evaluation

    Testing an agent model should not focus solely on final answer accuracy. A more practical approach is to establish a multidimensional evaluation framework.

    2.1 Accuracy and Task Completion

    Accuracy is useful for measuring answer quality on standardized questions. In agent scenarios, however, it is also necessary to determine whether the task was actually completed.

    For example, an automation workflow may produce text that appears correct, but it should not be considered fully successful if it read from the wrong data source or skipped a required step. Evaluation should therefore track metrics such as:

    • Accuracy on individual questions
    • End-to-end task completion rate
    • Tool call success rate
    • Parameter format accuracy
    • Recovery success rate after task interruption
    • Frequency of repeated executions and ineffective calls

    2.2 Response Latency

    Average latency reflects interaction efficiency, but it does not fully represent the user experience. Long-running tasks should distinguish between:

    • Time to first response
    • Inference time for an individual call
    • Time to the follow-up response after a tool returns
    • Total time required to complete the workflow
    • Latency changes under different concurrency levels

    For chat-based question answering, a difference of a few seconds can be highly noticeable. For background batch processing, throughput and stability may matter more than single-request latency.

    2.3 Token Consumption and Invocation Cost

    Agent tasks often include extensive context, tool results, and execution history, causing token usage to increase rapidly. Evaluation should separately measure input and output consumption and examine the following factors:

    • Whether token usage grows linearly as the number of task rounds increases
    • Whether irrelevant historical information is repeatedly included
    • Whether the model can summarize large tool responses
    • Whether failed retries multiply the cost
    • Whether faster execution comes at the expense of higher token consumption

    In one original test, an agent-oriented model achieved an overall accuracy of approximately 71.5% across about 15,000 Chinese test questions. Average latency per call was about 52 seconds, with average consumption of approximately 2,934 tokens. Based on the test methodology, the cost was about 60.8 yuan per 1,000 calls.

    These figures should be treated as a sample rather than fixed results for every business scenario. Actual costs are also affected by context length, concurrency, the number of tool calls, caching strategies, and retry mechanisms.

    3. How to Interpret Faster Responses with Limited Accuracy Gains

    When comparing a new model version with an older one, it is common to see a substantial improvement in response speed while overall accuracy changes only slightly. This does not necessarily mean that the model’s capabilities have stagnated. It may indicate a shift in optimization priorities.

    If the model is primarily designed for complex toolchains and long-running workflows, its training objectives may place greater emphasis on:

    • Completing planning more quickly
    • Reducing unnecessary reasoning and repeated output
    • Improving the stability of structured calls
    • Shortening waiting time between tools
    • Increasing overall throughput while keeping accuracy within an acceptable range

    In the Chinese comprehensive evaluation, overall accuracy increased from approximately 71.0% to 71.5%. Language understanding and instruction following improved from about 67.6% to 69.3%, while reasoning and mathematical calculation increased from approximately 73.6% to 74.8%.

    These results indicate some improvement in foundational capabilities. However, the model’s more significant value may lie not in answering individual questions, but in improving overall execution efficiency in complex workflows.

    It is also important to recognize the limitations of conventional tool-use benchmarks. In many evaluations, tool use simply means selecting a tool or outputting a block of call parameters. This does not fully represent multi-turn interaction in a production system.

    Real-world agent tasks may involve permission checks, API errors, empty data, changing response formats, timeouts, retries, and human confirmation. Therefore, a small change in a general tool-use score cannot be used to directly predict performance in a real business environment.

    4. Why Long-Chain Tasks Require Dedicated Testing

    The difficulty of a long-chain task does not necessarily come from one individual step. It often comes from the accumulation of errors across multiple steps.

    If the success rate of each step is 95%, the theoretical probability of successfully completing 10 consecutive steps is:

    1. 0.95^10 ≈ 59.9%

    This means that even when a model performs well on individual steps, end-to-end success may decline significantly as the workflow becomes longer.

    During model selection, enterprises should pay particular attention to whether the model can:

    1. Preserve important state.
    2. Avoid repeating completed steps.
    3. Recognize errors returned by tools.
    4. Re-plan after an exception.
    5. Determine when human confirmation is required.
    6. Resume execution after a timeout.

    A useful test set should include both normal and exceptional paths. For example, the data interface can intentionally return an empty result, missing fields, or a permission error. The evaluation should determine whether the model responds appropriately instead of continuing to generate a complete-looking but incorrect conclusion.

    5. System Design Recommendations for Production

    Model capability is only one part of an agent system. Overall reliability also depends on the surrounding architecture.

    5.1 Use Structured Tool Protocols

    Tool names, parameter types, required fields, and response formats should be explicitly defined. Rather than allowing the model to freely generate commands that cannot be reliably validated, systems should use structured data formats and validate parameters before execution.

    5.2 Separate Planning from Execution

    For complex tasks, the system can generate a plan first and then execute it step by step. This improves auditability and allows execution to resume from the most recent completed node after a failure. High-risk operations should also include a human confirmation step.

    5.3 Control Context Length

    Do not pass every historical message to the model in its original form. Context can be managed through summaries, state tables, and key event records. Only information that is valuable to the current step needs to be retained.

    5.4 Implement Retry and Circuit-Breaking Mechanisms

    A tool call should not be retried indefinitely after a failure. The system needs to distinguish temporary errors from logical errors and define a maximum retry count, backoff interval, and conditions for human takeover.

    5.5 Build Comprehensive Logging and Observability

    Recommended records include:

    • The user’s goal
    • The plan generated by the model
    • Every tool call
    • Tool responses
    • Token consumption
    • Execution latency
    • Failure reasons
    • Final task status

    With complete logs, developers can determine whether a problem originates from the model, the tools, the data, or the workflow orchestration layer.

    6. Choosing the Right Model for a Business Scenario

    Model selection should not be based solely on the highest score in one category. Evaluation criteria should be weighted according to the characteristics of the target business scenario.

    For real-time question answering, first-token latency and response speed may be the most important factors. For data analysis, reasoning accuracy and structured output should receive greater weight. For workflows that continuously call multiple tools, parameter correctness, exception recovery, and long-context stability should be tested carefully. For high-volume workloads, token consumption, concurrency, and the cost of failed retries should be evaluated together.

    A practical scoring formula is:

    1. Overall score =
    2. Task completion rate × business weight
    3. + Stability × business weight
    4. + Response efficiency × business weight
    5. - Cost per task × cost weight

    The weights should be determined by the actual business requirements rather than copied directly from public benchmark rankings.

    7. Overall Assessment

    The value of a model optimized for agent workflows may not be reflected in a dramatic improvement in conventional question-answering scores. Its stronger value may appear in execution efficiency, instruction following, tool coordination, and long-chain stability.

    The Chinese comprehensive evaluation discussed above indicates that this type of model can generally maintain foundational accuracy close to that of the previous generation while delivering some improvement in instruction understanding, reasoning efficiency, and response speed.

    However, public benchmark results only describe a model’s baseline capabilities. They cannot replace validation against real business workflows. Before production deployment, enterprises should build end-to-end test sets using their own data, tools, and processes. Particular attention should be paid to exception handling, permission boundaries, cost growth, and human fallback mechanisms.

    The most useful question when evaluating an agent model is therefore not simply, “Is it the strongest model?” Instead, organizations should ask:

    • Can it complete the target business tasks reliably?
    • Can it control invocation costs?
    • Can its behavior be monitored and audited?
    • Can it stop safely when something goes wrong?

    Only by combining model capability with engineering architecture, workflow design, and risk controls can agent technology deliver reliable production value.