LLM Agent Evaluation and Selection Guide

    This guide explains how to evaluate and select large language model (LLM) agents for real production use. It covers the shift from one-shot answering to closed-loop execution, why a single benchmark score is not enough, key evaluation metrics, common benchmark types, scenario-based selection guidance, and engineering practices for deployment, including model routing, verification, tool contracts, permissions, observability, and release strategy.

    1. From “Answering” to “Closed-Loop Execution”: The Core Shift in LLM Agent Capability

    LLM agent capability is often understood as the ability of a model to autonomously complete a full loop with minimal human intervention:

    Understand the goal → Plan the steps → Call tools → Check results → Fix errors

    The essential difference from traditional question-answering capability is this: question answering focuses on semantic matching between one input and one output, while agent tasks usually span multiple states, multiple tools, and multiple rounds of feedback.

    For example, if a user asks, “Help me check a failed task and try to fix it,” a question-answering model may only provide troubleshooting suggestions. An agent needs to continue executing multiple steps:

    • Read the task status
    • Locate relevant logs
    • Determine the failure type
    • Call diagnostic tools
    • Modify configuration
    • Re-trigger the task
    • Adjust the strategy if the task still fails

    In this process, the model must not only “say the right thing” but also “do the right thing,” and recover after errors occur.

    At a minimum, agent capability includes the following five layers:

    1. Goal understanding: Accurately identify the user’s real intent rather than being misled by surface wording.
    2. Task planning: Break a complex goal into executable steps and determine dependencies.
    3. Tool calling: Choose the correct tool and generate valid, complete, and usable call parameters.
    4. Result verification: Determine whether the tool’s result is successful instead of blindly moving to the next step.
    5. Error recovery: Adjust the plan based on error information, including retrying, degrading, switching paths, or requesting human intervention.

    If any of these stages is weak, the agent’s performance on long-horizon tasks will decline. For tasks that span dozens or even hundreds of steps, errors at individual steps can accumulate quickly and ultimately lead to overall failure.

    Therefore, agent evaluation should not focus only on whether a model can “chat.” It should focus on whether the model can reliably execute a complete closed loop.

    2. Why a Single Score Should Not Be the Basis for Model Selection

    A common mistake in agent selection is treating a single public benchmark score as an absolute standard. In reality, any single metric can only reflect one aspect of model capability.

    Public leaderboards are often based on real user conversations or broad scenario sampling. They can reflect overall user preference and capability trends, but they may not cover enterprise-specific workloads. Specialized benchmarks can evaluate specific capabilities such as terminal operation, graphical interface operation, or code repair, but they may lack cost constraints. Enterprise workloads often include additional engineering constraints such as data security, permission control, response latency, inference cost, and audit compliance.

    A more reasonable approach is to build a three-dimensional cross-evaluation framework:

    • Effectiveness dimension: task success rate, error recovery rate, and planning quality.
    • Engineering dimension: latency, stability, tool compatibility, and permission safety.
    • Cost dimension: input and output consumption, retry count, and human intervention cost.

    Only by considering all three dimensions together can a selection decision be practically deployed.

    A model that performs well on a public leaderboard may not fit every business scenario. A model with a lower price may not be more cost-effective on complex tasks. The real goal of selection is to find the right balance among effectiveness, cost, risk, and operational complexity.

    3. Key Metrics for Agent Evaluation

    1. Task Success Rate

    Task success rate is the most direct and important metric. It answers a simple question: can the model ultimately achieve the user’s goal?

    In evaluation, a task should be defined as “verifiably successful,” not merely “looking complete.”

    For example:

    • An operations agent may generate repair steps, but if the target system is still unhealthy, the task has not succeeded.
    • A coding agent may generate a patch, but if the tests do not pass, the task has not succeeded.

    Therefore, success rate must be bound to clear acceptance criteria, such as:

    • API return codes
    • Database state
    • Test results
    • Human confirmation outcomes

    2. Net Improvement Rate

    Net improvement rate is often used in blind tests of real conversations or in comparative evaluations. It measures the relative improvement of a model over a baseline model.

    It is not simply a win rate. It measures whether the model delivers stable quality gains across overall task quality.

    In practical selection, net improvement rate helps answer an important question: is a higher capability tier worth the extra cost?

    If a model provides only a small improvement over a basic model while introducing significantly higher cost or latency, it may not be worthwhile for large-scale scenarios. Conversely, if the model delivers a significant improvement on complex tasks, it may be highly valuable in high-risk scenarios.

    3. Step Efficiency

    For agents, more steps do not mean better performance. The goal is to complete the task with as few effective steps as possible.

    Step efficiency includes:

    • Average steps per task
    • Number of tool calls
    • Number of ineffective retries
    • Number of context switches
    • How close the execution path is to the shortest successful path

    High step efficiency usually means lower latency and lower cost.

    Some models may eventually complete a task, but only after many trial-and-error attempts. In production environments, this can amplify risk.

    4. Tool-Call Correctness

    Tool calling is often the deciding factor in whether an agent can be deployed successfully.

    The model must understand:

    • Tool descriptions
    • Parameter structures
    • Required fields
    • Return formats
    • Error semantics

    When evaluating tool calling, at minimum, you should check:

    • Whether the correct tool is selected
    • Whether parameters are complete and correctly typed
    • Whether the model can handle empty values, timeouts, and permission errors
    • Whether the model can correct parameters based on error messages
    • Whether the model avoids unauthorized or dangerous operations

    For complex toolchains, it is recommended to design independent acceptance tests for each tool rather than evaluating only end-to-end tasks.

    5. Error Recovery Capability

    Long-horizon tasks cannot be error-free. The key quality of a strong agent is not that it never fails, but that it can identify the cause, adjust its strategy, or exit safely after a failure.

    To evaluate error recovery, you can intentionally inject faults, such as:

    • Tool timeout
    • Parameter validation failure
    • Insufficient permissions
    • Malformed data
    • Temporary service unavailability
    • Mid-task changes to the user’s goal

    The key observations are:

    • Does the model blindly retry?
    • Does it fall into a loop?
    • Does it request human confirmation in high-risk scenarios?

    6. Latency and Cost

    Agent tasks usually require multiple model calls, so cost increases with the number of steps.

    When evaluating latency and cost, focus on:

    • Average consumption per task
    • P95 latency
    • Maximum retry count
    • Extra cost caused by a single failure
    • Human intervention rate
    • Additional overhead from long contexts

    A model may be highly capable, but if each task requires a large amount of context and multiple retries, its total cost may be much higher than its advertised unit price suggests.

    4. Common Evaluation Methods and Their Boundaries

    1. Public Leaderboards Based on Real Conversations

    These evaluations are often based on large volumes of real user requests and use blind testing. They can reflect a model’s overall performance in general scenarios.

    Advantages:

    • Broad coverage

    Limitations:

    • Difficult to precisely reflect enterprise-specific constraints

    Suitable for:

    • Initial screening of capability tiers
    • Understanding the gap between leading models and general models

    2. Terminal-Based Multi-Step Task Evaluations

    These evaluations simulate complex operations in command-line environments and assess a model’s ability to execute multi-step tasks, recover state, and correct errors.

    They are suitable for evaluating:

    • Operations
    • Development
    • Automation workflows

    Suitable for:

    • Tasks that require terminal operation
    • Script execution
    • System diagnosis

    3. Graphical Interface Operation Evaluations

    These evaluations assess a model’s ability to simulate human interaction with desktop software, such as:

    • Opening applications
    • Clicking buttons
    • Filling out forms
    • Reading interface state

    These tasks place high demands on:

    • Visual understanding
    • Interface element localization
    • State tracking

    Suitable for:

    • Tasks that have no stable API and must be completed through a graphical interface

    4. Code and Repository Task Evaluations

    These evaluations focus on a model’s understanding of:

    • Code context
    • Dependencies
    • Test structures
    • Error localization

    They are suitable for evaluating:

    • AI-assisted programming
    • Defect repair
    • Test generation

    Suitable for:

    • Coding assistants
    • Automated repair
    • R&D workflow assistance

    5. Enterprise Private Evaluation Sets

    No public evaluation can fully replace a private evaluation set.

    Enterprises should build their own golden task sets based on real business logs, covering:

    • High-frequency scenarios
    • Boundary scenarios
    • High-risk scenarios

    Suitable for:

    • Regression validation before official launch
    • Model switch evaluation
    • Stability checks after changes to prompts or tools

    5. Scenario-Based Selection Decision Matrix

    1. High-Complexity, Long-Horizon Tasks

    Typical characteristics:

    • Many steps
    • Many dependencies
    • High risk
    • High human intervention cost

    Selection focus:

    • Prioritize models with stronger planning capability
    • Prioritize long-context understanding
    • Prioritize error recovery capability

    These tasks should not be selected solely on low cost. One failure may lead to much higher rework costs.

    Recommendation:

    Use a combination of a strong execution model and a strong verifier, and add human confirmation before high-risk steps.

    2. Cost-Sensitive Batch Tasks

    Typical characteristics:

    • Large task volume
    • Lower value per task
    • Errors may be acceptable or easy to fix

    Selection focus:

    • Prioritize fast response
    • Prioritize low inference cost
    • Prioritize stable structured output

    You can compensate for model limitations with:

    • Rule-based checks
    • Template-driven workflows
    • Secondary review

    Recommendation:

    Use a lightweight model plus a rule engine plus exception escalation. Escalate only complex cases to a stronger model.

    3. Terminal Operations and System Diagnosis

    Typical characteristics:

    • Command execution
    • State checking
    • Log analysis
    • Fault recovery

    Selection focus:

    • Tool-call safety
    • Error recovery capability
    • Permission control
    • Auditability

    The model must be able to identify dangerous commands and request confirmation before irreversible operations.

    Recommendation:

    Build a sandbox environment, restrict the executable scope, and keep complete logs for all tool calls.

    4. Graphical Interface and Desktop Operations

    Typical characteristics:

    • The target software has no stable API
    • Tasks must be completed through the interface

    Selection focus:

    • Visual understanding
    • Element localization
    • State tracking
    • Exception recovery

    Because interfaces change frequently, rigid dependencies should be minimized. Add screenshot-based verification and human fallback mechanisms.

    Recommendation:

    Prefer models that support configurable operation traces, and add screenshot or result confirmation for key steps.

    5. AI-Assisted Programming and Code Repair

    Typical characteristics:

    • Requires understanding of the codebase
    • Requires understanding of dependencies
    • Requires understanding of test conventions
    • Requires understanding of error context

    Selection focus:

    • Long-context capability
    • Code understanding
    • Patch generation
    • Test verification

    The model should not only generate code. It should also be able to explain failure reasons and adjust its implementation.

    Recommendation:

    Place the model inside a closed loop of generation, execution, testing, and repair rather than limiting it to producing suggestions only.

    6. Customer Support and Knowledge-QA Enhancement

    Typical characteristics:

    • Emphasis on accurate answers
    • Emphasis on compliant expression
    • Emphasis on retrieval and tool usage

    Selection focus:

    • Retrieval-augmented capability
    • Knowledge citation capability
    • Refusal mechanisms
    • Sensitive content control

    For tasks that require checking orders, logistics, or configurations, you should also evaluate tool-call accuracy.

    Recommendation:

    Use a combination of retrieval, tool calling, and compliance review to prevent the model from fabricating facts.

    6. Engineering Implementation: Making Evaluation Results Useful in Production

    1. Establish Model Tiering and Task Routing

    A production system usually should not route all tasks to the same model.

    A more reasonable approach is to route tasks based on:

    • Risk level
    • Complexity
    • Cost budget

    For example:

    • High-risk, multi-step, high-dependency tasks: use high-capability models
    • Medium-complexity, verifiable tasks: use balanced models
    • High-frequency, low-value, quickly completable tasks: use lightweight models

    This tiered design can significantly reduce cost while preserving the capability ceiling for complex scenarios.

    2. Use Verifiers Instead of “Model Self-Assessment”

    It is not reliable to assume that a model has completed a task just because it says so.

    A more robust approach is to introduce independent verifiers, such as:

    • Test scripts
    • State queries
    • API assertions
    • Rule-based checks
    • Human review

    The verifier should sit outside the model, ensuring that task success is not determined solely by the model’s own judgment.

    A typical closed loop looks like this:

    1. Receive the user's goal
    2. ↓
    3. Model generates a task plan
    4. ↓
    5. Call tools or APIs
    6. ↓
    7. Verifier checks the result
    8. ↓
    9. If successful: return the result
    10. If failed: send the error details back to the model
    11. ↓
    12. Model revises the plan and retries

    The key to this structure is decoupling execution from verification.

    The model is responsible for reasoning and repair. The verifier is responsible for factual judgment.

    3. Tool Contracts Must Be Clear

    Whether an agent can call tools reliably depends heavily on how clearly the tools are defined.

    Each tool should at least include:

    • Name and purpose
    • Input parameters and types
    • Required fields
    • Return structure
    • Common error codes
    • Whether the operation is reversible
    • Whether human confirmation is required

    Tool descriptions should not be written only for engineers. They should also be easy for the model to understand.

    Parameter names, field descriptions, and examples should be concise and clear, avoiding ambiguity.

    4. Permission Control and Sandboxing

    Agents naturally have execution capabilities, so security design cannot be added later.

    Recommended practices follow the principle of least privilege:

    • Expose read-only tools by default
    • Require explicit authorization for write operations and delete operations
    • Route high-risk tasks into sandboxes or test environments
    • Keep audit logs for all tool calls
    • Set thresholds for batch operations and cross-system operations
    • Require human confirmation for irreversible operations

    In complex systems, an agent should be treated as a controlled executor, not an unbounded autonomous proxy.

    5. Observability: Logs, Tracing, and Alerts

    An agent system without observability is difficult to operate over the long term.

    At a minimum, you should record:

    • User input
    • Model plans
    • Tool-call parameters
    • Tool return results
    • Time taken per step
    • Failure reasons
    • Whether retries occurred
    • Whether human intervention occurred
    • Final task state

    On Baidu AI Cloud, for example, you can combine general cloud capabilities such as object storage, message queues, logging, and monitoring/alerting to build a complete agent runtime and evaluation pipeline.

    For example:

    • Use object storage to archive evaluation samples and execution traces
    • Use message queues to handle asynchronous tasks
    • Use logging for retrieval and analysis
    • Use monitoring and alerts to detect abnormal task failure clusters

    6. Build Offline Regression and Online Canary Mechanisms

    After an agent system goes live, any change to the model, prompt, tool description, or knowledge base may affect behavior.

    It is recommended to establish two mechanisms:

    • Offline regression: Before every change, run the golden task set to confirm that success rate, cost, and latency do not degrade significantly.
    • Online canary: Release changes to a small percentage of traffic and compare old and new versions in real conditions. Monitor failure rate, human takeover rate, and user feedback.

    Full rollout is appropriate only after both offline metrics and online performance remain stable.

    7. Common Pitfalls and How to Avoid Them

    Pitfall 1: Relying Only on Leaderboards

    Leaderboards can provide reference points, but they cannot replace business-specific evaluation.

    Enterprises should derive evaluation sets from real tasks to ensure that model capabilities match their own scenarios.

    Pitfall 2: Looking Only at Success Rate, Not Recovery Rate

    Long-horizon tasks inevitably involve uncertainty.

    Compared with “succeeding on the first try,” a production system often needs “controlled failure recovery” more.

    When evaluating, you should inject exceptions and observe whether the model can exit safely or request human help.

    Pitfall 3: Ignoring Inference and Operational Costs

    Agent cost is not limited to model fees. It also includes:

    • Retries
    • Human intervention
    • System resources
    • Rework caused by failures

    Full lifecycle cost should be included in evaluation.

    Pitfall 4: Missing Permission Boundaries

    Agents without permission controls may create risks such as data leakage, accidental deletion, or accidental modification.

    Production systems must treat security controls as equally important as capability evaluation.

    Pitfall 5: Treating the Agent as a One-Time Model Call

    An agent is a system problem, not a single-model problem.

    It involves:

    • Models
    • Tools
    • Verifiers
    • Memory
    • Scheduling
    • Monitoring
    • Human workflows

    Switching models without optimizing the system often yields limited gains.

    8. Conclusion: Selection Is Not About Finding the “Strongest Model,” but the “Most Suitable Solution”

    Evaluating and selecting LLM agent capability is essentially about balancing real business goals, engineering constraints, and cost boundaries.

    Public benchmarks can help identify capability trends. Specialized benchmarks can validate key skills. But the final production outcome still depends on:

    • The enterprise’s private evaluation sets
    • Verification mechanisms
    • Engineering capabilities

    For developers and enterprises, a more reliable path is:

    1. Define the task closed loop and acceptance criteria
    2. Use public benchmarks to shortlist capable candidates
    3. Validate real performance with a private golden task set
    4. Introduce verifiers, permission controls, and observability
    5. Use model tiering and task routing to reduce cost
    6. Continue iterating through canary releases and offline regression

    When the evaluation system moves from “single-point scores” to “closed-loop quality,” model selection becomes more than blind comparison. It becomes a reusable engineering decision.

    The most valuable goal is not chasing a single ranking, but building a system that can continuously measure, verify, and optimize agent capability.