LLM Agent Evaluation and Selection Guide
This guide explains how to evaluate and select large language model (LLM) agents for real production use. It covers the shift from one-shot answering to closed-loop execution, why a single benchmark score is not enough, key evaluation metrics, common benchmark types, scenario-based selection guidance, and engineering practices for deployment, including model routing, verification, tool contracts, permissions, observability, and release strategy.
1. From “Answering” to “Closed-Loop Execution”: The Core Shift in LLM Agent Capability
LLM agent capability is often understood as the ability of a model to autonomously complete a full loop with minimal human intervention:
Understand the goal → Plan the steps → Call tools → Check results → Fix errors
The essential difference from traditional question-answering capability is this: question answering focuses on semantic matching between one input and one output, while agent tasks usually span multiple states, multiple tools, and multiple rounds of feedback.
For example, if a user asks, “Help me check a failed task and try to fix it,” a question-answering model may only provide troubleshooting suggestions. An agent needs to continue executing multiple steps:
- Read the task status
- Locate relevant logs
- Determine the failure type
- Call diagnostic tools
- Modify configuration
- Re-trigger the task
- Adjust the strategy if the task still fails
In this process, the model must not only “say the right thing” but also “do the right thing,” and recover after errors occur.
At a minimum, agent capability includes the following five layers:
- Goal understanding: Accurately identify the user’s real intent rather than being misled by surface wording.
- Task planning: Break a complex goal into executable steps and determine dependencies.
- Tool calling: Choose the correct tool and generate valid, complete, and usable call parameters.
- Result verification: Determine whether the tool’s result is successful instead of blindly moving to the next step.
- Error recovery: Adjust the plan based on error information, including retrying, degrading, switching paths, or requesting human intervention.
If any of these stages is weak, the agent’s performance on long-horizon tasks will decline. For tasks that span dozens or even hundreds of steps, errors at individual steps can accumulate quickly and ultimately lead to overall failure.
Therefore, agent evaluation should not focus only on whether a model can “chat.” It should focus on whether the model can reliably execute a complete closed loop.
2. Why a Single Score Should Not Be the Basis for Model Selection
A common mistake in agent selection is treating a single public benchmark score as an absolute standard. In reality, any single metric can only reflect one aspect of model capability.
Public leaderboards are often based on real user conversations or broad scenario sampling. They can reflect overall user preference and capability trends, but they may not cover enterprise-specific workloads. Specialized benchmarks can evaluate specific capabilities such as terminal operation, graphical interface operation, or code repair, but they may lack cost constraints. Enterprise workloads often include additional engineering constraints such as data security, permission control, response latency, inference cost, and audit compliance.
A more reasonable approach is to build a three-dimensional cross-evaluation framework:
- Effectiveness dimension: task success rate, error recovery rate, and planning quality.
- Engineering dimension: latency, stability, tool compatibility, and permission safety.
- Cost dimension: input and output consumption, retry count, and human intervention cost.
Only by considering all three dimensions together can a selection decision be practically deployed.
A model that performs well on a public leaderboard may not fit every business scenario. A model with a lower price may not be more cost-effective on complex tasks. The real goal of selection is to find the right balance among effectiveness, cost, risk, and operational complexity.
3. Key Metrics for Agent Evaluation
1. Task Success Rate
Task success rate is the most direct and important metric. It answers a simple question: can the model ultimately achieve the user’s goal?
In evaluation, a task should be defined as “verifiably successful,” not merely “looking complete.”
For example:
- An operations agent may generate repair steps, but if the target system is still unhealthy, the task has not succeeded.
- A coding agent may generate a patch, but if the tests do not pass, the task has not succeeded.
Therefore, success rate must be bound to clear acceptance criteria, such as:
- API return codes
- Database state
- Test results
- Human confirmation outcomes
2. Net Improvement Rate
Net improvement rate is often used in blind tests of real conversations or in comparative evaluations. It measures the relative improvement of a model over a baseline model.
It is not simply a win rate. It measures whether the model delivers stable quality gains across overall task quality.
In practical selection, net improvement rate helps answer an important question: is a higher capability tier worth the extra cost?
If a model provides only a small improvement over a basic model while introducing significantly higher cost or latency, it may not be worthwhile for large-scale scenarios. Conversely, if the model delivers a significant improvement on complex tasks, it may be highly valuable in high-risk scenarios.
3. Step Efficiency
For agents, more steps do not mean better performance. The goal is to complete the task with as few effective steps as possible.
Step efficiency includes:
- Average steps per task
- Number of tool calls
- Number of ineffective retries
- Number of context switches
- How close the execution path is to the shortest successful path
High step efficiency usually means lower latency and lower cost.
Some models may eventually complete a task, but only after many trial-and-error attempts. In production environments, this can amplify risk.
4. Tool-Call Correctness
Tool calling is often the deciding factor in whether an agent can be deployed successfully.
The model must understand:
- Tool descriptions
- Parameter structures
- Required fields
- Return formats
- Error semantics
When evaluating tool calling, at minimum, you should check:
- Whether the correct tool is selected
- Whether parameters are complete and correctly typed
- Whether the model can handle empty values, timeouts, and permission errors
- Whether the model can correct parameters based on error messages
- Whether the model avoids unauthorized or dangerous operations
For complex toolchains, it is recommended to design independent acceptance tests for each tool rather than evaluating only end-to-end tasks.
5. Error Recovery Capability
Long-horizon tasks cannot be error-free. The key quality of a strong agent is not that it never fails, but that it can identify the cause, adjust its strategy, or exit safely after a failure.
To evaluate error recovery, you can intentionally inject faults, such as:
- Tool timeout
- Parameter validation failure
- Insufficient permissions
- Malformed data
- Temporary service unavailability
- Mid-task changes to the user’s goal
The key observations are:
- Does the model blindly retry?
- Does it fall into a loop?
- Does it request human confirmation in high-risk scenarios?
6. Latency and Cost
Agent tasks usually require multiple model calls, so cost increases with the number of steps.
When evaluating latency and cost, focus on:
- Average consumption per task
- P95 latency
- Maximum retry count
- Extra cost caused by a single failure
- Human intervention rate
- Additional overhead from long contexts
A model may be highly capable, but if each task requires a large amount of context and multiple retries, its total cost may be much higher than its advertised unit price suggests.
4. Common Evaluation Methods and Their Boundaries
1. Public Leaderboards Based on Real Conversations
These evaluations are often based on large volumes of real user requests and use blind testing. They can reflect a model’s overall performance in general scenarios.
Advantages:
- Broad coverage
Limitations:
- Difficult to precisely reflect enterprise-specific constraints
Suitable for:
- Initial screening of capability tiers
- Understanding the gap between leading models and general models
2. Terminal-Based Multi-Step Task Evaluations
These evaluations simulate complex operations in command-line environments and assess a model’s ability to execute multi-step tasks, recover state, and correct errors.
They are suitable for evaluating:
- Operations
- Development
- Automation workflows
Suitable for:
- Tasks that require terminal operation
- Script execution
- System diagnosis
3. Graphical Interface Operation Evaluations
These evaluations assess a model’s ability to simulate human interaction with desktop software, such as:
- Opening applications
- Clicking buttons
- Filling out forms
- Reading interface state
These tasks place high demands on:
- Visual understanding
- Interface element localization
- State tracking
Suitable for:
- Tasks that have no stable API and must be completed through a graphical interface
4. Code and Repository Task Evaluations
These evaluations focus on a model’s understanding of:
- Code context
- Dependencies
- Test structures
- Error localization
They are suitable for evaluating:
- AI-assisted programming
- Defect repair
- Test generation
Suitable for:
- Coding assistants
- Automated repair
- R&D workflow assistance
5. Enterprise Private Evaluation Sets
No public evaluation can fully replace a private evaluation set.
Enterprises should build their own golden task sets based on real business logs, covering:
- High-frequency scenarios
- Boundary scenarios
- High-risk scenarios
Suitable for:
- Regression validation before official launch
- Model switch evaluation
- Stability checks after changes to prompts or tools
5. Scenario-Based Selection Decision Matrix
1. High-Complexity, Long-Horizon Tasks
Typical characteristics:
- Many steps
- Many dependencies
- High risk
- High human intervention cost
Selection focus:
- Prioritize models with stronger planning capability
- Prioritize long-context understanding
- Prioritize error recovery capability
These tasks should not be selected solely on low cost. One failure may lead to much higher rework costs.
Recommendation:
Use a combination of a strong execution model and a strong verifier, and add human confirmation before high-risk steps.
2. Cost-Sensitive Batch Tasks
Typical characteristics:
- Large task volume
- Lower value per task
- Errors may be acceptable or easy to fix
Selection focus:
- Prioritize fast response
- Prioritize low inference cost
- Prioritize stable structured output
You can compensate for model limitations with:
- Rule-based checks
- Template-driven workflows
- Secondary review
Recommendation:
Use a lightweight model plus a rule engine plus exception escalation. Escalate only complex cases to a stronger model.
3. Terminal Operations and System Diagnosis
Typical characteristics:
- Command execution
- State checking
- Log analysis
- Fault recovery
Selection focus:
- Tool-call safety
- Error recovery capability
- Permission control
- Auditability
The model must be able to identify dangerous commands and request confirmation before irreversible operations.
Recommendation:
Build a sandbox environment, restrict the executable scope, and keep complete logs for all tool calls.
4. Graphical Interface and Desktop Operations
Typical characteristics:
- The target software has no stable API
- Tasks must be completed through the interface
Selection focus:
- Visual understanding
- Element localization
- State tracking
- Exception recovery
Because interfaces change frequently, rigid dependencies should be minimized. Add screenshot-based verification and human fallback mechanisms.
Recommendation:
Prefer models that support configurable operation traces, and add screenshot or result confirmation for key steps.
5. AI-Assisted Programming and Code Repair
Typical characteristics:
- Requires understanding of the codebase
- Requires understanding of dependencies
- Requires understanding of test conventions
- Requires understanding of error context
Selection focus:
- Long-context capability
- Code understanding
- Patch generation
- Test verification
The model should not only generate code. It should also be able to explain failure reasons and adjust its implementation.
Recommendation:
Place the model inside a closed loop of generation, execution, testing, and repair rather than limiting it to producing suggestions only.
6. Customer Support and Knowledge-QA Enhancement
Typical characteristics:
- Emphasis on accurate answers
- Emphasis on compliant expression
- Emphasis on retrieval and tool usage
Selection focus:
- Retrieval-augmented capability
- Knowledge citation capability
- Refusal mechanisms
- Sensitive content control
For tasks that require checking orders, logistics, or configurations, you should also evaluate tool-call accuracy.
Recommendation:
Use a combination of retrieval, tool calling, and compliance review to prevent the model from fabricating facts.
6. Engineering Implementation: Making Evaluation Results Useful in Production
1. Establish Model Tiering and Task Routing
A production system usually should not route all tasks to the same model.
A more reasonable approach is to route tasks based on:
- Risk level
- Complexity
- Cost budget
For example:
- High-risk, multi-step, high-dependency tasks: use high-capability models
- Medium-complexity, verifiable tasks: use balanced models
- High-frequency, low-value, quickly completable tasks: use lightweight models
This tiered design can significantly reduce cost while preserving the capability ceiling for complex scenarios.
2. Use Verifiers Instead of “Model Self-Assessment”
It is not reliable to assume that a model has completed a task just because it says so.
A more robust approach is to introduce independent verifiers, such as:
- Test scripts
- State queries
- API assertions
- Rule-based checks
- Human review
The verifier should sit outside the model, ensuring that task success is not determined solely by the model’s own judgment.
A typical closed loop looks like this:
Receive the user's goal↓Model generates a task plan↓Call tools or APIs↓Verifier checks the result↓If successful: return the resultIf failed: send the error details back to the model↓Model revises the plan and retries
The key to this structure is decoupling execution from verification.
The model is responsible for reasoning and repair. The verifier is responsible for factual judgment.
3. Tool Contracts Must Be Clear
Whether an agent can call tools reliably depends heavily on how clearly the tools are defined.
Each tool should at least include:
- Name and purpose
- Input parameters and types
- Required fields
- Return structure
- Common error codes
- Whether the operation is reversible
- Whether human confirmation is required
Tool descriptions should not be written only for engineers. They should also be easy for the model to understand.
Parameter names, field descriptions, and examples should be concise and clear, avoiding ambiguity.
4. Permission Control and Sandboxing
Agents naturally have execution capabilities, so security design cannot be added later.
Recommended practices follow the principle of least privilege:
- Expose read-only tools by default
- Require explicit authorization for write operations and delete operations
- Route high-risk tasks into sandboxes or test environments
- Keep audit logs for all tool calls
- Set thresholds for batch operations and cross-system operations
- Require human confirmation for irreversible operations
In complex systems, an agent should be treated as a controlled executor, not an unbounded autonomous proxy.
5. Observability: Logs, Tracing, and Alerts
An agent system without observability is difficult to operate over the long term.
At a minimum, you should record:
- User input
- Model plans
- Tool-call parameters
- Tool return results
- Time taken per step
- Failure reasons
- Whether retries occurred
- Whether human intervention occurred
- Final task state
On Baidu AI Cloud, for example, you can combine general cloud capabilities such as object storage, message queues, logging, and monitoring/alerting to build a complete agent runtime and evaluation pipeline.
For example:
- Use object storage to archive evaluation samples and execution traces
- Use message queues to handle asynchronous tasks
- Use logging for retrieval and analysis
- Use monitoring and alerts to detect abnormal task failure clusters
6. Build Offline Regression and Online Canary Mechanisms
After an agent system goes live, any change to the model, prompt, tool description, or knowledge base may affect behavior.
It is recommended to establish two mechanisms:
- Offline regression: Before every change, run the golden task set to confirm that success rate, cost, and latency do not degrade significantly.
- Online canary: Release changes to a small percentage of traffic and compare old and new versions in real conditions. Monitor failure rate, human takeover rate, and user feedback.
Full rollout is appropriate only after both offline metrics and online performance remain stable.
7. Common Pitfalls and How to Avoid Them
Pitfall 1: Relying Only on Leaderboards
Leaderboards can provide reference points, but they cannot replace business-specific evaluation.
Enterprises should derive evaluation sets from real tasks to ensure that model capabilities match their own scenarios.
Pitfall 2: Looking Only at Success Rate, Not Recovery Rate
Long-horizon tasks inevitably involve uncertainty.
Compared with “succeeding on the first try,” a production system often needs “controlled failure recovery” more.
When evaluating, you should inject exceptions and observe whether the model can exit safely or request human help.
Pitfall 3: Ignoring Inference and Operational Costs
Agent cost is not limited to model fees. It also includes:
- Retries
- Human intervention
- System resources
- Rework caused by failures
Full lifecycle cost should be included in evaluation.
Pitfall 4: Missing Permission Boundaries
Agents without permission controls may create risks such as data leakage, accidental deletion, or accidental modification.
Production systems must treat security controls as equally important as capability evaluation.
Pitfall 5: Treating the Agent as a One-Time Model Call
An agent is a system problem, not a single-model problem.
It involves:
- Models
- Tools
- Verifiers
- Memory
- Scheduling
- Monitoring
- Human workflows
Switching models without optimizing the system often yields limited gains.
8. Conclusion: Selection Is Not About Finding the “Strongest Model,” but the “Most Suitable Solution”
Evaluating and selecting LLM agent capability is essentially about balancing real business goals, engineering constraints, and cost boundaries.
Public benchmarks can help identify capability trends. Specialized benchmarks can validate key skills. But the final production outcome still depends on:
- The enterprise’s private evaluation sets
- Verification mechanisms
- Engineering capabilities
For developers and enterprises, a more reliable path is:
- Define the task closed loop and acceptance criteria
- Use public benchmarks to shortlist capable candidates
- Validate real performance with a private golden task set
- Introduce verifiers, permission controls, and observability
- Use model tiering and task routing to reduce cost
- Continue iterating through canary releases and offline regression
When the evaluation system moves from “single-point scores” to “closed-loop quality,” model selection becomes more than blind comparison. It becomes a reusable engineering decision.
The most valuable goal is not chasing a single ranking, but building a system that can continuously measure, verify, and optimize agent capability.
