Evaluating and Selecting LLMs for AI Agents: A Practical Guide

1. Why Leaderboards Alone Are Not Enough

When selecting a model for an agent, comparing parameter counts, single-turn Q&A accuracy, or public leaderboard rankings often fails to reflect real production capability. An agent typically needs to complete a chain of connected operations: decomposing tasks, calling tools, managing state, modifying code, recovering from errors, and verifying results. Its final performance depends on the entire execution chain, not on any single generated response.

A model that performs well in ordinary Q&A may still show clear problems in the following scenarios:

• It cannot accurately understand the dependencies between steps in a multi-step task;
• Its tool call parameter formats are unstable;
• It does not adjust its strategy after an execution failure;
• It changes unrelated files when modifying code;
• It misses key constraints in long contexts;
• Its output looks complete but has not been verified.

Model selection should therefore shift from asking “Which model is stronger?” to asking “Which model best suits the current task?” Candidate models can generally be grouped into two categories:

1. Balanced models: fast response and lower cost, suitable for most routine development and office automation tasks.
2. High-end models for complex tasks: more robust reasoning chains, suited to long workflows, multi-tool use, strict constraints, and tasks where failures are costly.

The first category emphasizes throughput and cost efficiency. The second emphasizes success rate and controllability in complex scenarios.

2. Capability Differences Between the Two Model Types

Comparison dimension	Balanced models	High-end models for complex tasks
Primary positioning	Balance of speed, cost, and capability	Complex agents and enterprise-grade tasks
Typical strengths	Low latency, low unit cost, suited to large-scale calls	Stronger long-horizon reasoning, task planning, and error recovery
Context handling	Sufficient for most project-level tasks	Better suited to large codebases and long-document workflows
Tool calling	Stable for standard calls	More reliable for multi-round calls, conditional branching, and error repair
Output control	Suited to standardized templates	Better for strict constraints and complex output
Suitable tasks	Information extraction, customer support, simple coding, batch processing	Architecture analysis, complex refactoring, automated troubleshooting, continuous execution
Main risk	Tends to end early on complex tasks	Higher cost and generally longer response time
Selection keywords	High concurrency, low cost, fast response	High success rate, strong reasoning, low manual intervention

This classification does not mean balanced models are necessarily weak, or that high-end models suit every request. If a task only requires structured extraction, a high-end model wastes resources. If a task involves production changes, choosing only the lowest price can increase manual review and rework costs.

3. How to Break Down Agent Capability Evaluation

1. Task Planning

Task planning determines whether a model can turn a vague goal into executable steps. A strong model does not produce a conclusion immediately. It first identifies the goal, constraints, dependencies, and acceptance criteria.

To evaluate planning, you can design tasks such as:

• Extracting functional boundaries from a requirements description;
• Splitting a large task into subtasks that can each be verified independently;
• Determining the execution order based on prerequisites;
• Replanning after requirements change midway;
• Identifying steps that cannot be completed and actively requesting the necessary information.

Metrics include plan completeness, step reasonableness, the proportion of repeated operations, and the rate of missed requirements. For enterprise use, also check whether the model performs operations outside its authorized scope.

2. Tool Calling and Parameter Reliability

An agent’s practical capability depends heavily on tool calling. Tools can include database queries, file reads and writes, code execution, retrieval systems, messaging, or monitoring and alerting.

Key evaluation points include:

• Whether the model selects the correct tool;
• Whether parameter names and data types are correct;
• Whether the model handles empty results, timeouts, and abnormal returns;
• Whether the model checks necessary conditions before making a call;
• Whether the model can continue the task based on tool results;
• Whether the model avoids repeated and unnecessary calls.

Set clear input and output constraints for each tool, and record the call trace. Do not judge only whether the final answer is correct. Also analyze whether the call process can be audited and replayed.

3. Long Context and State Management

A long context does not simply mean a large pile of text. What matters is whether the model can locate key information within a large volume of content and consistently remember earlier decisions.

Focus tests on:

• Cross-references across multiple documents;
• Call-relationship analysis in large codebases;
• The effect of early constraints on later results;
• State consistency across multi-turn conversations;
• Identifying and handling conflicting information.

In engineering practice, avoid indiscriminately placing all historical records into the context. A better approach is to store task state, key decisions, tool results, and pending items in a structured form, then inject them into the model as needed. This reduces cost and limits context pollution.

4. Code Understanding and Modification

For coding agents, generating code that runs is not enough. You also need to assess the scope of changes and the maintainability of the result.

Build a test set around the following areas:

• Locating defects based on existing code;
• Adding features without breaking existing interfaces;
• Writing tests and explaining their coverage;
• Making consistent changes across multiple files;
• Analyzing error logs and proposing fixes;
• Running verification steps proactively after making changes.

Metrics can include test pass rate, number of modified files, proportion of unrelated changes, regression defects, and manual revision time. For high-risk projects, also check whether the model clearly states which parts it did not verify, rather than presenting speculation as a confirmed conclusion.

4. Combined Calculation of Cost, Latency, and Success Rate

Model pricing should not be measured only by the cost of a single call. Agents often require multiple rounds of reasoning and tool calls. The actual cost can be estimated as follows:

Cost per task = Input consumption × Input price + Output consumption × Output price + Tool and execution resource costs

When retries after failure are included, use:

Expected cost = Cost per task × Average number of call rounds ÷ Task success rate

For example, a low-cost model may have a lower cost per call. If it frequently produces parameter errors and needs repeated retries, its final cost may approach or even exceed that of a high-end model. Conversely, for a large volume of simple requests, the extra capability of a high-end model may not translate into real benefit.

Enterprises should therefore record:

• Average time to first byte;
• Total task completion time;
• Number of call rounds per task;
• Success rate and manual takeover rate;
• Average cost per successful task;
• Stability under peak concurrency.

This gives a closer picture of real operating results than comparing only the price per million tokens.

5. Designing an Evaluation Set

1. Layer Tasks by Difficulty

You can group test tasks into four levels:

• L1: Single-turn tasks, such as summarization, classification, and field extraction;
• L2: Multi-step tasks, such as retrieving materials and producing an analysis report;
• L3: Tool-coordinated tasks, such as reading data, processing it, and outputting results;
• L4: Long-horizon tasks, such as cross-file refactoring, continuous troubleshooting, and complex workflow automation.

Assign different weights to each level. Do not let a large number of simple questions hide defects in complex tasks.

2. Build a Unified Scoring Table

Include at least the following metrics:

Metric	Description
Task completion rate	Whether the final acceptance criteria are met
Factual accuracy rate	Whether key conclusions are supported by evidence
Tool success rate	Whether tool selection and parameters are correct
Constraint compliance rate	Whether permission, format, and scope limits are followed
Error recovery rate	Whether the model can adjust its strategy after a failure
Manual intervention rate	How often human takeover is required
Cost per task	Average consumption to complete one valid task
End-to-end latency	Total time from request to final result

For high-risk tasks, also count separately any unauthorized operations, sensitive information leaks, and unverified conclusions.

3. Use Real Business Samples

Public benchmarks reflect general capability but cannot replace validation on real business work. Enterprises can build an internal evaluation set by drawing de-identified samples from historical support tickets, code defects, operations incidents, knowledge base queries, and data processing workflows.

Version the evaluation set and add new cases regularly, so the model does not overfit to a fixed set of question types. After each model upgrade, run regression tests to confirm that gains in specific capabilities have not reduced format compliance, stability, or safety.

6. Selection Guidance for Different Scenarios

Scenarios That Should Favor Balanced Models

• Batch text classification and information extraction;
• Routine knowledge Q&A;
• Simple code completion and format conversion;
• Low-risk content generation;
• Interactive scenarios where response speed matters;
• Systems with high request volume and highly standardized tasks.

Stability for these tasks can be improved further with templates, structured output, and rule-based validation. Escalate a small number of complex requests to a high-end model.

Scenarios That Should Favor High-End Models for Complex Tasks

• Cross-module code analysis and refactoring;
• Multi-round tool calling with conditional branching;
• Complex fault diagnosis;
• Joint analysis of large volumes of documents and specifications;
• Automated workflows that must follow multiple strict constraints;
• Tasks where failures are costly or manual review is expensive.

For these scenarios, focus less on call price and more on first-pass completion rate and controllability.

7. Recommended Tiered Routing Architecture

A reliable production architecture is “lightweight model first, high-end model as fallback.” The basic flow is as follows:

User request
   ↓
Task classification and risk identification
   ↓
Assess complexity, context size, and tool requirements
   ↓
Balanced model handles routine tasks
   ↓
Low confidence, execution failure, or task escalation
   ↓
High-end model takes over
   ↓
Rule validation, permission checks, and human review
   ↓
Return the final result and record the trace

Routing conditions can include:

• Input length exceeds a threshold;
• The task involves multiple tools or multiple files;
• The task includes production changes;
• First-pass result confidence is insufficient;
• Tool calls fail consecutively;
• Output fails format or business rule validation.

The model should not have unlimited permissions. The tool layer should enforce least privilege, parameter validation, operation approval, and audit logging. The model should only propose actions or make controlled calls.

8. Conclusion

Selecting an agent model is an engineering decision that balances effectiveness, cost, latency, and risk. Balanced models generally suit large-scale, standardized, low-risk tasks. High-end models for complex tasks suit long-horizon reasoning, multi-tool coordination, and scenarios where failures are costly.

The best practice is not to use one model for every problem. It is to build a tiered strategy:

1. Build an evaluation set from real business tasks;
2. Divide model usage by complexity and risk;
3. Compare success rate, cost, and latency with a unified set of metrics;
4. Run routine tasks cheaply through a routing mechanism;
5. Set up a high-end model as a fallback for complex tasks;
6. Combine permission controls, log auditing, and human review to maintain safety;
7. Continuously run version regression tests and monitor performance in production.

Model capability only yields reliable selection conclusions when it is measured within the complete agent workflow.