Agent Evaluation and Model Selection: A Practical Guide
Learn how to evaluate LLM agents for real-world tasks. This guide covers agent-specific evaluation dimensions, a two-stage testing workflow, tiered model routing between balanced and flagship models, and a cost-performance decision framework for choosing the right model mix.
Agent Evaluation and Model Selection: A Practical Guide
Why Benchmark Scores Aren’t Enough for Agent Selection
The biggest difference between agent applications and ordinary chat applications is that agents must complete a full loop: understand the goal, break it into steps, call tools, maintain state, validate results, and recover from failures. A model that performs well in single-turn Q&A will not necessarily execute multi-step tasks reliably. If you base your selection only on general leaderboards or one-off demos, you risk overlooking these critical issues:
- End-to-end completion rate: whether the task is actually completed, not just whether the intermediate steps look plausible.
- Long-horizon stability: whether goal drift, repeated calls, or missed constraints appear as step count grows.
- Tool call accuracy: whether parameter structures, call timing, and error handling behave as expected.
- Controllability and constraint adherence: whether the model strictly respects permissions, formats, workflows, and safety boundaries.
- Cost and latency: the per-call price is only the surface cost; retries, human review, and waiting time all count toward total cost.
- Context and output management: whether long documents, long logs, and multi-turn state are used effectively, and whether outputs are easy for downstream systems to consume.
Agent model selection should therefore be treated as an engineering decision: measure capability with reproducible evaluation sets, measure cost against business constraints, and control risk with canary rollout mechanisms.
Understanding Two Model Tiers: Balanced and Flagship
In most model matrices, models tend to fall into two positioning tiers: “balanced” and “flagship.” These are not simple substitutes for each other; they target different task profiles:
| Dimension | Balanced-Tier Model | Flagship-Tier Model |
|---|---|---|
| Core positioning | Balance of speed, cost, and intelligence | Complex long-horizon tasks and enterprise-grade controllability |
| Unit cost | Significantly lower | Higher |
| Latency | Usually faster | Usually medium or higher |
| Complex task capability | Covers most medium-complexity agent tasks | Stronger advantage in complex planning, long-horizon execution, and strict constraints |
| Context and output | Typically offers long context and large output limits | Also emphasizes long context and large output |
| Adaptive thinking | Depends on the specific service | Depends on the specific service |
| Benchmark performance | Should be monitored against your own task set | Usually ranks higher in complex evaluations |
| Knowledge cutoff | Check the actual service documentation | Check the actual service documentation |
In short: balanced-tier models cover most agent scenarios at a significantly lower cost, while flagship-tier models retain an advantage on the most complex long-horizon tasks, tightly constrained workflows, and higher controllability requirements. For enterprises, the right strategy is usually not “pick one” but tiered routing by task: high-frequency, low-risk, latency-sensitive tasks go to the balanced tier first; high-risk, long-chain, tightly constrained tasks are escalated to the flagship tier.
Building an Agent Evaluation Framework
1. End-to-End Task Success Rate
This is the core metric. Judge success by “whether the final business outcome is correct,” not by whether the intermediate text reads fluently. Define automatically verifiable results for each task type, such as whether structured fields are complete, whether tool calls produce the expected side effects, and whether the final answer satisfies constraints. For tasks that cannot be judged automatically, use dual review or rule-based checks with sampling.
2. Step Efficiency and Tool Call Quality
A common agent problem is not “can’t do it” but “does too much, does it messily.” Track average step count, the ratio of invalid tool calls, parameter error rate, and duplicate call rate. Models with high step efficiency typically reduce latency and cost, and also lower the risk of side effects.
3. Long-Horizon Stability and Error Recovery
For long-horizon tasks, observe whether the model stays consistent with its goal as steps accumulate. Include scenarios with distractors, transient failures, and abnormal tool responses in your evaluation, and check whether the model can detect errors, backtrack, and re-plan. Flagship-tier models often prove their value in exactly these complex scenarios.
4. Controllability and Constraint Adherence
Enterprise agents must respect permission boundaries, output formats, operational workflows, and audit requirements. Evaluations should include constraints such as “tool X must not be called,” “output must follow a specified structure,” and “no unauthorized reads,” and measure the violation rate. For high-risk tasks, controllability should carry more weight than raw capability scores.
5. Latency, Throughput, and Cost
Cost is more than input/output pricing. Factor in retry costs, human review costs, business losses from failures, and experience losses from latency. A model with a lower unit price but a high retry rate may end up costing more overall; a highly capable model with excessive latency may be unsuitable for real-time interaction.
6. Context Utilization and Output Consumability
Long context does not mean effective use of context. Check whether the model can locate key information in long logs, long documents, and multi-turn state, and whether it outputs structures that downstream systems can easily parse. When output limits are large, also watch for redundant content.
7. Knowledge Freshness and Safety Boundaries
The knowledge cutoff date affects performance on factual tasks. For time-sensitive workloads, combine retrieval augmentation, tool queries, and cache refresh mechanisms rather than relying entirely on parametric knowledge. Also evaluate security risks such as prompt injection, unauthorized tool calls, and sensitive data leakage.
A Two-Stage Evaluation Method: Offline Replay Plus Online Canary
Stage 1: Offline Replay
Build a tiered evaluation set: simple tasks, medium-complexity tasks, long-horizon tasks, and high-risk constrained tasks. For each task, store the input, available tools, expected results, and scoring rules. Run multiple samples per model and record success rate, variance, average step count, and cost. Never draw conclusions from a single run.
Stage 2: Online Canary
After passing offline evaluation, move to small-scale canary traffic. Set guardrail metrics: task success rate, violation rate, average latency, human takeover rate, and total cost per task. Keep fast rollback capability available during the canary period, and route failure samples back into the evaluation set to form a closed loop.
Pseudocode:
type TaskCase = {id: stringcomplexity: "low" | "medium" | "long_horizon" | "high_risk"input: stringtools: string[]expected: ResultSpecconstraints: string[]}type ModelProfile = {tier: "balanced" | "flagship"inputCostPerMillion: numberoutputCostPerMillion: numberlatencyLevel: "fast" | "medium"maxContext: numbermaxOutput: numberadaptiveThinking: boolean}function estimateCost(profile, inputTokens, outputTokens, retryFactor) {return retryFactor * (inputTokens / 1000000 * profile.inputCostPerMillion +outputTokens / 1000000 * profile.outputCostPerMillion)}function routeTask(task, profile) {if (task.complexity == "high_risk" || task.complexity == "long_horizon") {return "flagship"}if (task.complexity == "low" || task.complexity == "medium") {return "balanced"}return profile.tier}function evaluate(cases, profile) {result = []for (case in cases) {outcome = runAgent(case, profile)score = scoreOutcome(outcome, case.expected, case.constraints)cost = estimateCost(profile, outcome.inputTokens, outcome.outputTokens, outcome.retries)result.push({ caseId: case.id, score, cost, latency: outcome.latency })}return summarize(result)}
A Cost-Performance Decision Model
Make decisions using “total cost per task” rather than “price per million tokens”:
Total cost = model call cost + retry cost + human review cost + failure loss + business loss from latency
Balanced-tier models typically reduce call costs and latency significantly on medium-complexity tasks. Flagship-tier models on long-horizon tasks may offset part of their unit-price premium through fewer steps, lower retry rates, and lower human takeover rates. Calculate costs separately by task type instead of covering all scenarios with a single average score.
A practical decision tree:
- Is the task high-risk or tightly constrained? If yes, prefer the flagship tier and strengthen guardrails.
- Is the task long-horizon and multi-step with high failure costs? If yes, prefer the flagship tier or use hybrid routing.
- Is the task high-frequency, low-risk, and latency-sensitive? If yes, prefer the balanced tier.
- Can complexity be reduced through task decomposition, retrieval, or tool-based validation? If yes, optimize the workflow first before deciding on a model tier.
- Do you have rollback-capable canary deployment and monitoring alerts? If not, build that engineering capability before going live.
Common Pitfalls and Practical Recommendations
Common pitfalls include selecting models based on leaderboards alone, choosing by price alone, ignoring retry and human review costs, treating long context as infinite memory, lacking failure rollback, and failing to feed online failure samples back into the evaluation set. Enterprises should build tiered evaluation sets and model routing mechanisms, use the balanced tier as the default execution layer, use the flagship tier as the escalation layer for complex tasks, and use logging, monitoring alerts, and object storage to build a replayable evaluation loop. In a Baidu AI Cloud environment, you can combine general cloud offerings such as object storage, logging services, and monitoring alerts to build the infrastructure for task replay, metric observability, and canary rollback.
Ultimately, the core of agent model selection is not finding the “absolutely strongest model,” but finding the model mix that matches your task risk, latency requirements, cost budget, and engineering governance capability. Through reproducible evaluation, tiered routing, and continuous canary testing, teams can maintain a stable balance between capability, cost, and controllability.
