Evaluating and Selecting LLM Agents for Tool Calling: A Practical Guide
A practical framework for evaluating AI agents beyond traditional benchmarks, covering tool selection, parameter generation, multi-step execution, accuracy, latency, token usage, cost, reproducibility, and business-oriented model selection. Includes a minimal Python evaluation script and a step-by-step workflow for validating candidate models in production-like environments.
Introduction: Why Agent Evaluation Needs a New Approach
Traditional large language model evaluations have focused mainly on question-answering accuracy and text-generation quality. As AI agents become more widely adopted, models are no longer limited to answering questions. They must also call external tools, decompose multi-step tasks, and maintain consistent state across longer execution chains. In these scenarios, conventional evaluation metrics are no longer sufficient.
Developers often face a practical problem during model selection: a model may rank highly on general-purpose benchmarks but perform only moderately well in a specific business workflow. This happens because the capabilities measured by public benchmarks do not always match the requirements of a particular application. Understanding this gap is the starting point for effective agent evaluation.
1. From Single-Turn Questions to Multi-Step Execution
A traditional question-answering task usually follows a simple pattern: provide one question and evaluate the resulting text. Agent tasks are more complex and typically include the following steps:
- Understand the user’s intent and determine whether a tool is needed.
- Select the appropriate tool and construct parameters according to its interface.
- Parse the tool’s response and decide what to do next.
- Complete the task after multiple interactions, or apply a reasonable fallback strategy when something fails.
Each step can introduce errors, and those errors may accumulate throughout the execution chain. For example, selecting the wrong tool in the first step can cause every subsequent step to move away from the intended goal. Agent evaluation should therefore examine more than the final answer. It should also inspect the intermediate process, including whether the tool choice was correct, the parameters were complete, and the step sequence was reasonable.
Another frequently overlooked capability is persistent task execution. Suppose a user asks an agent to check a data source every morning and send a notification when an anomaly is detected. The model must understand the time constraint, retain the task context, and trigger the appropriate action at the right time. This type of capability is difficult to measure with a single test case and usually requires simulating multiple interactions over a period of time.
2. Evaluation Dimensions: Accuracy, Latency, Token Usage, and Cost
A practical evaluation should cover at least four major categories of metrics, while recognizing that these metrics involve trade-offs.
2.1 Accuracy
Accuracy is the most intuitive metric, but the definition of “correct” must be explicit. For multiple-choice tasks, correctness can be determined by comparing the selected option. For open-ended responses, evaluation may rely on rule-based matching, comparison with a reference answer, or scoring by another evaluator model. Results produced using different evaluation methods are not directly comparable, so reports should clearly state how correctness was determined.
Accuracy should also be broken down into sub-dimensions, such as:
- Language understanding and instruction following: Whether the model can correctly execute instructions with multiple conditions and constraints.
- Reasoning and calculation: Whether multi-step deductions and calculations are correct.
- Tool calling: The accuracy of tool selection, parameter generation, and response parsing.
This breakdown makes small changes in the overall score easier to interpret. For example, a 0.5-percentage-point improvement in the total score may hide a substantial improvement in instruction following that was offset by unchanged or lower scores in other categories.
2.2 Response Latency
Response time affects both user experience and the overall efficiency of an agent workflow. A single task often requires multiple model calls, so small differences in per-call latency can become significant across the full execution chain.
In addition to the average, evaluations should track latency percentiles such as P50, P90, and P99. Long-tail latency often determines the timeout rate. The network environment, concurrency level, and input length should also be recorded; otherwise, results collected under different conditions will not be comparable.
2.3 Token Usage
Token usage directly affects both billing and context-window consumption. A model may achieve higher accuracy, but if each call consumes significantly more tokens, the total cost can rise in high-frequency scenarios.
Input and output tokens should be measured separately. Their prices are often different, and tool responses can substantially increase the amount of input content sent in subsequent model calls.
2.4 Cost
The estimated cost of a single call can be calculated as follows:
Cost per call = input tokens × input price + output tokens × output price
Converting the result into the cost per 1,000 calls makes model costs easier to compare. Prices should be based on the official pricing information available at the time of evaluation. They may vary by plan, volume discount, or region, so the evaluation report should include the pricing date.
2.5 Trade-Offs Between Metrics
Accuracy, latency, and cost are rarely optimized at the same time. Common trade-offs include:
- Larger models often provide higher accuracy, but also have higher latency and per-call prices.
- Shorter outputs can reduce cost and latency, but may omit necessary details.
- Additional reasoning steps may improve performance on complex tasks while increasing token usage.
Model selection should therefore not focus on maximizing a single metric. Instead, define acceptable ranges based on the business’s tolerance for errors, sensitivity to latency, and budget constraints.
3. Building a Reproducible Evaluation Set
The credibility of an evaluation depends heavily on how the test set is designed. The following practices are broadly applicable.
Reflect the real task distribution. Test cases should come from real business requests or be synthesized according to the actual distribution of tasks. A test set containing only simple questions cannot accurately represent an agent’s performance in complex workflows.
Avoid data leakage. Test samples should not appear in the model’s training data or in prompt examples. Otherwise, the measured score may be artificially high. For public datasets, check the dataset version and update date.
Use multiple difficulty levels. Dividing samples into easy, medium, and difficult categories helps identify the model’s capability boundaries. Many models perform similarly on simple tasks, while the main differences appear in difficult cases.
Record complete execution traces. For agent tasks, save every step, including the input, tool-call parameters, tool response, and model output. Trace data can be used both for metric calculation and failure analysis. It can reveal whether a failure was caused by an incorrect tool choice or an invalid parameter format.
Control randomness. When possible, fix sampling parameters and run each sample multiple times to reduce the impact of random variation.
4. How to Interpret Evaluation Results
After collecting evaluation metrics, it is important to determine what they do and do not demonstrate.
4.1 The Gap Between General Benchmarks and Business Workflows
General benchmarks measure foundational model capabilities and are useful for broad comparisons. Business workflows, however, often involve specific tools, process constraints, and different costs of failure. A model that leads a general benchmark may not be the best option for a particular tool chain.
Model selection should therefore include a business-specific test set rather than relying exclusively on public rankings.
4.2 Capabilities Not Covered by the Evaluation
Evaluation frameworks usually cover only a subset of an agent’s capabilities. For example, many general-purpose evaluations do not adequately measure complex tool-chain orchestration, the reliability of scheduled tasks, or task allocation among multiple agents.
Reports should clearly identify which dimensions were not measured. This helps readers understand the scope of the results and prevents foundational capability scores from being treated as a complete representation of agent performance.
4.3 The Statistical Meaning of Score Differences
Whether a difference of a few percentage points is meaningful depends on the sample size and variance. With a small test set, random fluctuations may be larger than the actual performance difference.
Consider calculating confidence intervals or repeating the experiments multiple times to verify the stability of the conclusions.
5. Engineering Practice: A Minimal Evaluation Script
The following example demonstrates the structure of a basic evaluation. The model-call wrapper and field names must be adapted to the specific API being used; the example focuses only on the evaluation workflow.
import timefrom dataclasses import dataclassfrom statistics import mean@dataclassclass CaseResult:correct: boollatency_s: floattotal_tokens: intdef check_answer(text: str, expected: str) -> bool:# Choose the evaluation method based on the task:# exact matching, rule-based extraction, or evaluator-model scoringreturn expected.strip() in textdef run_case(call_model, case: dict) -> CaseResult:start = time.perf_counter()resp = call_model(case["prompt"]) # The caller wraps the specific model APIlatency = time.perf_counter() - startreturn CaseResult(correct=check_answer(resp["text"], case["expected"]),latency_s=latency,total_tokens=resp["usage"]["total_tokens"], # The field name depends on the API)def summarize(results: list[CaseResult], price_per_m_tokens: float) -> dict:accuracy = sum(r.correct for r in results) / len(results)avg_latency = mean(r.latency_s for r in results)avg_tokens = mean(r.total_tokens for r in results)cost_per_1k_calls = avg_tokens * price_per_m_tokens / 1_000_000 * 1000return {"accuracy": round(accuracy * 100, 2),"avg_latency_s": round(avg_latency, 2),"avg_tokens": round(avg_tokens, 1),"cost_per_1k_calls": round(cost_per_1k_calls, 2),}
This example illustrates three key principles:
- The evaluation logic is decoupled from the model-call implementation, making it easier to change the evaluation method.
- Latency and token usage are recorded for every sample, ensuring consistent measurement.
- Cost is estimated from token usage and pricing, with the unit price passed as a parameter so it can be updated when pricing changes.
The script can be extended to group results by sub-dimension, record execution traces for failed samples, and run the same test set against multiple models in parallel.
6. A Recommended Model Selection Workflow
The methods above can be organized into a stable model selection process:
- Define the requirements. List the core task types in the business workflow and set an acceptable minimum accuracy, maximum latency, and maximum cost per call for each type.
- Build a business-specific test set. Extract samples from real logs and cover major task types as well as common failure scenarios.
- Run an initial screening. Use a smaller sample set to compare candidate models quickly and eliminate options that clearly fail to meet the requirements.
- Conduct a full evaluation. Run the complete test set against the shortlisted models, record execution traces, and calculate metrics and confidence intervals for each dimension.
- Validate with a gradual rollout. Test the selected candidates on a small portion of real traffic and monitor changes in production error rates, timeout rates, and costs.
- Establish continuous monitoring. Model versions, pricing, and business requirements can all change. Repeat evaluations regularly and retain historical records of key metrics.
The central idea is to move from “checking benchmark rankings” to making decisions based on the application’s own data. This requires some upfront effort, but it can significantly reduce the risk of rework caused by a mismatch between model capabilities and business requirements.
7. Conclusion
Evaluating AI agents is more complex than evaluating traditional question-answering systems. It requires attention to tool calling, instruction following, long execution chains, response latency, token usage, and cost.
General benchmarks are useful for broad comparisons, but they cannot replace testing designed around real business workflows. During model selection, define clear requirements, compare candidates using a reproducible test set and consistent measurement methods, and continue monitoring performance after deployment.
Only by evaluating accuracy, latency, and cost within the same framework can teams make decisions that are both reliable and economically sustainable.
