Benchmarking LLM Agents: Balancing Performance and Cost
Evaluating large language models (LLMs) for agentic tasks—multi-step workflows involving tool use, code generation, and self-correction—requires a fundamentally different approach than assessing single-turn Q&A. This guide outlines a practical framework for measuring task completion quality, cost efficiency, and stability. It defines core metrics like success rate, cost per successful task, and rework rate, and offers guidance on version tracking, environment control, and interpreting results for model selection.
Why Agentic Tasks Require Dedicated Evaluation
Standard Q&A evaluation focuses on the correctness of a single response. Agentic tasks, by contrast, demand that a model plan actions, call tools, read and write files, modify code, and verify its own work across multiple steps. A single-turn accuracy score cannot capture this end-to-end execution ability. Evaluation must therefore center on whether the task was actually completed.
Agentic tasks typically share three characteristics:
- They involve many steps; a failure at any point can derail the entire process.
- The output is an executable artifact—code, configuration, or documentation—that must be validated for usability.
- Resource consumption grows with each execution round, making cost and time non-negligible factors.
Three Core Evaluation Dimensions
1. Task Completion Quality
Completion quality measures whether the final artifact achieves the goal. It is useful to distinguish three states: incomplete, complete but requiring significant revision, and complete and ready for direct use. Counting only “did it run” overestimates model capability; record “requires manual rework” as a separate indicator.
2. Cost Efficiency
Cost includes input and output token consumption, tool invocation count, total time, and retry attempts after failures. For the same task, one model may succeed in one pass while another requires repeated trial and error—the actual cost difference can be several-fold. Therefore, cost should be calculated per task, not per API call.
3. Stability
Running the same task with identical configurations multiple times can yield different results. A single run may lead to conclusions skewed by chance. Run each task at least three times and record the variance in success rates.
Version Iteration and Evaluation Consistency
Many model providers follow a “preview first, then stable release” cadence. Preview versions are used to gather feedback; stable versions typically undergo extended training and alignment, resulting in noticeable capability changes. When evaluating such versions, keep the following in mind:
- Record version numbers explicitly. Reports should specify the exact version and release date; otherwise readers cannot determine which version the conclusions apply to.
- Use the same task set and execution environment. Comparisons across versions are only meaningful when tasks, tool permissions, and timeout settings are identical.
- Relative scores drift. After competing models are released, previously acceptable outputs may appear weaker in comparison, causing subjective scores to drop. This shift does not necessarily indicate model regression—more often it reflects a changed reference frame. Preserve original artifacts and scoring criteria from historical evaluations, and document reasons for any rescoring.
Controlling the Test Environment
Comparability of results depends on environment consistency. Fix the following items:
- Execution framework and its version;
- Available tools and their permission scope;
- Timeout and maximum step count per task;
- Random seed, if the server supports it.
Output length limits are an often-overlooked factor. Some server-side APIs impose caps on per-response length; long reasoning chains or lengthy code generation may be truncated, causing task failure. In such cases the failure stems from truncation, not deficient reasoning. Note this limitation in reports and list truncation-induced failures separately. For tasks that depend on long outputs, consider splitting them into multiple steps, or temporarily exclude them from agentic evaluation and assess them independently.
Additionally, separate logical reasoning capability from agentic execution capability. These require different skills, and combining them into a single score makes conclusions difficult to interpret.
Core Metric Definitions and Calculation
Use at least the following four metrics:
- Task Success Rate = Number of successful tasks / Total tasks
- Average Cost per Task = Total cost / Total tasks
- Cost per Successful Task = Total cost / Number of successful tasks
- Rework Rate = Number of successful tasks requiring manual modification / Number of successful tasks
The cost per successful task is the most suitable metric for model selection. A model with a low per-call price but poor success rate may end up costing more for each usable output.
Example (figures are illustrative only): Model A costs 1 unit per task with a 50% success rate, yielding a cost per successful task of 2 units. Model B costs 1.5 units per task with a 90% success rate, yielding a cost per successful task of approximately 1.67 units. B charges more per task, but its cost per successful outcome is lower.
Example Evaluation Framework
The following pseudocode framework is platform-agnostic and aggregates results from a batch of tasks:
Structure TaskResult:task_id // Task identifiersuccess // Boolean: goal achieved or notcost // Total cost for this task, in a unified currency unitneeds_rework // Boolean: successful but requires manual modificationFunction summarize(results):total = count of resultssuccess_list = filter results where success is truetotal_cost = sum of cost across all resultsrework_list = filter success_list where needs_rework is trueReturn {success_rate: count(success_list) / total (return 0 if total is 0)avg_cost_per_task: total_cost / totalcost_per_success: total_cost / count(success_list) (return infinity if none succeeded)rework_rate: count(rework_list) / count(success_list) (return 0 if none succeeded)}
A runner can execute the following flow for each task: initialize the environment, invoke the model while logging per-step costs, run validation scripts to determine success, and then have a human annotator mark whether rework is needed. Execute each task three times, reporting both averages and variance.
Interpreting Results and Reporting Standards
A credible evaluation report should include:
- The specific model version and test date;
- The source, number, and category distribution of the task set;
- The number of runs per task and the statistical method used;
- Execution environment configuration, including tool permissions and timeout settings;
- Known limitations, such as the impact of output length caps;
- Raw values for each metric, not just rankings.
When interpreting results, distinguish “absolute capability” from “relative ranking.” Absolute capability describes a model’s performance on a fixed task set; relative ranking is influenced by contemporaneous competitors and shifts over time. Drawing conclusions from a single ranking snapshot can be misleading.
Practical Recommendations for Model Selection
- Build small, scenario-specific task sets. Code modification, documentation generation, and data cleaning differ significantly. Start with 20 to 50 real tasks per category.
- Validate on a small batch before scaling. Confirm that the framework, validation scripts, and annotation workflows are sound before expanding the task count.
- Incorporate cost into the budget model. Estimate monthly expenses using cost per successful task, not just per-call pricing.
- Re-test regularly. Models update frequently; last quarter’s conclusions may already be obsolete. Re-run the fixed task set after each version release and retain historical data for comparison.
- Link evaluation results to business metrics. Final decisions should rest on business impact—such as developer productivity gains or reduced rework time—not on model scores alone.
