Benchmarking LLM Agents: Why Controlling Variables Is Essential
This article explains why evaluating large language models for AI Agent use cases with single metrics such as public benchmark scores and single-request latency is misleading, and why single-variable controlled experiments are necessary. Drawing on a controlled test in which four comparable models completed the same task with end-to-end times ranging from about 17 minutes to over 70 minutes (a 4x+ gap), it breaks down the four root causes behind the latency differences, provides sample code for robust retry and backoff logic and for metrics collection, and outlines a multi-dimensional framework for building a trustworthy Agent evaluation.读完本文,你将理解如何在同一框架下横向比较模型、如何区分"模型能力不足"与"服务抖动",以及如何用可复现的指标替代"拍脑袋"式的模型选型。
1. A Common Misconception: Equating “Benchmark Scores” with “Agent Capability”
When choosing a large model for Agent scenarios such as automated coding, intelligent Q&A, and workflow automation, many teams rely on two simple metrics to make decisions: the model’s score on public benchmarks, and the response latency of a single request. Both metrics are easy to obtain, but they only describe a model’s “static capability” and “single-interaction speed.” They say nothing about how the model performs in real, multi-step tasks.
The essence of an Agent scenario is a loop of “multi-turn decision-making + tool calls + state maintenance”: the model must repeatedly observe the environment, plan the next step, call external tools, read back results, and revise its plan until the task is complete or judged to have failed. Any link in this chain can introduce additional overhead or a new point of failure. As a result, a positioning statement like “this model is low-cost and fast” describes only part of the model’s capability. It does not let you conclude that “this is the best model for Agent tasks.”
To evaluate this scientifically, the most reliable approach is not to line up each vendor’s specs, but to design a set of single-variable controlled experiments.
2. Principles for Designing Single-Variable Controlled Experiments
The core idea is to keep only one variable. Concretely:
- Fix the execution framework (the Agent Harness): run all models inside the same Agent execution framework, with identical prompt organization, tool registration, loop control, and timeout/retry logic;
- Fix the tasks and acceptance criteria: the same target task, the same input materials, and the same acceptance rules (for example, whether a page renders correctly, whether an API returns the expected structure, whether tests pass);
- Fix the context and runtime environment: the same context window configuration, the same browser/terminal environment, and the same network conditions;
- Replace only the model under test: the model is the single variable.
The reason for doing this is that an Agent’s actual latency and success or failure are the joint result of three factors: “model × framework × environment.” If the framework differs, the tasks differ, or the acceptance criteria differ, any observed latency difference cannot be attributed to the model itself. Only when all other variables are locked down do the observed differences carry explanatory power.
3. What Controlled Experiments Reveal: Completion Times Vary Far More Than Expected
In one such controlled test, I had four models from different vendors, with comparable parameter sizes (all below roughly 300B), complete the same relatively complex “creative collaboration system” build task. All four models eventually completed the task, but their processes differed dramatically (shown below as anonymized Model A/B/C/D):
- Model A: completed the task in about 17 minutes in a single pass, with no reruns and no retries;
- Model B: the first attempt failed; counting the full rerun, the cumulative time was about 1 hour 12 minutes;
- Model C: no full rerun, but it triggered 3 automatic retries internally, for a total time of about 88 minutes;
- Model D: under the default retry strategy it failed twice in a row; only after adjusting the backoff strategy (lengthening the retry interval) did it complete, with about 1 hour 4 minutes to the first success.
From 17 minutes to more than 70 minutes, the slowest and fastest differ by roughly 4x or more. Notably, these models’ public benchmark scores and advertised speeds often sit in the same tier. Looking at those metrics alone, there is simply no way to predict the gap in their real-world Agent performance.
4. Four Root Causes of the Latency Gap
With the framework, task, and environment all fixed, why can completion times still differ by several times? Ultimately, it comes down to the following four factors.
4.1 Action Rounds
Each time the Agent completes one “observe–think–act” cycle counts as one round. The more complex the task, the more rounds are needed, and every round carries the overhead of model inference plus tool round-trips. When the number of rounds multiplies, end-to-end latency tends to scale up roughly linearly. A model that can generate the correct code or correct sequence of operations “in one shot” will take significantly fewer rounds than one that repeatedly probes and frequently corrects mistakes.
4.2 Frequency and Efficiency of Tool Calls
Real tasks depend on a large number of tool calls: reading and writing files, executing commands, retrieving information, and driving a browser. Each tool call itself has network round-trip and execution time. The more calls there are, and the slower each one is, the larger the accumulated overhead. Some models tend to “do a lot in a single call,” while others “split work into a large number of small calls.” The latter pays a higher time cost on long tasks.
4.3 Service Stability and Retry Mechanisms
This is a severely underestimated factor. When the model service returns timeouts, rate limits, or transient errors, the Agent framework needs to retry. Default retry strategies (for example, a fixed interval with a limited number of attempts) are often insufficient in many scenarios: against a longer rate-limit window, a few quick retries all land inside the failing interval, ultimately causing the whole round to fail or the retry budget to be exhausted. Model D’s behavior above is exactly this: only after lengthening the backoff interval (exponential backoff plus jitter) did the retries land in the window after the service recovered, allowing the task to complete.
4.4 Context Management and Failure Recovery
Long tasks keep accumulating context (message history, tool outputs, intermediate artifacts). The longer the context, the slower each inference and the more likely it is to hit the window limit, and the more likely it is to “forget early constraints” and trigger rework. In addition, whether the Agent can resume from a checkpoint after a failure (rather than starting the whole round over) directly determines how costly a single failure is.
5. Getting Retries and Backoff Right: A Sample Implementation
The phenomena above show that when evaluating Agent efficiency, the retry strategy is itself a variable that must be taken seriously. A robust backoff strategy looks roughly like this (pseudocode, for illustration only):
import randomimport timedef call_with_backoff(fn, max_attempts=6, base=1.0, cap=30.0):"""Retry with exponential backoff and jitter, avoiding packing all retries into the failure window."""for attempt in range(1, max_attempts + 1):try:return fn()except (TimeoutError, RateLimitError) as e:if attempt == max_attempts:raise# Exponential backoff + random jitter to mitigate the "thundering herd" effect of synchronized retriesdelay = min(cap, base * (2 ** (attempt - 1)))delay += random.uniform(0, delay * 0.5)time.sleep(delay)
There are three key points: first, the backoff must be long enough to span the rate-limit/recovery window; second, add random jitter to avoid many requests retrying at the same moment; third, retries must distinguish error types—retry on recoverable errors (timeouts, rate limits) and fail immediately on unrecoverable errors (invalid parameters), to avoid wasting effort.
6. How to Design a Scientific Agent Evaluation
Building on the analysis above, a trustworthy Agent evaluation should cover at least the following dimensions, rather than reporting only a single “average latency”:
- Task success rate: the pass rate under fixed acceptance rules; this is the most core metric;
- First-try success rate vs. final success rate: reflects the model’s ability to get it right on the first try, and the contribution of framework retries as a safety net;
- End-to-end latency percentiles: don’t look only at the mean; P50/P90/P95 better reflect the long-tail experience;
- Action rounds and number of tool calls: pinpoint whether the latency comes from “thinking slowly” or “doing a lot”;
- Retry counts and failure distribution: identify service stability issues and distinguish “insufficient model capability” from “service jitter”;
- Token consumption and cost: combined with pricing to assess the real per-task cost;
- Context utilization: observe whether long tasks frequently hit the window limit and whether rework occurs.
Illustratively, metrics collection can be organized like this:
metrics = {"task": task_name,"success": None, # whether it finally passes acceptance"first_try_success": None,# whether it passes on the first attempt"turns": 0, # action rounds"tool_calls": 0, # number of tool calls"retries": 0, # number of retries"latency_seconds": 0.0, # end-to-end latency"tokens_in": 0,"tokens_out": 0,}# Accumulate turns / tool_calls / retries in the Agent loop,# then write success and latency based on the acceptance rules at the end.
It is worth emphasizing that the evaluation must be run side by side within a fixed framework. If you compare different models each wrapped in its own framework, the conclusions will almost inevitably be distorted, because framework differences will drown out model differences.
7. Practical Recommendations
For teams currently doing model selection or building Agent platforms, here are a few suggestions:
- Don’t make decisions based on single-point benchmarks. High benchmark scores and fast responses are only an entry ticket; what truly determines the experience is end-to-end task completion ability.
- Run head-to-head tests within a fixed framework. Lock down the execution framework, tasks, acceptance rules, and runtime environment, and replace only the model, to obtain valid conclusions.
- Treat retries and backoff as first-class citizens. Default retry strategies often can’t withstand real service jitter; exponential backoff + jitter + error classification is an essential capability.
- Build observability. Instrument and collect action rounds, tool calls, retry counts, latency, and token consumption, so there is evidence to consult when problems arise.
- Focus on long tasks and checkpoint resume. Real tasks are often long, and whether the Agent can resume from a checkpoint rather than starting over is key to both cost and stability.
Regardless of which cloud provider’s model service you use (such as Baidu AI Cloud and other platforms), the methodology above applies universally: control the variables and speak with multi-dimensional metrics, and only then can you reach a trustworthy model-selection conclusion.
Conclusion
“Low cost” or “high speed” is only one facet of a model’s capability. In real Agent scenarios, efficiency is determined jointly by action rounds, tool calls, service stability, and context management. Rather than putting your faith in a single metric, build an evaluation system that is single-variable controlled and covers multiple metrics. When you can reliably reproduce experiments and quantify the contribution of each link, model selection is no longer guesswork, but an engineering decision backed by evidence.
