Cross-Model AI Agent Benchmark: The Gap Is More Than Just Speed
A hands-on, cross-model benchmark of AI agents shows that model speed alone does not determine how fast an agent completes a real task. With the harness, task, context, and acceptance rules held constant and only the underlying model swapped, end-to-end runtime ranged from 17 minutes to 88 minutes — roughly a 5x gap — even though all four models eventually finished. This article breaks down the four failure patterns behind that spread, the observable variables that drive it (action turns, tool calls, service stability, and context management), and how to implement retry strategies with backoff, jitter, and budget control. It closes with production-oriented practices for layered routing, staged retries, observability, automated acceptance, and task-level cost control.
1. Why “Model Speed” Doesn’t Mean “Agent Speed”
In real agent deployments, four issues are tangled together: model capability, tool calling, service stability, and the scheduling framework. Asking only “which model is smarter” or “which model is faster” can’t answer the real question: how long does the same task actually take to finish?
Two sets of experiments from recent practice illustrate this:
- Experiment 1: Fixed the model, swapped the execution framework. The same model ran the most complex creative-collaboration system task across different agent execution harnesses (commonly called “harness” in the industry). The fastest version finished in about 17 minutes.
- Experiment 2: Fixed the harness, swapped the model. The task, materials, context, browser environment, and acceptance rules were all identical; only the underlying model changed. The results:
- All four models eventually completed the task;
- End-to-end time ranged from 17 minutes to 88 minutes — roughly a 5x difference;
- Failure paths differed sharply: some succeeded on the first try, some needed a full rerun, some recovered through the framework’s built-in retry, and some only finished after adjusting retry parameters.
Agent execution efficiency is therefore not a single-dimension comparison. It is the combined capability along an entire chain:
Agent efficiency ≈ model capability × tool use × framework strategy × service stability
If any link fails, total runtime blows up quickly. For teams that want to “get real development work done with AI at low cost,” “task completion rate” and “number of human interventions” matter more than “per-token price.” These two metrics directly determine the total cost.
2. Experiment Design: Allowing Only One Variable
To make cross-model conclusions credible, every other variable must be controlled.
| Category | Details |
|---|---|
| Held constant | Harness (unified), task and prompts, reference materials, context length, browser environment, acceptance rules |
| Sole variable | Underlying model |
| Comparison samples | Four models under 100B parameters, positioned as lightweight and efficient, plus one long-context version as a control |
Models at this parameter scale are usually positioned as “low cost” and “high throughput,” and are among the most common choices when enterprises process tasks in bulk. But “small parameter count” doesn’t equal “short task time.” The comparison of end-to-end time and failure-recovery behavior across the four models (times normalized by the rules in the table):
| Model | Default retry behavior | Full rerun? | Completed? | Normalized time |
|---|---|---|---|---|
| Model A | Passed first time | No | Yes | 17 min |
| Model B | Failed first time, succeeded on rerun | Yes | Yes | 1 h 12 min |
| Model C | Succeeded after 3 internal auto-retries | No | Yes | 88 min |
| Model D | Failed twice in a row, succeeded after extended backoff | Yes | Yes | 1 h 04 min |
Note: The long-context control model’s time fell between Model B and Model C and is not listed separately in the table.
A “long runtime” doesn’t necessarily mean “poor capability.” What really matters is where the failure happened, whether the framework could recover automatically, and how much that recovery cost.
3. Four Kinds of Failure: Where the 17-to-88-Minute Gap Comes From
3.1 First-Try Success: Using the “Smooth Path” as a Baseline
Model A passed all acceptance rules on the first try, finishing in 17 minutes total. This smooth path reflects how well “model capability” matches “task difficulty”: when task boundaries are clear and materials are complete, even a lightweight model can produce acceptable results quickly.
3.2 Full Rerun: The Most Expensive Kind of Failure
Model B failed on its first run, then completed on a rerun for a total of 1 hour 12 minutes. The problem lies in the “full rerun”: the framework defaults to defining “failure” as “the whole task failed” and then starts over from the beginning. This means:
- Files that were already generated must be regenerated;
- Steps that already passed must run again;
- Call costs that were already paid must be paid again.
If one run takes 20–30 minutes, a single rerun is enough to double the time. Retry granularity determines cost. The right approach is to retry by step or stage, not by the whole task.
3.3 Internal Auto-Retry: Long Time but a Complete Success Path
Model C did not trigger a full rerun. It relied on the framework’s built-in 3 auto-retries and finished in 88 minutes total. The upside is that the success path stays continuous, with no situation where “the first half’s work is entirely voided”; the downside is that every retry costs waiting time. This shows:
- A retry mechanism can turn “occasional failure” into “slow but successful”;
- But without a backoff time cap, tail latency keeps accumulating.
3.4 Default Retry Isn’t Enough — Extended Backoff Is Needed
Model D’s default retry strategy failed twice in a row, and it only completed after the backoff parameters were adjusted, at a normalized time of 1 hour 04 minutes. It reveals a common misconception: the default retry strategy is not a cure-all. Default parameters are usually aimed at “brief transient faults” and are powerless against sustained service jitter or rate limiting.
So in production, the retry strategy needs to be a configurable option rather than a hardcoded magic number.
4. Four Observable Variables: Why the Same Task Takes 5x Longer
4.1 Action Turns
For the same task, some models finish in 10 turns, others in 30 or more. Each extra turn means one model inference plus one tool call. Double the turns and both cost and latency roughly double.
A sensible approach is to set a maximum turn limit and, when it is exceeded, have the model output “current status and a next-step suggestion” rather than keep spinning.
4.2 Tool Calls and Multimodal Consumption
Modern agent tasks often involve browser operations, file read/write, screenshot understanding, and more. The number of tool calls and their failure rate directly determine whether the task can progress smoothly:
- Broken browser selectors, page-load timeouts, and screenshot-recognition errors all trigger extra retries;
- Multimodal inputs (screenshots, design drafts) make each call slower and more expensive;
- Long-context models may be more stable when there is “a lot of material,” but “each inference is slower,” so total time isn’t necessarily shorter.
4.3 Service Stability
The same model called at different times can have completely different failure rates. Rate limiting, timeouts, and gateway errors can turn “17 minutes” into “80 minutes.” These problems can’t be solved by “switching to a stronger model”; they can only be covered by engineering means.
4.4 Context Management
Context that is too long makes the model “forget” early constraints; context that is too short makes it repeatedly read materials. A reasonable approach includes:
- Put the task description and acceptance criteria at the front and mark them explicitly;
- Retrieve reference materials on demand rather than stuffing them all in at once;
- Produce staged summaries during long tasks to control context growth.
5. Implementing Retry Strategy in Engineering
The following is example retry code with budget control. The core idea is “bounded, backed off, and differentiated”:
import random, timeRETRYABLE = {429, 500, 502, 503, 504} # retryable errorsdef call_with_retry(fn, *, budget=6, base=2.0, cap=30.0):for attempt in range(budget):try:return fn()except ApiError as e:if e.status not in RETRYABLE or attempt == budget - 1:raise # non-retryable or budget exhausted; raise directly# exponential backoff + jitter to avoid a retry stormsleep = min(base * (2 ** attempt), cap)time.sleep(sleep / 2 + random.random() * sleep / 2)
Key points:
- Differentiate error types: 4xx client errors generally shouldn’t be retried; 5xx errors and 429 are the retry targets;
- Backoff cap: avoid long waits dragging down the whole task;
- Jitter: prevent multiple requests from retrying at the same time and causing an “avalanche”;
- Budget control: fold the number of retries into the task’s total budget rather than retrying indefinitely.
6. Practices for Production Environments
6.1 Layered Routing: Lightweight Models for Simple Tasks, Strong Models for Complex Ones
Not every task needs the strongest model. Tasks can be split into “well-defined” ones (like format conversion and simple lookups) and “exploratory” ones (like solution design and complex debugging), each routed to a different model to balance cost and quality.
6.2 Retry by Stage, Not by Task
Break the task into independently verifiable stages (e.g., “requirement parsing → solution generation → code implementation → test acceptance”) and retry each stage separately. This way a failure only affects the current stage and doesn’t waste completed work from earlier stages.
6.3 Build Observability: Record Every Turn and Every Call
At minimum, record: model name, start/end time, action turns, tool-call count, error codes, and retry count. With this data, you can actually answer “why was this run slower than the last.”
6.4 Automate the Acceptance Rules
Manually judging “whether it’s done” is both slow and subjective. Acceptance criteria should be written as scripts (test cases, page assertions, file checks) so the agent can also see the acceptance results, reducing wasted rework.
6.5 Cost Control Should Be at the “Task” Level, Not the “Token” Level
A low per-token price doesn’t mean a cheap task. The total cost of one task = model call cost + tool call cost + human intervention cost. Retries and rework are often the biggest share.
7. Conclusion
The conclusion from this set of comparison experiments is plain: positioning a model as “low cost” or “high speed” only describes part of its capability. Real agent efficiency depends on the combined ability of action turns, tool calls, service stability, and context management.
For engineering teams, choosing a model is only the first step. More importantly, you need to build the “benchmark → route → retry → accept → observe” chain. Only then can you find the balance between cost, speed, and reliability that truly fits your business scenario.
