Comparing AI Agent Efficiency Across Four Models Under a Unified Harness
Agent performance depends as much on the execution framework (the Harness) as on the model itself. In this study, the harness is held constant and only the model is swapped: three mainstream lightweight Flash-class models (each on the order of 300B parameters or below) and one ultra-long-context general model are run against the same creative-collaboration build task, using identical materials, prompts, browser environment, and acceptance rules. All four models eventually complete the task, but total wall-clock time varies by roughly 5x. The differences are driven less by raw capability than by decision stability, tool-call hit rate, service stability, retry strategy, context management, and acceptance feedback. The article breaks down why the process differs, offers a concrete retry-and-budget wrapper for agent harnesses, and explains how to measure and select models using end-to-end metrics rather than single-point benchmarks.
1. Background and Experimental Design
1.1 From “Swapping Frameworks” to “Swapping Models”
In agent applications, “who does the work” matters just as much as “how the work gets done,” and these correspond to two layers: the model and the Harness (the agent execution framework). The Harness drives the model through the “reason-act-observe” loop, manages tools such as the browser, file system, and terminal, controls retries and rollbacks, and validates the output.
In the previous round, the model was fixed and four different harnesses were swapped. The fastest harness completed the most complex task among them, building a creative-collaboration system, in 16 minutes 55 seconds. That result showed that even with an unchanged model, differences in the engineering shell alone can open up a several-fold gap.
This round reverses the variable: all harnesses are unified and only the model is swapped. The candidates include three mainstream lightweight Flash-class models (all on the order of 300B parameters or below) and one ultra-long-context general model (as a cross-reference). The former emphasize speed or cost; the latter emphasizes how much context a single pass can carry. The four models come from different vendors and happen to cover several typical positioning profiles.
1.2 Controlled Variables
| Item | This Round’s Setting |
|---|---|
| Execution framework (Harness) | Unified to the same framework |
| Task | Build the same creative-collaboration system with multiple pages and interactions |
| Materials | Same set of design mockups, copy, and image assets |
| Context | Same system prompt, requirements document, and tool descriptions |
| Browser environment | Same runtime, used for page validation and screenshots |
| Acceptance rules | Same set of automated checks and manual review checklist |
| Sole variable | The model |
1.3 Metric Definitions
Evaluating an agent can’t rely only on “does the final screenshot look right.” You need quantifiable, reproducible metrics:
- Total wall-clock time: The real elapsed time from task dispatch to the first acceptance pass, including waiting and retries.
- Manual interventions: The number of times a human had to edit prompts, adjust strategy, or manually fall back and re-trigger. This is a hidden cost that’s often overlooked.
- Full-run restarts: The number of times the task restarted from scratch, losing all prior state.
- Automatic retries: The number of times a single call or single-step action was automatically resent under service errors, tool failures, and similar conditions.
- Action rounds and tool calls: The number of loops that complete a decision and execute an action, plus the total number of tool invocations for browsing, reading/writing files, running commands, and so on.
- First-pass success rate: The proportion of attempts that pass acceptance on the first try.
2. Results: The Process Matters More Than the Final Screenshot
All four models eventually completed the task; the differences lie in “how they got there”:
| Model | Total time (incl. retries) | Full-run restarts | Auto-retry behavior | Manual interventions |
|---|---|---|---|---|
| Model A: a vision-enhanced Flash model | ~17 min | 0 | No obvious retries | Not recorded |
| Model B: a general-purpose Flash model | ~1 hr 12 min | 1 (first attempt failed) | Not recorded | Not recorded |
| Model C: an open-weight Flash model | ~88 min | 0 | ~3, handled automatically by the framework | Not recorded |
| Model D: an ultra-long-context general model | ~1 hr 04 min after first success | 0 | ~85; two consecutive failures under the default policy, completed after adjusting backoff | At least 1 (adjusted retry strategy) |
A few notes:
- The times in the table are observed values from a single run by the author; each model ran only once, and results are affected by server load, time of day, and the specific task, so they shouldn’t be treated as stable benchmarks.
- “Failure” here means the task as a whole didn’t pass acceptance; individual tool-call failures, retries, or local rollbacks are not counted.
- “Retry” needs to be distinguished by layer: Model C’s retries happen within a single call, with session state basically preserved; Model D’s retries ultimately manifested as the whole task starting over.
The time curves of the four models can be summarized into four typical patterns:
- Clean run: Decisions are direct, tool-call hit rate is high, and it takes the least time.
- Full rework: The first round went off track and failed acceptance, forcing a complete restart.
- Local repair: Repeated retries and fixes on top of the existing state, with low per-step cost.
- Cascading amplification: Default retry parameters can’t handle the anomalies, so after adjusting the strategy the retry count explodes and nearly all the time is spent waiting.
The last model is especially typical. Under the default retry policy it failed two tasks in a row, and each single failure amounted to scrapping a full attempt. Only after adjusting the backoff strategy did it succeed, accumulating roughly 85 retries, which is equivalent to the time it takes to execute the task logic once being diluted dozens of times over by failed retries.
These results show that positioning like “low-cost” or “high-speed” only describes part of a model’s capability. Real agent efficiency is a product:
Single-step capability × decision stability × tool-call success rate × service stability × retry strategy × context management
If any single factor becomes a weak link, the efficiency of the entire chain is dragged down.
3. Breaking Down the Causes of Process Differences
3.1 Decision Quality: Going the Right Direction Beats Going Fast
Completing a task with an agent is essentially a serial “reason-act-observe” loop, where each step’s decision depends on the results of all previous steps. A single directional error, like choosing the wrong technical path or misreading the requirements, gets amplified in subsequent steps.
- Model A ran straight through on the correct path, with few decision steps and zero rework.
- Model B failed on the first attempt; the most likely cause is that the overall approach or a key implementation step went off track, and the cost of starting over is far higher than a local fix.
- Model C didn’t restart at the task level but corrected deviations through internal retries, which shows its local error-correction ability is decent, but each correction consumes extra action rounds.
So the premise that “Flash-class lightweight models are faster” holds only if they’re also on the correct path. Otherwise, the speed advantage is completely offset by rework costs.
3.2 Tool Calls: Hit Rate Is a Hidden Cost
Take browser operations as an example: element-location failures, pages not rendering in time, and abnormal API responses all trigger repeated operations and extra observations:
- Each invalid call consumes a round of reasoning and a tool round-trip, and may also write error information into the context, interfering with later decisions.
- Model A may have hit the clickable target directly, aided by the prompt or tool descriptions.
- A good portion of Model C’s internal retries likely come from this kind of local failure.
The success rate of tool calls often explains the differences in end-to-end time better than a model’s base reasoning score.
3.3 Service Stability: The Retry Strategy Determines Whether Anomalies Get Absorbed
The comparison between Model C and Model D is telling. Both encountered anomalies, but one was absorbed by the framework and the other evolved into a full-run failure. There are three common retry strategies:
| Retry location | Retry granularity | Cost | Applicable scenario |
|---|---|---|---|
| Framework-internal auto-retry | A single model call | Low; session state basically preserved | Network jitter, occasional timeouts |
| In-framework local rollback | A single step or subtask | Medium; some intermediate artifacts must be rebuilt | Tool-call failures, single-step decision errors |
| Full-run restart | The entire task | High; all prior results scrapped | Context pollution, acceptance failure that can’t be fixed locally |
Default backoff parameters are often too aggressive: fixed intervals, no upper bound, no distinction between error types. Once the server keeps rate-limiting or timing out, retries pile up rapidly in a short window. More reasonable approaches include:
- Use exponential backoff with random jitter to avoid a large batch of tasks retrying at the same instant;
- Distinguish retryable errors (like rate-limiting, timeouts) from non-retryable ones (like invalid parameters, content-safety blocks);
- Set a per-step retry limit, a task-level time budget, and a total retry budget, and fall back once any is exceeded.
3.4 Context Management: Long Doesn’t Mean Useful
Context is the agent’s “working memory.” Repeatedly read page content, long tool outputs, and historical error logs quickly fill up the window, causing three problems:
- Early key constraints get diluted and the model forgets the original requirements;
- The longer the context, the more input tokens per call, so responses get slower and costlier;
- Once compression or summarization drops details, the model re-explores problems that were already solved.
Ultra-long-context models can theoretically retain more history, but the premise is that the framework effectively organizes and trims information rather than stuffing everything into the window. If there’s no layered memory (like a summary layer plus a retrieval layer), even a huge window can’t buy efficiency.
3.5 Self-Check Capability: Who Catches “It’s Wrong”
- Model A may have a better self-verification instinct, completing a self-check before delivery and reducing rework at the acceptance stage.
- Model B’s problem may be prematurely declaring “done” when the actual output didn’t meet acceptance criteria.
Whether an agent proactively verifies its own output before submitting, such as running check scripts or comparing screenshots, directly affects the first-pass success rate. The more specific the framework’s acceptance feedback, down to the page, element, and expected difference, the stronger the direction for the model’s corrections.
4. Actionable Recommendations for the Execution Framework
import randomimport timefrom dataclasses import dataclassRETRYABLE_STATUS = {429, 500, 502, 503, 504} # Rate-limiting and server errors; retryableFATAL_STATUS = {400, 401, 403} # Parameter or auth issues; retrying is pointless@dataclassclass Budget:max_turns: int = 60 # Max action rounds allowed for a single taskmax_attempts: int = 3 # Max attempts for a single callmax_wall_clock: float = 1800.0 # Task-level total time limit (seconds)def call_with_retry(fn, budget: Budget, trace: list):"""Call wrapper with backoff and budget control: elapsed time and retries fully traced."""attempt = 0while True:attempt += 1try:result = fn()trace.append({"stage": "model_call", "attempt": attempt, "status": "ok"})return resultexcept ServiceError as exc:# Non-retryable error: raise immediately to avoid pointless waitingif exc.status_code in FATAL_STATUS:raiseif attempt >= budget.max_attempts:raise BudgetExhausted("model_call", attempt) from exc# Exponential backoff + full jitter: base 1s, cap 30s, plus random offsetdelay = min(30.0, 2 ** (attempt - 1)) * random.uniform(0.5, 1.0)trace.append({"stage": "model_call", "attempt": attempt,"status": "retry", "delay": round(delay, 2)})time.sleep(delay)def run_task(agent, task, verify, budget: Budget):"""A full-run restart is the last resort: trigger only on acceptance failure or budget overrun."""start, rounds = time.monotonic(), 0while True:rounds += 1state, trace = agent.new_session(task), []try:while len(state.history) < budget.max_turns:if time.monotonic() - start > budget.max_wall_clock:raise BudgetExhausted("wall_clock", rounds)action = call_with_retry(lambda: agent.decide(state), budget, trace)observation = agent.act(action) # Browser / file / terminal tool calls, etc.state.append(action, observation)if agent.done(action) and verify(observation):return {"success": True, "rounds": rounds,"elapsed": round(time.monotonic() - start, 1), "trace": trace}# Exhausting the turn count is treated as this round's failure; move to the next roundexcept BudgetExhausted as exc:if rounds >= budget.max_rounds:return {"success": False, "rounds": rounds, "reason": str(exc)}
Four key points of this wrapper:
- Distinguish error types first, then decide whether to retry, avoiding pointless waiting.
- Exponential backoff with jitter to prevent retry storms.
- Triple budget (turns, attempts, wall-clock) as a fallback, ensuring the task never loops infinitely.
- Trace every step, so afterward you can pinpoint exactly where the time went.
5. How to Measure and Select Correctly
5.1 Replace Single-Point Metrics with End-to-End Metrics
Faced with “which model should I choose,” don’t just look at single-point benchmark scores or unit price. Instead, run end-to-end evaluation on your own real task set, focusing on four metrics:
- First-pass success rate: Reflects the ability to get it right on the first try; it’s the baseline for efficiency.
- Total wall-clock time (mean and P90): P90 exposes long-tail problems better than the mean, such as occasional long retries.
- Average token consumption per successful task: Combining input, output, and retry consumption, multiply by unit price to estimate the real cost.
- Manual intervention rate: Directly determines the human effort required after scaling to production.
At the same time, run each model at least 5 to 10 times, with fixed service windows and parameters, to reduce the interference of occasional fluctuations.
5.2 Tiered Selection by Scenario
- Scenarios that prioritize speed with clearly bounded tasks (like single-page generation, style tweaks): Prefer Flash-class lightweight models, paired with strict acceptance rules so that if they go off track, someone catches it.
- Complex multi-step tasks: Overall planning and self-check ability matter more; single-decision quality is more critical than response speed.
- Tasks requiring comprehension across many files or long documents: Ultra-long-context models have an advantage, but must be paired with layered memory and context trimming.
- Cost-sensitive batch tasks: Use “total cost per successful task” rather than unit price as the decision basis.
When possible, a hybrid strategy can also work: use a model with stronger reasoning ability for the planning phase, a lightweight model for the execution phase, and let the framework make the final call at the acceptance phase.
5.3 Treat Stability as a Selection Dimension
The same model can perform very differently on retries under different time windows and loads. It’s advisable to continuously record rate-limit rate, timeout rate, and P90 latency, and factor them into selection scoring. If your orchestration and evaluation platform needs baseline capabilities such as containerized deployment, object storage to save run traces, log search, and monitoring/alerting, you can use general cloud products from cloud providers such as Baidu AI Cloud to build them.
6. Summary
This experiment narrowed the variable down to a single dimension, the model, yet the conclusion goes beyond a simple “who’s stronger” ranking: all four models completed the task, but their total times differed by about 5x, and not a single model ran clean the whole way through.
What truly determines agent efficiency is this chain:
Decision stability, tool-call success rate, service stability, retry strategy, context management, and the quality of acceptance feedback
For developers, selection is just the starting point. Getting retries right, managing context well, and setting acceptance rules precisely often bring more direct gains than switching to any model.
