Evaluating Agent Task Efficiency: Switching Models Isn't the Same as Switching Results
This article reports a controlled experiment that fixes the execution framework (harness), task, and acceptance criteria while swapping in four different models, and shows that the model alone does not decide Agent efficiency. Through process data such as action rounds, tool-call behavior, retry handling, and time breakdown, it explains four engineering variables that shape end-to-end performance—action-round count and effective context use, tool-call quality and self-correction, service stability and retry backoff, and context management—then offers a metrics framework, practices for observable and regression-ready evaluation, common pitfalls, and practical selection guidance. Readers will learn how to measure and improve Agent efficiency by "task validated" rather than by per-call price or final screenshots.
1. Background: A Shift in Perspective—What Hidden Variables Lie Beyond the Model
When selecting an Agent application, the assumption that “a stronger model completes tasks faster and cheaper” seems self-evident. It holds up reasonably well for a single Q&A exchange, but it breaks down once the task becomes real engineering work that requires dozens of tool calls, edits across multiple files, and a run-through to verify results.
In an earlier set of experiments, we held the model constant and swapped only the execution framework (the harness), running different frameworks on the same task of building a creative collaboration system. The best-performing framework finished the system in 16 minutes and 55 seconds. In other words, the framework alone can produce a clear efficiency difference.
This round reverses the variables: the framework is fixed, and only the model changes. The models were chosen for three reasons:
- All are lightweight models in the under-300B-parameter range. One targets low-cost, low-latency scenarios, and another has visual multimodal capability—covering two typical product positions.
- We also picked a long-context model as a reference, to see whether a difference in context window can offset a disadvantage in scale.
- Lightweight models are called more frequently, so latency and stability problems are amplified, and weaknesses in retry strategies are easier to expose.
The models under test:
- Model A: a lightweight model positioned for low cost and high response speed
- Models B and C: general-purpose lightweight models with similar parameter counts, where C has visual multimodal capability
- Model D: a long-context reference model
To be clear, this is a single-sample controlled experiment. It does not answer “which model is stronger,” but rather “under the same framework and the same acceptance criteria, by what path and in how much time does each model finish the task.”
2. Experiment Design: Locking a Single Variable and Unifying Acceptance
To keep results comparable, the experiment followed the single-variable principle: the execution framework, task goals, materials and requirements description, initial context, browser tooling and permissions, and acceptance criteria were all fixed, and only one model changed each time. The framework stayed as close to default configuration as possible; we deliberately did not bring model-specific prompt engineering, tool customization, or retry-parameter tuning into the experiment, because those are themselves variables we want to observe.
| Variable | Handling this round | Under observation |
|---|---|---|
| Execution framework | All four runs use the same framework | No (fixed) |
| Task and materials | The same creative collaboration system prototype, including a front-end interface, data structures, and sample data | No (fixed) |
| Initial context | The same requirements description and design constraints | No (fixed) |
| Browser and runtime | The same browser tooling and runtime environment | No (fixed) |
| Acceptance rules | The same acceptance checklist, automated checks plus manual spot checks | No (fixed) |
| Underlying model | One of four | Yes (the only variable) |
Beyond the framework, we also tracked three kinds of “process data.” They determine the final result, yet are often ignored by evaluations that only look at screenshots:
- Action rounds: the number of iterations in which the model plans, calls tools, reads results, and revises its approach.
- Tool-call behavior: number of calls, failure types, and how often the same error repeats.
- Time composition: per-round time, retry time, and time spent re-running an entire round after a full-round failure.
For acceptance, once the first run finishes (whether it succeeds or fails), a unified acceptance check runs. Passing it is recorded as “first-time success.” Failed models may re-run or continue through the retry mechanism until acceptance passes, and we record “cumulative success” along with its time composition. A single result only describes that run’s behavior and is not treated as a product-level conclusion.
3. Process Observation: Four Models, Four Paths to Completion
| Model | Full-round re-run | Auto-retry behavior | First-time success | Outcome |
|---|---|---|---|---|
| A | No | No full-round re-runs recorded | 17 min | Complete |
| B | Yes; first round failed acceptance | No auto-retry recorded | 1 hr 12 min (incl. re-run) | Complete |
| C | No | 3 times | 88 min | Complete |
| D | No | 2 consecutive failures under the default policy; succeeded after extended backoff | 1 hr 04 min | Complete |
All four models ultimately delivered a verifiable result, but the process differences are far larger than what the final screenshots suggest:
- Model A was the fastest, passing in 17 minutes on a single run, with no full-round re-run worth recording.
- Model B failed acceptance on the first round and completed only after one re-run, for a cumulative 1 hour 12 minutes—the longest of the four.
- Model C did not trigger a full-round re-run, but the framework’s internal auto-retry mechanism fired 3 times, for a cumulative 88 minutes.
- Under the framework’s default retry policy, Model D failed twice in a row and the task was interrupted. After extending the retry backoff strategy, subsequent calls succeeded, reaching first-time success at a cumulative 1 hour 04 minutes.
As a rough estimate, A’s 17 minutes corresponds to roughly 10–20 action rounds, while C’s 88 minutes is about 5 times A’s. If per-round time did not increase manyfold, the gap likely comes from two sources: repeated probing on the same error, and the extra time spent waiting for retries. The situations of B and D further show that failure-recovery ability translates directly into end-to-end time.
The most revealing cases are C and D:
- C’s 3 auto-retries are a “hidden cost” of the framework. There was no full-round re-run and the result eventually passed acceptance, but latency was significantly stretched.
- D’s problem lay in the retry strategy itself: the default backoff parameters, when facing minute-long throttling or service jitter, placed every retry inside the failure window, so the retry budget was quickly exhausted.
This is precisely the point: the model is a necessary condition for completing the task, but the retry strategy and service stability together determine whether the task finishes on time.
4. Breakdown: Four Engineering Variables That Decide Agent Efficiency
4.1 Action Rounds and Effective Context Use
Executing a task in an Agent is essentially iterating in a loop of “model reasoning → tool call → result feedback.” End-to-end time can be approximated as:
end-to-end time ≈ Σ(per-round reasoning time + per-round tool execution time) + retry wait and full-round re-run time
The more rounds there are, the longer the context carried into each round: files already read, solutions already tried, and terminal output all keep accumulating, and input tokens grow roughly quadratically with the number of rounds. So the round count both linearly amplifies latency and non-linearly amplifies call cost.
Cutting rounds cannot be done by compressing output; it has to come from acquiring information more efficiently:
- In the exploration phase, use directory search or retrieval-style tools to locate relevant files first, then read them on demand, instead of scrolling through files one by one.
- Before launching the application, clarify the launch method, port conventions, and dependencies in one go to reduce trial and error.
- Chain “read → analyze → modify → verify” into a reusable checklist to avoid omissions and rework.
- Advance independent subtasks (such as implementing multiple modules separately) in parallel to shorten the critical path.
Illustration:
Inefficient path: guess → try → error → guess again …… (round count keeps rising)Efficient path: locate via retrieval → precise read → one-shot change → automated acceptance
4.2 Tool-Call Quality and Self-Correction
In real engineering, errors are nearly unavoidable—syntax mistakes, failed module imports, an occupied port, dependency conflicts, and so on. What separates good models from weak ones is often not “whether they make mistakes” but whether they can locate and fix a mistake within one or two rounds.
Self-correction ability can be evaluated from three angles:
- Localization accuracy: whether it can find the real cause from the error message rather than only reading the surface hint.
- Correction effectiveness: whether it re-runs verification after a fix instead of simply declaring completion.
- Convergence speed: whether the same class of error keeps recurring, and whether the number of occurrences drops each round.
In evaluation, you can count “the number of times the same error recurs” and “the rounds needed from failure to recovery.” For example, when a port is occupied:
- A strong model directly adjusts the launch parameters or terminates the occupying process, then re-verifies that the service is available;
- A weak model may restart repeatedly while always using the same command.
4.3 Service Stability and Retry Backoff Strategy
Long tasks usually run from tens of minutes to several hours, with high call volume and a continuously growing context. Encountering throttling, timeouts, and transient server-side errors is almost inevitable. The retry mechanism decides whether the Agent “waits a moment and keeps going” or “grinds to a halt.”
Fixed-interval retries look simple, but during a failure window that lasts several minutes they pack every retry into the same time span and wastefully exhaust the budget. A more robust engineering approach is exponential backoff plus random jitter: each wait grows by a factor, and a random perturbation is added, so many tasks do not all retry at the same instant.
Illustrative pseudocode:
BASE, CAP, MAX_ATTEMPTS, JITTER = 2, 60, 8, 0.2 # units: secondsdef call_with_retry(request, is_retryable):for attempt in range(MAX_ATTEMPTS):try:return invoke(request)except TransientError as err:if not is_retryable(err) or attempt == MAX_ATTEMPTS - 1:raisedelay = min(BASE * (2 ** attempt), CAP)sleep(delay * (1 + random.uniform(-JITTER, JITTER)))
This sample code has three key points:
- Retryability judgment: retry only transient errors such as throttling and timeouts; deterministic errors like bad parameters or permission problems should fail immediately to avoid wasted attempts.
- Backoff cap: put an upper bound on a single wait to avoid backoff periods that drag the task out.
- Random jitter: spread out retry times to reduce pressure on the server.
Configuration should also distinguish scenarios. Human-in-the-loop interaction, where a single round is measured in seconds, is not suited to overly long backoff; long tasks that run for tens of minutes can tolerate a longer wait window. In long-task scenarios, the default parameters are often not the optimal configuration.
Three points to keep in mind for the engineering implementation:
- For long tasks, save intermediate state at breakpoints so a failure can resume rather than restart the whole round.
- Retry parameters should be centrally managed and configured per scenario—for example, maintaining a backoff configuration split by task type.
- Log every retry as an independent event rather than hiding it behind a “success” result.
4.4 Context Management and Information Compression
The more complex the task, the faster the context expands. Here is an easily overlooked detail: how large the context window nominally is does not equal how much of it the model can use effectively. Early research on the “lost in the middle” phenomenon shows that when key information sits in the middle of the context, the model tends to use it less effectively than information at the beginning or end. So “fitting it in” and “being able to use it” are two different things.
In practice:
- Before the task starts, fix key information such as requirement constraints and acceptance criteria in the system prompt or task description so they always sit at the head of the context.
- During execution, produce rolling summaries of historical logs that keep three categories of information: “completed changes, pending problems, and key errors.”
- Store larger artifacts such as logs and screenshots in object storage, keeping only the summary and reference addresses in the context.
- At the end of the task, generate a delivery summary to ease manual review and later iteration.
In addition, some platforms offer a prefix-caching mechanism that lets stable system prompts and task descriptions be reused across multiple calls. This both reduces the latency of repeated computation and lowers the cumulative cost of long tasks. The specific capability depends on the platform actually being integrated, so it should be verified separately during selection.
5. A Metrics Framework: Defining “Efficiency” with Data
Comparing only final screenshots easily leads to one-sided conclusions. It is worth building a metrics framework with at least the following dimensions:
| Metric | Definition | Problem it reflects |
|---|---|---|
| First-time success rate | Share of tasks that pass acceptance on the first run | Ability to get it right the first time |
| End-to-end time | Total time from task dispatch to passing acceptance, including retries and re-runs | The user’s actual wait time |
| Action rounds | The number of model–tool iterations needed to complete the task | Execution efficiency and cost structure |
| Tool failure rate | Failed calls / total calls | Environment and call quality |
| Correction convergence rounds | The rounds needed from the first error to a passing fix | Self-repair ability |
| Retry count and retry-time share | Number of retry triggers; retry wait time / end-to-end time | Service stability and retry strategy |
| Human intervention count | The number of times a human must supply information or fix things manually | Degree of automation |
It also helps to introduce a “task difficulty × completion quality” scoring model: the difficulty coefficient is assessed from the number of files, dependency complexity, and number of acceptance items, while completion quality is weighted across functionality, styling, and code standards.
Take this experiment as an example: the contrast between A and D illustrates a key point. If you look only at unit price, the low-cost model seems more attractive; but once the call count is multiplied several times over by wasted rounds and retries, both the actual cost per task and the user’s wait time rise. Cost evaluation should therefore rest on “how many rounds of calls and how much time each validated task takes,” not on the price of a single call.
6. Engineering Adoption: Making Evaluation Observable and Regression-Ready
To turn this kind of evaluation into a routine basis for selection, three engineering pieces need to be in place:
- Observable: assign a unified trace identifier to every model call and tool call, and record the model version, round, tool name, latency, return status, and error type. Use a log service for centralized storage and retrieval, and set thresholds on metrics such as timeout rate and retry count through monitoring and alerting, notifying promptly on anomalies.
- Regressable: maintain a fixed task set and acceptance scripts, and re-run them automatically after each model version update or framework-parameter change, comparing changes in key metrics so that problems like “the new version is slower” are caught before launch rather than after.
- Reproducible: containerize the runtime environment together with the acceptance scripts, and store results in object storage for long-term retention, so results produced at different times and by different members can be compared directly.
For teams using Baidu AI Cloud, these pieces can be supported by general cloud capabilities such as model services, object storage, log services, monitoring and alerting, and container products, reducing the investment in self-built infrastructure.
7. Common Pitfalls and Selection Guidance
Four pitfalls to watch for
- The single-sample trap: Model A’s 17 minutes is a single result, and re-running the same model can swing by minutes. A single sample is not enough to support a conclusion.
- The task-preference trap: a task like a creative collaboration system carries a degree of subjectivity, and differences in models’ interaction-design preferences may make certain kinds of scoring inherently advantageous.
- The hardware-and-environment trap: differences in runtime environment, browser version, and resource specs can let environmental noise mask the real gaps between models.
- The default-parameter trap: the framework’s retry, timeout, and context-compression parameters are usually general configurations and may not fit a particular model’s service characteristics. Adjusting the default parameters alone can itself produce a multiples-of efficiency difference.
Selection guidance
- Locate the bottleneck first: is it cost sensitivity, interaction-latency sensitivity, or insufficient stability in long workflows? Different bottlenecks call for different optimization priorities.
- Adopt a layered strategy: route high-frequency, lightweight tasks such as exploration, retrieval, and formatting to low-latency models; route architecture design and complex debugging to more capable models; and for tasks involving screenshots and interfaces, consider models with visual capability.
- For long workflows, prioritize long-context capability and a context-compression strategy, and reduce reliance on single-call success rates through segmented execution and resume-from-breakpoint.
- Build your own task set and metrics baseline, and treat the model as a replaceable part rather than a one-shot answer. Model versions iterate frequently; today’s benchmark conclusion may be invalid a few months from now. What endures is the evaluation method and the engineering configuration.
8. Conclusion
The most valuable part of this experiment is not how close the four final screenshots are, but the truth revealed by the process data: with the same framework and the same acceptance criteria, what decides Agent efficiency is never the model alone. Action rounds, tool calling and self-correction, service stability and retry strategy, and context management together form the complete efficiency chain.
So rather than repeatedly asking “which model is strongest,” it is better to answer three questions:
- Where is my task bottleneck?
- Are my retry and context strategies robust?
- Do I have a sustainable, regression-ready evaluation system?
Do these three things well, and the selection conclusion becomes more reliable—and the Agent can deliver steadily and efficiently in real business.
