Multi-Model AI Agent Benchmark: Where Efficiency Gaps Come From on Identical Tasks

    This article presents a controlled benchmark comparing four LLMs running as AI agents on a single, identical creative-studio collaboration task. By holding the execution framework, context budget (262K window, 200K task budget), the MCP-based browser-automation tool, and the acceptance rules constant, the only variable left is the model itself. All four models eventually passed acceptance, yet end-to-end wall-clock time ranged from about 17 minutes to 88 minutes—a gap of more than five times. Crucially, the difference came from rework and retry behavior rather than raw generation speed. The article breaks down the four drivers of agent efficiency (action rounds, service stability and retry strategy, context management, and first-pass quality), defines a set of metrics for reproducible comparison with sample Python code, and offers engineering takeaways for teams that build or evaluate their own agents.

    Introduction: An Agent’s “Speed” Is Not the Same as Tokens per Second

    A model that generates text quickly is not necessarily the most efficient agent. To deliver a runnable system, a model must repeatedly cycle through “observe the result, judge the deviation, choose a tool, execute an action.” End-to-end time is determined by inference latency, the number of action rounds, tool execution time, failed retries, and context handling—all combined.

    Discussions about agent efficiency tend to fall into two categories. The first fixes the model and swaps the execution framework, comparing how different orchestration approaches affect task completion speed. The second does the reverse: it fixes the execution framework and swaps only the model. This article belongs to the second category. With the task, source materials, context budget, browser environment, and acceptance rules held completely identical, it compares the execution performance of four models side by side.

    Part 1: Experimental Design—Isolating Model Differences with a Single Variable

    1.1 Variable Control

    The four runs differ only in the model used:

    • Unified execution framework: All runs use the same agent execution framework.
    • Unified context budget: The context window is set to 262K tokens, with a task budget of 200K tokens.
    • Unified tool capability: All runs access the same browser-automation tool through the Model Context Protocol (MCP).
    • Unified acceptance standard: Resolution checkpoints, first-version screenshots, number of allowed fixes, and the deliverables checklist are all identical.

    The reason for this design is that model capability, framework capability, tool capability, and context capability stack on top of one another during a single long task. Only by fixing the last three can the difference in elapsed time be attributed primarily to the model itself.

    1.2 Task Definition: A Creative-Studio Collaboration System Built from Zero

    The task is to implement a creative-studio collaboration system (hereafter “the collaboration system”) with no existing pages to reference:

    • Input materials: Only 6 openly licensed museum artifact images, plus seed data for members, materials, versions, and comments.
    • Page states: Five core states—project overview, Moodboard, material-anchored comments, version approval, and publish preview.
    • Roles and interactions: Three roles—creative director, designer, and client. Materials must support drag-and-drop, comments must anchor to specific materials, versions must support switching, approval, or rejection, and state must survive a page refresh.
    • Acceptance checkpoints: Desktop at 1440×1000 and mobile at 390×844.

    This task is difficult for two reasons. First, it requires long chains of tool calls, where the page is built, checked, and corrected step by step through automation tools. Second, it involves front-end state management and role-based permissions, which single-pass generation rarely gets right on the first try. It places high demands on “maintaining goal consistency over a long run” and “recovering quickly from errors.”

    1.3 Acceptance Criteria

    Each run keeps only its first-version screenshot, with a single round of fixes allowed. The final deliverable must include code, screenshots, logs, a review record, and metrics data. This means “done” must meet a deliverable standard, not merely a claim of completion in the conversation.

    Part 2: Four Execution Results—All Completed, Wildly Different Processes

    In the end, all four runs passed acceptance, but the differences in process are far more instructive than the page screenshots:

    Item Model A Model B Model C Model D
    Type positioning Vision-enhanced General-purpose Fast-response Extra-long-context (control)
    End-to-end time ~17 min 1 h 12 min (incl. full rerun) 88 min 1 h 04 min (to first success)
    Rework approach No full rerun Full rerun after first-round failure No rerun; 3 internal automatic retries Default retry failed twice in a row; completed after widening backoff
    Main bottleneck Structural trade-offs and detail consistency First-round output failed acceptance Retry waiting and serial round-trips Default backoff strategy was too short

    Model A (vision-enhanced): Finished in about 17 minutes with no full rerun. The output used a light editorial style with simple structure, where project status, materials, and member relationships were clear at a glance. The approach ensured structural clarity first, then handled interactions.

    Model B (general-purpose): The first-round output used a black-and-gold with off-white color scheme and serif headings, and it drew out the permission matrix and approval states—the visual direction was already taking shape, but it did not pass acceptance. The fix round used a full rerun, bringing the total, including fixes, to 1 hour 12 minutes. This shows that a first round that “looks good” is not the same as one that “works”; strict acceptance will faithfully expose the deviation.

    Model C (fast-response): No full rerun, but the execution framework internally triggered 3 automatic retries, for a total of 88 minutes. A fast single response does not mean a fast overall result: when a task requires many consecutive rounds of tool calls, any single failure can force the following steps to queue and wait.

    Model D (extra-long-context, used as a horizontal control): Under the default retry strategy, it failed twice in a row. After the backoff time was widened, the task completed, reaching first success at a cumulative 1 hour 04 minutes. This exposes an easily overlooked engineering issue: the retry strategy itself is a variable that affects agent efficiency.

    The relative multiples make this more intuitive. Using Model A as the baseline (1×), Model D is about 3.8×, Model B is about 4.2×, and Model C is about 5.2×. With the same framework, tools, and acceptance criteria, simply swapping the model produces more than a fivefold difference in end-to-end time.

    Part 3: Breaking Down the Causes—Agent Efficiency Is Decided by Four Things

    3.1 Action Rounds: Effective Tool Calls Matter More Than the Number of Calls

    Whether an action is effective is not judged by how many tools were called, but by whether it pushed the task forward. Efficient execution shares common traits:

    1. Break the large task into small, verifiable steps, each with a clear completion criterion.
    2. Extract only the information needed for the next step from each tool result.
    3. When a deviation appears, make the smallest possible correction first, avoiding a full teardown and redo.

    Conversely, when a single generation round covers too much scope, an error tends to trigger a cascade of failures, ultimately forcing a rerun. This kind of rework is the most costly in wall-clock time.

    3.2 Service Stability and Retry Strategy: Failures Are the Norm, Not the Exception

    In a long task spanning dozens of model calls and browser operations, rate limiting, timeouts, and transient errors are almost inevitable. The key points of a retry strategy are:

    • Exponential backoff with jitter: Avoid many concurrent sessions retrying at the same instant and causing secondary congestion.
    • Distinguish error types: Parameter errors and content-policy blocks should not be retried; only rate limiting and transient network errors are worth backing off and retrying.
    • Set a retry limit: And make critical steps idempotent to prevent duplicate writes.

    Illustrative code:

    1. import random
    2. import time
    3. # Retry only transient errors such as rate limiting and timeouts;
    4. # raise immediately for parameter errors and content-policy blocks.
    5. RETRYABLE = {"rate_limited", "timeout", "server_error"}
    6. def call_with_backoff(fn, max_attempts=4, base=2.0, cap=60.0):
    7. for attempt in range(max_attempts):
    8. try:
    9. return fn()
    10. except TransientError as exc:
    11. if exc.code not in RETRYABLE or attempt == max_attempts - 1:
    12. raise
    13. # Exponential backoff + random jitter (full jitter), capped at `cap` seconds
    14. delay = random.uniform(0, min(cap, base * 2 ** attempt))
    15. time.sleep(delay)

    Model D’s experience shows that default backoff parameters are set for general scenarios. When a long task encounters denser transient congestion, appropriately widening the cap and increasing the number of retries can actually converge faster.

    3.3 Context Management: A Large Window Does Not Mean High Utilization

    A 262K context window is only a capacity ceiling. When the task reaches its later stages, effective reasoning has to contend with several problems:

    • Early planning gets diluted as massive screenshot descriptions and log output flood the context, obscuring key constraints.
    • Repeated tool results eat into the budget.
    • Early error messages are forgotten, so the same problems recur.

    Two mature practices help here. First, compress key information—such as requirement constraints, the acceptance checklist, and current progress—into a structured state file that the agent reads on demand. Second, summarize and trim tool output, keeping only the parts relevant to the goal.

    3.4 First-Pass Quality: The One-Fix Rule Amplifies the Gap

    The “only one round of fixes allowed” rule places maximum weight on getting it right the first time. Efficient models deliver in layers—“structure, interaction, styling”—in the first round, first ensuring correct information hierarchy and state persistence after refresh, then handling visual details. Passing acceptance on the first round drives the fix cost directly to zero.

    Part 4: Defining Comparable Efficiency Metrics

    The reproducibility of this benchmark comes from metrics collected in a unified way. It is recommended to capture at least the following fields:

    Metric Meaning Notes
    wall-clock time End-to-end actual elapsed time Includes retry waiting and tool execution
    Action rounds Number of rounds in which the model participates in decisions and generation Measures task complexity and planning ability
    Tool calls Total number of tool calls actively initiated by the model Separates model-call time from tool-call time
    Retry count Number of framework automatic retries and full task reruns Reflects stability and rework
    First-pass rate Proportion of first-round outputs that directly pass acceptance The most direct quality signal
    Effective progress rate per step Number of rounds producing a valid state change ÷ total rounds Identifies ineffective actions

    An illustrative collection framework:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class RunMetrics:
    4. model: str
    5. wall_clock_s: float = 0.0 # End-to-end time, including tools and retries
    6. llm_latency_s: float = 0.0 # Total model inference time
    7. action_rounds: int = 0
    8. tool_calls: int = 0
    9. tool_retry: int = 0
    10. task_reruns: int = 0 # Number of full task reruns
    11. first_pass: bool = False # Whether the first version directly passed acceptance
    12. def summary(self) -> dict:
    13. return {
    14. "model": self.model,
    15. "wall_clock_min": round(self.wall_clock_s / 60, 1),
    16. "llm_share": round(self.llm_latency_s / max(self.wall_clock_s, 1e-6), 3),
    17. "tool_retry_rate": self.tool_retry / max(self.tool_calls + self.tool_retry, 1),
    18. "first_pass": self.first_pass,
    19. "task_reruns": self.task_reruns,
    20. }

    llm_share represents the proportion of end-to-end time spent on model inference. The lower this value, the more time is being spent on tool execution and retry waiting, which means the optimization focus should shift to tool orchestration and retry strategy rather than swapping the model.

    Part 5: Engineering Takeaways

    1. Don’t look only at unit price and response speed. A low-cost, high-response-speed model description captures only part of its capability. Real efficiency depends on the combination of action rounds, tool calls, service stability, and context management.
    2. Treat the retry strategy as a model runtime parameter. Backoff parameters should be adjusted dynamically based on task length and the error distribution.
    3. Replace teardown-and-redo with recoverable orchestration. Set checkpoints for critical steps and support resuming from breakpoints, so a single failure stays localized.
    4. Actively manage context in long tasks. Externalize goals, constraints, and progress as structured files to reduce wasted tokens.
    5. Drive iteration with objective acceptance. Use screenshots, logs, review records, and metrics as the delivery standard, making “done” measurable and comparable. This principle applies both to internal teams evaluating candidate models and to enterprises building their own agents.

    When the evaluation scales up, common cloud capabilities can carry the pipeline:

    • Use a container platform to schedule and isolate batch tasks.
    • Use object storage to archive screenshots and traces.
    • Use a logging service with monitoring and alerting for metrics aggregation.

    Taking Baidu AI Cloud as an example, applications typically run on containers or cloud servers, and pairing them with foundational cloud services such as object storage, logging, and monitoring and alerting is enough to build a repeatable evaluation pipeline.

    Part 6: Limitations and Notes

    This experiment is a controlled observation under a fixed task, fixed framework, and fixed tool set. The sample is a single complex task with one run per group, so the conclusions are better suited as a selection reference than as an absolute ranking. Actual performance is also affected by model version iteration, prompts, the quality of tool descriptions, and the temperature parameter, so teams are advised to repeat the measurement with their own business samples. In addition, capabilities such as long context and visual understanding are only one class of model characteristic and should not be equated directly with end-to-end efficiency.

    Conclusion

    After fixing the framework and swapping only the model, the results are still spread out: from 17 minutes to 88 minutes, a gap of more than five times, and the difference comes mainly from the number of reworks rather than single-pass generation speed. For development teams, rather than debating “which model is the strongest,” it is better to build your own evaluation pipeline: the same task, the same acceptance criteria, and unified metrics—let the data speak.