Model Capability Is Only One Part of Agent Efficiency

    This article looks at agent efficiency through a controlled cross-model benchmark. Four models were run under the same agent framework on the same multi-role creative collaboration task, with a fixed context window, task budget, tools, and acceptance rules. All four passed acceptance, yet their total completion time differed by several times. Instead of judging by the final screenshots, the article analyzes process metrics—time to first success, total time, action rounds, tool-call success rate, retries, and artifact completeness—and breaks the efficiency gap into four sources: the number of action rounds, failure-recovery strategy, context management, and server-side scheduling. It closes with five practices for reproducible benchmarking and practical guidance for model selection and engineering. The core takeaway: agent efficiency = model capability × framework design × service stability, and these three should be optimized together.

    1. Introduction: Why the Gap Matters More Once the Framework Is Fixed

    Agent applications have gradually moved from demos into production. A typical scenario: a user describes a requirement in a single sentence, and the agent autonomously browses reference material, writes code, invokes a browser to verify how the page renders, and iteratively fixes issues based on runtime results until acceptance criteria are met. This kind of application puts demands on model capability, the engineering framework, and service stability alike.

    When selecting a model, many teams tend to rely on public benchmark scores or a single “impressive” conversation. But an agent task is a loop:

    1. model reasoning → parse tool call → execute tool → read result → reason again → ...

    In this chain, the model is only one link. How the framework organizes prompts, executes tools, handles failures, and manages context directly determines both the final latency and the success rate. If you swap only the model and look only at the end result, it is hard to tell where a difference actually comes from.

    Fixing the model and swapping agent frameworks typically shows the maturity of framework engineering. Fixing the framework and swapping only the model gets closer to the core question: how does a model support agent execution? The rest of this article uses a multi-role creative collaboration application as a case study to describe how such a cross-model benchmark can be designed and what process metrics it produces.

    2. Benchmark Design: Locking Down Every Variable Except the Model

    2.1 One Unified Runtime Framework

    All four groups used the same agent framework. This framework is responsible for prompt organization, tool registration and invocation, the execution loop, retries, context management, and writing artifacts to disk. Adjustments were limited to parameters related to failure handling (see Section 4.2). Models were connected through a unified interface, and no configuration was changed beyond the model itself.

    2.2 One Unified Task and Input

    The task was a collaborative app for a creative studio, with three roles—creative director, designer, and client—and five pages: project overview, mood board, asset locating and commenting, version approval, and publish preview. The task has three defining traits:

    • Many states: projects, assets, versions, and comments are interrelated;
    • Complex interactions: asset dragging, comment anchoring, version switching, and approve/reject are tightly coupled;
    • Constraints: there are no ready-made pages to reference, so the model must design the information architecture and interactions from the requirements itself.

    The input was a small set of public image assets plus seed data for project members, assets, versions, and comments. The requirement description, role definitions, data structure definitions, and acceptance criteria were identical across all four groups.

    2.3 One Unified Toolset and Environment

    The tools available to the models were exactly the same, including file read/write, command execution, browser automation, and screenshot capture. There was no extra limit on the number of tool calls available when checking pages. The execution environment—data persistence, dependency installation, and so on—was also kept consistent, so that no group incurred extra waiting due to environment differences.

    2.4 One Unified Resource Budget

    All four groups used the same context window and task budget (for example, a 262K context window and a 200K task budget), which is one of the preconditions for a fair comparison. If the budget is too generous, the cost of long context and many rounds gets hidden; if it is too tight, “insufficient capability” and “insufficient budget” get conflated. The budget should therefore be set according to the target business scenario.

    2.5 One Unified Acceptance Rule

    The acceptance rules were fixed before the experiment started and were not adjusted during execution. They included:

    • Responsive check: no layout overflow or occlusion at desktop 1440×1000 and mobile 390×844;
    • Interaction check: each of the three roles has correct permissions and behavior, and state is not lost after a page refresh;
    • Traceability requirements: save the first-version screenshot, allow only one round of repair, and the final result must include code, screenshots, logs, review, and metrics.

    An illustrative acceptance checklist:

    1. task: creative_studio_collab
    2. pages: [overview, moodboard, asset_detail, version_review, publish_preview]
    3. roles: [creative_director, designer, client]
    4. acceptance:
    5. responsive:
    6. desktop: 1440x1000
    7. mobile: 390x844
    8. interactions:
    9. - drag_to_moodboard
    10. - comment_locate_asset
    11. - switch_version
    12. - approve_or_reject
    13. persistence:
    14. - state_survives_reload
    15. budget:
    16. context_tokens: 262144
    17. task_tokens: 200000
    18. repair_rounds: 1
    19. deliverables: [code, screenshot, log, review, metrics]

    3. Process Metrics: More Valuable Than the Final Screenshot

    All four groups completed the task, and their final-state screenshots each had a distinct style. But what is truly worth analyzing is how they reached the final state. None of the following metrics can be skipped:

    • Time to first paint: the moment the app first opens and the main page content becomes visible;
    • Time to first success: the moment, from task dispatch, at which the acceptance rules first pass completely. Repair happens after first success, so repair time is not counted here;
    • Total time: from task start to delivery of all artifacts, including time spent rerunning after failures;
    • Action rounds: the number of times the model reasons and outputs an action, where each round usually corresponds to at least one model call;
    • Tool-call count and success rate: the proportion of browser-verification failures is especially important;
    • Retry count and retry pattern: internal automatic retries, a single repair round, or a full rerun;
    • Artifact completeness: whether code, screenshots, logs, review, and metrics are all present.

    3.1 Latency Across the Four Groups

    Group Model scale and characteristics Execution path Time to first success Total time
    A Under 300B, with visual understanding Passed on the first attempt, no rerun, no retries ~17 min ~17 min
    B Under 300B Failed acceptance on the first round, passed after one repair round ~1h 12m ~1h 12m
    C Under 300B No full rerun; 3 automatic retries during execution 88 min 88 min
    D Larger-parameter control model Failed twice in a row under the default retry policy; completed after extended backoff ~1h 04m ~1h 04m

    Group A took less than one-fifth of the time of Group C, and all four results passed acceptance. This means the speed difference cannot be explained by differences in result quality.

    One thing to add: in this round, all repairs happened after first success, so time to first success equals total time. In other words, repair itself was almost never the bottleneck—the real bottleneck was the process of reaching the first success.

    It is also worth stressing that these are observations from a single task run one time. They reflect only each model’s execution behavior under this framework and this task, and should not be generalized into an overall ranking of model capability.

    3.2 Observations on UI Style

    The visual directions of the several apps were clearly different: some used a light, structurally minimal editorial style that made project status, assets, and member relationships clear at a glance; others paired black-and-gold with off-white and serif headings, presenting a museum-like editorial feel alongside a permission matrix and approval status. Such differences are worth referencing for creative apps, but they are hard to quantify. A more reliable approach is to break them into measurable items such as layout consistency, contrast, and responsive behavior, and to introduce review scoring.

    4. Four Sources of the Efficiency Differences

    4.1 Action Rounds Determine the Execution Pace

    Every agent action goes through “think—call—read—think again.” To implement the same feature, some models habitually batch the writes of multiple files and command executions into a single round, while others go back and forth one file and one command at a time.

    Assume the round-trip overhead per tool call is 2–5 seconds; the gap between 100 calls and 300 calls is minutes to more than ten minutes. In long tasks, the more target states the model must maintain, the more easily this gap accumulates—for example, role permission matrices, version linkage, and state persistence after refresh.

    4.2 Failure-Recovery Strategy Affects the Worst Case

    Both Group B and Group D hit failures midway; the difference lay in how they recovered:

    • Group B used a framework-level single repair round: first locate why acceptance failed, then fix it within the one allowed repair round;
    • Group D failed twice in a row under the default retry strategy, and only completed after switching to extended backoff with random jitter;
    • Group C did not do a full rerun, but automatically retried 3 times internally during execution.

    This shows that no single strategy is inherently better. Fast retries with a fixed count and fixed interval keep hitting the same wall when downstream services rate-limit or queue; backoff intervals that are too long, meanwhile, stretch the overall time. A more reliable approach is to retry only on recoverable errors and to fold retries into a global time budget:

    1. import random
    2. import time
    3. def call_with_backoff(fn, max_attempts=5, base=2.0, cap=60.0, deadline=None):
    4. """Exponential backoff retry with random jitter (example logic)."""
    5. for attempt in range(1, max_attempts + 1):
    6. try:
    7. return fn()
    8. except RetryableError:
    9. if attempt == max_attempts:
    10. raise
    11. delay = min(cap, base * 2 ** (attempt - 1))
    12. if deadline is not None:
    13. # Reserve time for this call and later retries so retries
    14. # don't eat into the main task budget
    15. delay = min(delay, max(0, deadline - time.monotonic() - base))
    16. time.sleep(delay + random.uniform(0, 1)) # Jitter to avoid synchronized retries
    17. raise RuntimeError("unreachable")

    4.3 Context Management Determines Whether Long Tasks Stay on Track

    After dozens of tool calls, the conversation inflates quickly. Screenshots, page text, command output, and error stack traces are all long text, and the early requirements can be “pushed out” of effective attention—showing up as repeated edits, missed role permissions, or overlooked acceptance items.

    If the context approaches the window limit, the framework must also compress or truncate, and compression itself consumes time and budget. Better practices include:

    • Manage context in layers by purpose: task descriptions and acceptance criteria that must be kept long-term are stored in a structured form at fixed positions;
    • Keep only necessary information: screenshots keep only dimensions and descriptions of key regions; logs keep only failure fragments;
    • Re-read files on demand, rather than pulling an entire large file into context at once.

    4.4 Server-Side Scheduling and Output Mode Affect Unit Cost

    For the same round of actions, different output modes can differ in latency by several times:

    • Outputting a long chain of thought before the tool call clearly raises per-call latency;
    • Issuing multiple independent calls in parallel can noticeably shorten wall-clock time;
    • When downstream services rate-limit or queue, the wait time is reflected directly in the total time.

    A benchmark therefore cannot look at model capability alone; it must also record wait time and distinguish “the model is slow” from “the service is slow.”

    5. Five Keys to a Reproducible Benchmark

    1. Fix the random factors: keep task dispatch time, the initial prompt, and dependency versions consistent to avoid bias from caching or version differences.
    2. Unify the delivery standard: all four groups use the same acceptance checklist, the same screenshot resolution and capture method, and acceptance is judged only by observable behavior.
    3. Keep the full trajectory: record every round’s action, tool result, latency, and retry as structured events for easy review.
    1. {"ts": 1024, "type": "action", "round": 7, "model_ms": 8300, "tool": "browser.click"}
    2. {"ts": 1035, "type": "tool_result", "round": 7, "ok": false, "reason": "selector_timeout"}
    3. {"ts": 1042, "type": "retry", "round": 7, "attempt": 1, "backoff_ms": 4000}
    1. Record failures, not just successes: failed rounds, failure reasons, and recovery methods are often the most valuable data for finding problems.
    2. Repeat across multiple runs and keep results separate: repeat each task several times and take the median; present latency and pass rate separately so that a single fluctuation is not mistaken for a conclusion.

    6. Implications for Model Selection and Engineering Practice

    • Account for cost by the actual task. A model with a low per-round price may cost more overall if it needs more rounds and retries to finish. A practical estimation approach:
    1. per-task cost ≈ average action rounds × (input context + output content) × unit price + extra cost from retries
    2. per-task time ≈ model inference time + tool execution time + waiting and retry time
    • Look at accuracy and efficiency together. Completion rate determines whether the task can be delivered; latency and retry count determine the experience and the cost.
    • Efficiency problems can be solved at the engineering layer. Batch calls, parallel execution, layered context, and jittered backoff retries all bring clear improvements; model selection and framework optimization should proceed together.
    • For long-running agents, it is advisable to configure log collection, call-chain tracing, and monitoring alerts for the agent process. External capabilities can draw on general-purpose cloud services—for example, object storage to hold screenshots and code artifacts, container platforms to host the runtime environment, and logging services to aggregate execution traces. Baidu AI Cloud also offers products in these general categories.

    7. Conclusion

    Under the same framework, four models all completed the same complex task, yet their latencies differed by several times. The value of this set of results lies not in “who is better,” but in revealing an engineering fact:

    Agent efficiency = model capability × framework design × service stability

    When evaluating, fixing the framework and swapping only the model lets you isolate and magnify the model factor for observation; but in real-world building, all three need to be optimized together. For developers, rather than repeatedly asking “which model is strongest,” it is better to first define the task, the budget, and the acceptance rules clearly, and to continuously measure every iteration with reproducible process metrics.