AI Agent Efficiency Benchmarking: Fix the Execution Framework, Swap Only the Model

    This article presents a controlled-variable methodology for measuring AI agent efficiency: hold the execution framework, task, assets, browser environment, and acceptance rules constant, and vary only the underlying model. Using a creative studio collaboration app as the benchmark task — five page states, three roles, state that survives a refresh, and desktop plus mobile viewports — four comparable models are evaluated under a strict one-repair-round rule. Although all four finished, total time ranged from roughly 17 minutes to nearly 90, revealing four distinct behavior patterns. The article breaks down the real drivers of efficiency (action turns, tool-call quality, service stability, context management), provides a reusable YAML benchmark configuration, and outlines the implications for model selection and engineering practice.

    1. Why Swapping Only the Model Beats Swapping Only the Framework

    When evaluating agent capability, there are two common comparison setups: fix the model and swap the execution framework, or fix the framework and swap only the underlying model. The first measures how well the shell fits the task. The second measures how the brain itself performs in a standardized environment.

    I ran the first kind of experiment: I changed frameworks under the same model, and the fastest one built the most complex creative collaboration system in under 17 minutes. So this time I inverted the variable — the execution framework stays completely fixed, and the task, assets, context, browser environment, and acceptance rules all remain identical. Only the model changes.

    The value of this control is straightforward: the framework’s tool set, scheduling logic, retry strategy, and context window are all constant, so any efficiency difference can be attributed to the model alone. That produces conclusions that are reproducible and explainable, rather than conclusions distorted by framework-specific behavior.

    2. Task and Constraint Design

    The benchmark task has to be heavy enough to create separation between models. This evaluation uses a creative studio collaboration system. The only inputs are a handful of openly licensed images plus seed data for team members, assets, versions, and comments. There are no existing pages to reference, so the model has to produce five page states on its own:

    • Project overview
    • Moodboard
    • Asset annotation and comments
    • Version approval
    • Publish preview

    The pages must support three roles (creative director, designer, client) and implement interactions such as asset drag-and-drop, comment anchoring, version switching, and approve/return actions. State must also survive a page refresh. On the acceptance side, both a desktop viewport (roughly 1440×1000) and a mobile viewport (roughly 390×844) are checked.

    The critical constraints: only the first-round screenshot is kept for each run, only one repair round is allowed, and the final delivery must include code, screenshots, logs, review, and metrics. These rules force the model to get it right the first time instead of masking problems with unlimited iteration.

    3. Results: Same Finish Line, Very Different Paths

    Four models of comparable parameter scale — three positioned for low cost or high throughput, one long-context model as a reference — all completed the task, but the process differences are far more informative than the final screenshots. The behavior falls into four typical patterns:

    Behavior pattern Characteristics Duration
    Efficient first pass Lean structure, clear goal, almost no rework ~17 minutes
    Full rerun after a failed round First version failed acceptance and had to be rebuilt entirely ~1 hour 12 minutes
    Multiple built-in automatic retries No full rerun, but roughly 3 automatic retries triggered internally ~88 minutes
    Extended backoff after retries exhausted Default retries failed consecutively; success only came after adjusting the backoff strategy First success at ~1 hour 04 minutes

    This comparison makes one point directly: a low-cost or high-throughput positioning only describes part of a model’s capability. Real agent efficiency also depends on the number of action turns, the quality of tool calls, service stability, and context management.

    4. What Actually Determines Efficiency

    1. Number of action turns and convergence speed
    The fastest model was often not the one with the prettiest single output, but the one that approached the acceptance criteria with fewer action turns. Every wasted turn amplifies total time linearly, so taking fewer detours matters more than a flashy one-shot generation.

    2. Tool call quality
    The success rate and combination of file read/write, browser rendering, and command execution directly determine how likely rework is. Models with sensible tool-call granularity that can self-correct after a failure significantly reduce the overhead of manual and automatic retries.

    3. Service stability
    Output throughput, timeout rate, and how often rate limits are hit are all infrastructure-side variables of the model service. The same model can take twice as long under high concurrency or throttling, which is exactly where extended backoff strategies pay off.

    4. Context management
    A long-context reference model can hold more task information at once and avoid repeated retrieval, but the longer the context, the higher the risk of diluted attention and interference from irrelevant information. The context window is a double-edged sword and needs to be paired with a sensible retrieval and trimming strategy.

    5. A Reusable Benchmark Methodology

    To make this kind of experiment reliably repeatable, make the evaluation environment configuration-driven:

    1. # Controlled variable: switch only the model field, keep everything else constant
    2. task: creative_collab_app
    3. harness: fixed_v1 # execution framework fixed
    4. assets: seed_dataset_v3 # assets fixed
    5. browser:
    6. viewports: [1440x1000, 390x844]
    7. acceptance:
    8. first_shot_required: true
    9. repair_rounds: 1 # only one repair round allowed
    10. artifacts: [code, screenshots, logs, review, metrics]
    11. model: <model_under_test> # the only variable

    Metric collection should cover both outcome and process:

    • Outcome: whether acceptance passed, first-version pass rate, number of repair rounds
    • Process: total duration, number of action turns, number and failure rate of tool calls, number of automatic retries, context usage

    6. Implications for Selection and Engineering Practice

    First, do not substitute a speed or price label for a capability judgment. Run an end-to-end evaluation against your own tasks and acceptance criteria. Second, include service stability in the selection criteria: pay attention to rate limits, timeouts, and retry backoff, and add circuit breaking and backoff on the calling side when needed. Third, make retry strategy configurable: when the default policy is exhausted, automatically extending backoff often rescues a run better than simply switching to a different model. Fourth, context management is engineering work: use retrieval trimming and structured prompts to keep the model focused on the information that matters within a limited window.

    Ultimately, the core of evaluating AI agents is not assigning models a static ranking. It is building a reproducible process that distinguishes fast-and-stable from slow-and-reworked. Fixing the framework and swapping only the model is the key step for separating model capability from the noise of the framework and the environment.