Benchmarking Mid-Sized Models in the Same Agent Framework: A Four-Model, One-Task Experiment

    This article presents a controlled end-to-end experiment in which the agent harness and task are fixed while four mid-sized models under 300B parameters are varied. It reviews the setup, each model’s runtime and retry behavior, four practical findings on speed versus efficiency, action rounds, layered retry policies, and context management, plus a reusable evaluation checklist for model selection.

    Why Run a Fixed-Framework, Model-Only Evaluation

    In the previous phase of my experiments, I concentrated the variables on the framework side: I fixed the same model and compared four agent orchestration frameworks for their engineering capability. The conclusion was fairly clear: the framework with the most complete structure and the strictest constraints delivered the most complex creative collaboration system task fastest, in about 17 minutes.

    This time, I reversed the experiment: the framework is fixed, and the model is the only variable.

    I designed it this way because, in real engineering work, teams often look at only two surface-level indicators when choosing a model: parameter scale and inference speed (or price per million tokens). But the actual delivery efficiency of an agent task is determined by a set of less visible variables:

    • Single-turn reasoning quality: whether the model can make correct decisions in fewer rounds;
    • Tool-call compliance: whether parameter formats are stable and whether the model follows the calling protocol;
    • Long-context management: whether key constraints are forgotten after multiple rounds accumulate;
    • Service stability: request failure rate, rate limiting, and timeout behavior;
    • Error recovery: whether the retry strategy after a failure is efficient.

    These variables cannot be seen from a benchmark score alone. They can only be observed in an end-to-end task. That is exactly where the value of this experiment lies.

    Experiment Design: Pinning Down Every Variable

    To ensure a fair comparison, I fixed the following:

    1. Harness (agent execution framework): unified
    2. Task goal: the same complex frontend system (multiple modules, pages, and interaction logic)
    3. Materials and context: identical prompts, reference screenshots, and constraint documents
    4. Browser environment: same version, same viewport
    5. Acceptance rules: the same automated checks (page rendering, interaction reachability, visual fidelity)
    6. Only variable: underlying model (four mid-sized models under 300B parameters)

    All four tested models were publicly available services from different vendors, positioned similarly as low-cost, low-latency mid-sized flagships. To add a reference point, one 256K long-context model participated as a control sample in the comparison.

    I chose mid-sized models rather than very large ones because they are closer to the real cost range for enterprise deployment. Most businesses will not call the largest model for every agent request; cost-effective models are the daily workhorses.

    Process and Results for the Four Experiments

    All four models eventually completed the task, but the process differences were far more valuable than the final delivered screenshots.

    1. Model A: Passed on the First Attempt in 17 Minutes

    The most stable model never triggered a full-round rerun. Its action rounds were compact, and its tool calls had almost no format errors. It went from start to acceptance in about 17 minutes. Its characteristic was that it “thought before acting” at every step. The generated code had a high first-compile pass rate, reducing back-and-forth repair rounds.

    2. Model B: First Failure, 1 Hour 12 Minutes Total After Rerun

    The second model was interrupted midway through the first round. After restarting the whole task, it completed it, with a total time of 1 hour 12 minutes. Investigation found that the interruption was not closely related to the model’s reasoning ability; it came more from occasional server-side timeouts. This exposed an often-overlooked issue: the stability of the model service is itself part of Agent efficiency. For models with the same level of capability, if availability differs by an order of magnitude, end-to-end delivery time may multiply several times.

    3. Model C: 88 Minutes with 3 Internal Automatic Retries

    The third model did not trigger a full-round rerun, but it produced three step-level automatic retries internally, for a total time of 88 minutes. Its single-step retry mechanism kept the task from being interrupted, but each retry meant extra inference overhead and context expansion. This type of model is suitable for scenarios that tolerate latency but require the task to stay online.

    4. Model D: The Cost of Retry Backoff, 1 Hour 4 Minutes to First Success

    The control model’s default retry policy failed twice consecutively and entered extended backoff (exponential waiting) before recovering. The cumulative time to first success was 1 hour 4 minutes. Exponential backoff protects the server side, but for the client it multiplies waiting time—something very noticeable in interactive development scenarios.

    Four Conclusions from the Experiment

    Conclusion 1: Speed Positioning ≠ End-to-End Efficiency

    All four models were labeled “fast,” but actual completion times ranged from 17 minutes to nearly 1.5 hours. Nominal speed reflects only single-inference latency, while an Agent task consists of dozens or hundreds of inferences, tool calls, and retries. Friction in any part of the chain is amplified cumulatively.

    Conclusion 2: Action Rounds Are a Hidden Cost Multiplier

    Models with fewer rounds were not only faster but also had lower total token consumption. When evaluating models, I recommend treating “total rounds to task completion” as a primary metric rather than looking only at final artifact quality.

    Conclusion 3: Retry Policies Need Layered Design

    Step-level retry is better than full-round rerun, but retries should not accumulate indefinitely. A configuration approach worth considering in practice:

    1. retry_policy:
    2. step_level: # retry on single-step failure
    3. max_attempts: 3
    4. backoff: fixed # fixed short interval, avoid exponential waiting
    5. round_level: # restart after full-round interruption
    6. max_attempts: 1
    7. preserve_context: true # preserve completed context, avoid starting from scratch

    Conclusion 4: Context Management Determines Long-Task Stability

    In multi-round tasks, constraint information (acceptance rules, design specifications) is easily diluted by outputs from later rounds. In a fixed framework, “inject key constraints once every N rounds” can significantly reduce the rate of drift later in the task.

    A Reusable Evaluation Checklist

    If you want to run a similar evaluation in your own business, I recommend collecting at least the following metrics:

    Dimension Metrics
    Efficiency Total time, total rounds, total token consumption
    Stability Step-level retry count, server-side failure count
    Quality Acceptance-rule pass rate, manual repair effort
    Cost Converted cost (unit price × total consumption)

    Final Thoughts

    The core lesson from this experiment is this: model selection cannot rely only on leaderboard scores and nominal speed. You must run end-to-end controlled-variable tests in your own Harness and on your own task distribution. Among models with similar parameter scales, real delivery efficiency can differ by more than 4x, and those gaps are hidden in “invisible variables” such as rounds, retries, stability, and context. For engineering teams, building evaluation infrastructure has more long-term value than chasing any newly released model.