How to Compare LLM Agents Under a Fixed Execution Framework: A Model Benchmarking Methodology

    This article presents a methodology for benchmarking large language models as AI Agents by holding the execution framework constant and varying only the model. Drawing on a controlled experiment in which four models built the same creative collaboration system, it shows that final screenshots can look identical while end-to-end runtimes range from 17 minutes to 88 minutes. It breaks down where that gap comes from—action rounds and tool-call efficiency, service stability and retry strategy, context management, and first-time correctness—and explains why advertised speed or cost tells only part of the story. It closes with a reproducible evaluation design (repeat runs, randomized order, unified metrics and failure budgets), a standard metric set, and practical guidance on layered scheduling and observability for production Agent systems.

    1. Introduction: The Model Isn’t Weak, So Why Is the Agent Still Slow?

    In Agent applications, many people choose a model based on leaderboard scores or advertised pricing. But when running a complex task that requires dozens of tool calls, a familiar pattern often appears: the models are all capable, the final deliverables meet the quality bar, yet the end-to-end runtime stretches from ten-plus minutes to one or two hours, or more. The bottleneck usually isn’t how “smart” a model is. It’s how each step executes—how many rounds it takes on average, how much context each round carries, and how the system recovers when a tool call fails.

    In a prior controlled experiment, a team held the reasoning model fixed and swapped only four Agent execution frameworks (hereafter “execution frameworks,” meaning the runtime responsible for task orchestration, tool calls, browser operations, and result validation), then compared how efficiently they completed the same complex creative collaboration system. The results:

    Execution Framework Completion End-to-end time
    Codex Not completed —
    OpenCode Not completed —
    OpenDesign Not completed —
    Execution Framework A Completed 16 min 55 s

    This time we reversed the variable: the execution framework is fixed and held to one version, while the task, materials, context, browser, and validation rules all stay the same—only the model changes.

    Note: The models evaluated here have been anonymized and are referred to below as Models A through D. The times reported are single-run recorded values used to illustrate process differences; they do not represent the stable performance of any model.

    2. Experiment Design: Swap Only the Model, Fix Everything Else

    1. Fixed Variables: Guaranteeing a Fair Comparison

    Dimension What is held constant
    Execution framework The same runtime and version, with identical toolset, loop logic, and retry/timeout policies
    Task Build a creative collaboration system with full interactive capabilities from scratch
    Materials & context Requirements, reference material, initial codebase, and system prompt are all identical
    Browser The same browser environment, used for page operations and screenshots
    Validation rules The same functional acceptance checklist, auto-checked item by item
    Runtime environment The same development environment and dependency versions, with the full run logged end to end

    2. The Variable: Four Models with Different Positioning

    Model Positioning
    Model A A lightweight general model that emphasizes high throughput and low latency
    Model B The latest release of a “Flash” series that emphasizes inference speed
    Model C An experimental version that supports visual input
    Model D A control model that emphasizes long context (256K)

    3. Task and Acceptance

    The task starts from an empty project directory: build out the pages, wire up the interaction logic, add data persistence, then verify the key flows through actual browser operations. It’s a typical “multi-step, long-chain, observable” Agent task. Acceptance recognizes only the final result—whether the pages render and interact as expected, and whether data reads and writes correctly. No matter how winding the process is, if the final result meets the bar, it passes. As a result, the final screenshots from the four models look identical; all the differences live in the process data.

    3. Results: All Completed, but the Processes Differ Sharply

    Model Final status End-to-end time Full reruns In-framework auto-retries Key process
    Model A Pass 17 min 0 Not mentioned Passed in a single run
    Model B Pass 1 h 12 min 1 — First round failed; passed after a rerun
    Model C Pass 88 min 0 3 Kept advancing through internal auto-retries
    Model D Pass 1 h 04 min cumulative on the first successful run First two runs both failed Two consecutive failures under the default retry policy Completed after switching to an extended backoff

    Three points stand out:

    • Speed is not efficiency. Model C supports visual input and could have saved a tool call in the screenshot-verification step, yet its three internal auto-retries made it the slowest of the group. A local speed advantage was offset by stability problems.
    • Retry cost amplifies with chain length. The behavior of Models B and D both make the same point: the retry count itself isn’t the problem—where the retry resumes from is. A full rerun throws away all completed work and pays the inference and tool-call costs all over again.
    • Default policies aren’t necessarily tuned for Agent scenarios. Model D failed twice under the default retry policy and only succeeded after its backoff was adjusted. Defaults tuned for conversational scenarios where “one call decides success or failure” may not suit an Agent chain that runs for tens of minutes.

    This leads to our first conclusion: an advertised low cost or high speed describes only part of a model’s capability. An Agent’s real efficiency depends on the combined effect of action rounds, tool-call quality, service stability, and context management.

    4. Deep Dive: The Four Dimensions That Actually Open the Gap

    The total time of a single run can be approximated as:

    1. T_total ≈ Σ (t_prefill,i + t_decode,i) # Each round's prefill and decode time
    2. + Σ t_tool,j # Each tool call's execution time (build, launch, screenshot, etc.)
    3. + Σ t_wait,k # Waiting caused by queueing, rate limits, and service jitter
    4. + Σ t_retry,m # Retry/backoff waits plus the cost of redoing completed work

    Prefill time correlates with the input token count per round, and decode time correlates with the output token count per round. This explains why both “too much in one round” and “too little in one round” slow things down: the former tends to trigger timeouts and full reruns, while the latter multiplies the round count.

    1. Action Rounds and Tool-Call Efficiency

    Each cycle of a model—perceive the environment, decide, call a tool, observe the result—forms a loop. Reducing rounds and raising the information yield of each round is the most direct way to shorten wall-clock time.

    Typical behaviors of an inefficient model include:

    • Fragmented exploration: reading one file or checking one variable at a time, accumulating information serially until the round count balloons.
    • Repeated labor: keeping no “done” list, so it repeatedly reads the same file or rebuilds the same module.
    • Trial-and-error calls: guessing tool parameters wrong and probing repeatedly instead of first confirming the tool’s signature and return structure.

    Efficient models tend to read several files at once, issue independent calls in parallel, and merge large operations into verifiable units—for example, generating a whole module at once and building immediately.

    To measure this dimension, use the effective round ratio:

    1. Effective Round Ratio = rounds that changed an acceptance item's status / total rounds

    The higher this ratio, the more output the model produces per round.

    2. Service Stability and Retry Strategy

    An Agent chain spans model inference, tool execution, builds, and browser operations, and any of these steps can hit rate limits, timeouts, or transient faults. Retries themselves aren’t the danger—what’s dangerous is a retry strategy that doesn’t match the failure type. Failures should be handled by category:

    Failure type Typical symptom Recommended strategy
    Recoverable transient fault Rate limiting (429), gateway or timeout errors Exponential backoff + random jitter + a cap on attempts
    Deterministic error Invalid parameters, context overflow, insufficient permissions Fail fast, fix, then retry; don’t blindly back off
    Environment failure Missing dependencies, port already in use Fix the environment first, then resume the current step

    A retry with backoff and a cap can be implemented with a skeleton like this:

    1. import random
    2. import time
    3. # Retryable HTTP status codes: rate limiting and common gateway/timeout errors
    4. RETRYABLE_STATUS = {429, 500, 502, 503, 504}
    5. def is_retryable(exc):
    6. status = getattr(exc, "status", None)
    7. return isinstance(exc, TimeoutError) or status in RETRYABLE_STATUS
    8. def call_with_backoff(fn, max_attempts=4, base_delay=1.0, cap=30.0):
    9. """Exponential backoff retry for transient faults; deterministic errors raise immediately"""
    10. for attempt in range(1, max_attempts + 1):
    11. try:
    12. return fn()
    13. except Exception as exc:
    14. if not is_retryable(exc) or attempt == max_attempts:
    15. raise
    16. delay = min(cap, base_delay * 2 ** (attempt - 1))
    17. time.sleep(delay * (0.5 + random.random())) # jitter to avoid retry storms

    Rather than a “full rerun,” it’s more sensible to build recoverability at the step level:

    • Idempotent tool operations: submit requests with fixed identifiers and compare content fingerprints so that repeated execution produces no side effects.
    • Stage snapshots: save a project snapshot after each acceptance-worthy milestone, and resume from the most recent snapshot on retry.
    • Retry budget: set a total retry count, a cumulative backoff time, and an overall wall-clock budget per task to avoid “infinite retries.”

    3. Context Management

    Context is an Agent’s “field of view,” but more isn’t always better. A longer input carries three costs:

    • Prefill time increases every round, and token cost rises roughly linearly;
    • Key constraints can be drowned out by large volumes of intermediate output, causing attention dilution;
    • Long-context models see their effective use of information decline as they approach the window limit.

    In long-chain tasks, logs, errors, page screenshots, and file contents keep piling up. Without management, every later round pays a heavy “reading cost.” Common practices include:

    • On-demand loading: use retrieval instead of full injection, placing only the snippets relevant to the current step into context;
    • Structured compression: periodically summarize into a four-part tuple of goal, done, to-do, and known constraints, replacing verbose raw output;
    • Tool-result truncation: keep the key information (such as the head of an error stack) and discard redundant output;
    • Subtask isolation: hand independent subtasks to sub-agents and return only their conclusions, keeping the intermediate process from polluting the main context.

    4. First-Time Correctness and Self-Verification

    The same task can be completed correctly on the first round or only after repeated fixes, and the difference in time can be severalfold. A model that fully understands the requirements before acting and builds and self-checks during execution can catch failures internally.

    Common forms of “hidden rework” include:

    • Missing a minor item on the acceptance checklist and getting bounced back at validation;
    • Breaking feature B while changing feature A, triggering a regression;
    • Surface-fixing an error without locating its root cause.

    Efficient models, by contrast, tend to verify as they go: they run the build and tests after finishing each module, and they read the full error before acting. In Agent scenarios, “a little slower to confirm” often saves more time than “a little faster to trial-and-error.”

    5. Evaluation Design: Making Conclusions Reproducible and Explainable

    A single experiment only shows “what happened.” To learn “how it usually goes,” the evaluation plan needs to add the following:

    1. Multiple repeats with randomized order: run each model at least 3–5 times and shuffle the execution order to avoid bias from caching, time of day, and similar factors.
    2. A unified statistical basis: use the median and P90 of each metric across runs as the primary indicators, and also report success rate and first-time success rate—not just the single best or worst run.
    3. A unified failure budget: give every model the same maximum rounds, cumulative retry count, and total wall-clock budget so that no single model can inflate the average by “trying indefinitely.”
    4. Standardized process instrumentation: uniformly record rounds, tool calls, elapsed time, token usage, and retry reasons to guarantee a fair comparison.
    1. from statistics import mean, median
    2. from dataclasses import dataclass
    3. @dataclass
    4. class RunRecord:
    5. model_id: str
    6. run_id: str
    7. passed: bool # Whether all acceptance items passed
    8. wall_seconds: float # End-to-end wall-clock time
    9. rounds: int # Total rounds
    10. effective_rounds: int # Rounds that changed an acceptance item
    11. tool_calls: int # Number of tool calls
    12. tool_failures: int # Number of failed tool calls
    13. retries: int # Number of retries
    14. retry_wait_seconds: float
    15. input_tokens: int
    16. output_tokens: int
    17. cost: float # Converted using the actual billing rules of the service used
    18. def summarize(records):
    19. passed = [r for r in records if r.passed]
    20. if not passed:
    21. return {"pass_rate": 0.0}
    22. walls = [r.wall_seconds for r in passed]
    23. return {
    24. "runs": len(records),
    25. "pass_rate": round(len(passed) / len(records), 2),
    26. "wall_seconds_p50": round(median(walls), 1),
    27. "wall_seconds_mean": round(mean(walls), 1),
    28. "avg_rounds": round(mean([r.rounds for r in passed]), 1),
    29. "effective_round_ratio": round(mean(
    30. [r.effective_rounds / max(r.rounds, 1) for r in passed]), 2),
    31. "tool_failure_rate": round(mean(
    32. [r.tool_failures / max(r.tool_calls, 1) for r in passed]), 3),
    33. "retry_wait_ratio": round(mean(
    34. [r.retry_wait_seconds / r.wall_seconds for r in passed]), 3),
    35. "avg_tokens": round(mean(
    36. [r.input_tokens + r.output_tokens for r in passed])),
    37. "avg_cost": round(mean([r.cost for r in passed]), 4),
    38. }

    Overall, to assess a model’s real performance in Agent scenarios, at least five metrics matter:

    Metric Meaning How to use it
    End-to-end time (P50/P90) The completion time users actually perceive Prioritize optimizing it; it directly determines the interaction experience
    Success rate / first-time success rate How far the task can be completed and the share done right the first time Judge against the business’s tolerance for failure
    Effective round ratio Output efficiency per round Measures “whether it takes detours”
    Tool failure rate / retry-wait ratio How robust the execution chain is When too high, investigate environment and strategy first
    Cost per task The average cost of one task Weigh it together with time and success rate

    6. Implementation Guidance: From “Pick a Model” to “Combined Scheduling”

    The value of a single comparison isn’t a leaderboard—it’s a basis for engineering decisions:

    • Choose by task distribution. High-frequency, short-chain lightweight tasks can favor low-latency, low-cost models; long-chain, high-complexity tasks should weigh first-time correctness and context-handling ability more heavily.
    • Schedule in layers. Use more capable models for planning and key decisions, and hand deterministic operations like file editing and format conversion to faster, lighter models or script tools, lowering overall time and cost.
    • Evaluate continuously. Model versions keep iterating, so establish a fixed evaluation task set and a metrics dashboard, and regression-test with the same standard after each upgrade rather than drawing conclusions from a single experience.
    • Observability first. Bring rounds, tool calls, retries, and elapsed time into your logging and monitoring; when something goes wrong, you can quickly pinpoint whether it’s a model-capability, service-stability, or environment issue.

    When building this kind of evaluation and runtime environment in the cloud, you can combine general cloud capabilities:

    • Use a container platform to host the sandboxed development environment and browser, isolated per task and elastically scaled;
    • Use object storage to archive project snapshots, screenshots, and run logs for later review;
    • Use a message queue to handle asynchronous scheduling of batch evaluation tasks;
    • Use a log service and monitoring/alerting to track changes in elapsed time, success rate, and retry rate.

    7. Conclusion

    This “reversed” experiment reinforces a point once more: in Agent scenarios, a model’s capability positioning and its real-world efficiency are not the same thing. The final screenshots can be identical, yet behind them may lie a 17-minute vs. 88-minute difference. Only by fixing the execution framework and then breaking down and quantifying action rounds, tool calls, retry strategy, and context management one by one can we see where the gap truly comes from—and choose the combination of models that genuinely fits the task at hand.