Evaluating Multiple LLMs for AI Agents with a Fixed Execution Framework: Engineering Practices and Findings
This article presents a controlled experiment comparing four low-cost, mid-size LLMs as the foundation for AI agent workflows. By fixing the execution framework, tasks, assets, browser environment, and acceptance criteria—varying only the underlying model—the author isolates model-level differences in agentic performance. The article covers experiment design, results analysis (completion time, retries, action rounds), and a five-step methodology teams can use to run reproducible model evaluations, concluding that real agent efficiency depends on action-round effectiveness, tool-call quality, service stability, and context management rather than benchmark scores or nominal context length.
1. Background: Why Fix Variables for Cross-Model Evaluation
As AI agents move into production engineering, more teams face the same selection question: which underlying LLM should drive their automated development workflows?
Benchmark scores alone are almost useless for answering this. A widely observed industry phenomenon is that models with similar scores on standard benchmarks can perform dramatically differently once they enter real-world scenarios involving multi-turn tool calling, long-context interaction, browser operations, and deliverable acceptance. In my previous comparison, I fixed the model and only swapped the execution framework (harness). The conclusion was that different frameworks produce huge efficiency differences—the fastest one completed a complex creative collaboration system build.
This round of experiments does the opposite: I locked down the framework, tasks, assets, context, browser environment, and acceptance rules, changing only the underlying model. This follows the basic principle of controlled experiments: only when a single independent variable changes can result differences be attributed to that variable. Otherwise, any timing comparison is contaminated by the framework itself.
2. Experiment Design: Four Low-Cost Models in a Similar Parameter Tier
The four models in the comparison all sit in the “small-to-mid parameter, speed- and cost-focused” tier, with one long-context variant included as a horizontal reference. The selection logic has two parts:
- Only same-tier comparisons are meaningful. Including a flagship large-parameter model would drown out the variable and fail to answer the more practical engineering question: “among low-cost models, which is the better agent foundation?”
- Cost-sensitive scenarios are the mainstream demand. Most enterprise development pipelines need high-frequency, batchable agent calls, where cost and latency are hard constraints.
For the task, I chose a creative collaboration system with complex interaction logic: multi-page orchestration, linked component states, and visual fidelity to design specs. Acceptance rules were fixed in advance, covering three categories of hard checks—structural completeness, visual consistency, and functional usability—to avoid subjective “looks close enough” judgments.
Variables worth calling out: all tasks shared the same asset package and context summary, browser environments were isolated, and the acceptance script was executed by the same program. In other words, the framework layer provides a unified action loop, tool set, and failure-retry mechanism; the models compete on “how well they play the same hand.”
3. Results: Elapsed Time Is the Surface; Process Data Is the Truth
All four models eventually passed acceptance, but the process differences were striking:
- Model A: passed on the first attempt in about 17 minutes, with no full-run re-executions.
- Model B: failed the first acceptance round and only passed on a second full rerun, totaling about 1 hour 12 minutes.
- Model C: no full rerun, but the framework triggered 3 internal automatic retries during execution, totaling about 88 minutes.
- Model D: failed twice consecutively under the default backoff policy; after adjusting backoff parameters and the retry budget it completed, with about 1 hour 04 minutes accumulated to first success.
Translating this scorecard into engineering terms yields four key signals.
3.1 The Hidden Cost of Action Rounds
The fastest model finished in 17 minutes. Its core advantage was not “generation speed” but decision accuracy—nearly every round’s action plan converted directly into effective tool calls, with no dead ends and backtracking. Each additional action round brings extra context accumulation, token consumption, and queuing latency. The first-order metric of agent efficiency is “effectiveness per action round,” not the fluency of single-turn output.
3.2 Retries Amplify Capability Gaps
Both Model B and Model D relied on retries to finish, but for different reasons. The former made directional errors (the first build deviated from acceptance goals), indicating weak planning capability; the latter leaned toward service stability issues and output degradation caused by context length. Engineering practice must distinguish these two failure types: planning failures require better task decomposition and self-check loops, while stability failures require more robust server-side guarantees and backoff strategies.
3.3 Framework Defaults Are Not Universal
Model D’s case deserves special attention: it failed repeatedly under the default retry policy and only succeeded after widening backoff intervals and relaxing the retry budget. This shows that a framework’s fault-tolerance strategy is coupled with model characteristics—the same default parameters affect different models’ success rates completely differently. When selecting a stack, don’t just ask “can the framework run it”; ask “which model was the framework tuned for.”
3.4 Long Context Does Not Mean Long-and-Effective
The long-context reference model gained no overwhelming advantage from its larger context capacity. This confirms the industry consensus: effective context utilization—precisely locating information across tens of thousands of words without being derailed by intermediate noise—matters far more than the nominal window length. For long tasks, context management strategy (when to compress, what history to keep, how to retrieve) is often a more decisive variable than the model’s advertised capability.
4. Methodology: How to Run a Reproducible Cross-Model Evaluation
If you want to implement a similar evaluation in your own team, the following five steps are general best practices distilled from this experiment:
Step 1: Lock the framework before swapping models. First, fix an execution framework you’re comfortable with and run the full pipeline end to end. Confirm the task itself is completable and acceptance is automatable before starting variable substitution. This step eliminates 90% of confounding factors.
Step 2: Design “hard acceptance,” not “soft comparison.” Replace human “feels about right” judgments with programmatic checks (structure, styling, functionality), adding screenshot-level visual verification when necessary. A comparison without hard acceptance produces non-reproducible conclusions.
Step 3: Record process metrics, not just final states. At minimum, collect: total action rounds, tool call counts, success rate per tool type, number and cause of triggered retries, and cumulative context tokens. Final-state screenshots show the “what”; process data explains the “why.”
Step 4: Fix a task difficulty gradient. Prepare three tiers—“simple fix, medium build, complex system”—and have the same model group pass them level by level. Low-cost models may all pass on small tasks; only complex tasks separate them.
Step 5: Include cost in the conclusion. Real cost is elapsed time × call count × unit price. A “slow but first-attempt-success” model can cost far less overall than a “fast but retry-heavy” one.
5. Conclusion
The most valuable output of this experiment is not a “who’s faster” ranking, but three repeatedly validated engineering judgments:
- Low cost and high speed are only part of a model’s positioning. Real agent-scenario efficiency is jointly determined by action rounds, tool-call quality, service stability, and context management.
- Frameworks and models have an adaptation relationship. Default parameters are assumptions, not facts—retry strategies should be tuned separately for each model during selection.
- Long-context capability and long-task capability are two different things. Context compression and retrieval strategies should be evaluated as independent engineering components.
For teams currently selecting models for agents, we recommend starting with the five-step methodology above: control variables first, apply hard acceptance, then collect process data, and finally use “completion rate × unit cost” as the decision metric. Only after such a rigorous self-built evaluation does model selection stop being guesswork.
