Benchmarking Multiple Models in a Unified Agent Harness: Speed, Retries, and Stability
This article reports a controlled experiment comparing multiple models under the same agent harness, task, context, browser environment, and acceptance rules. All tested models completed the task and passed screenshot acceptance, but their execution patterns differed sharply: one-pass completion, full rerun after a first-round failure, internal local retries, and backoff-based convergence. The article analyzes wall-clock time and retry behavior, proposes a five-dimension evaluation framework for agent-ready models, and offers engineering recommendations for checkpointing, layered retry design, and success-time-based evaluation.
Benchmarking Multiple Models in a Unified Agent Harness: Speed, Retries, and Stability
In the previous experiment, I fixed the model and swapped four different agent orchestration frameworks to verify the large impact of the harness itself on task completion efficiency: the same task could take several times longer under different frameworks. This time, I reversed the variable: fix the framework and replace only the model, to see where the real performance differences lie under an identical execution environment.
1. Experimental Design: Why Variables Must Be Controlled
The biggest pitfall in agent task evaluation is failure to isolate variables. If task descriptions differ, context differs, or acceptance criteria differ, comparisons between models lose meaning. Therefore, this experiment imposed strict constraints:
- Harness fixed: the same agent orchestration system was used throughout, including task planning, tool invocation, browser operations, error retry, and all other stages.
- Task fixed: a medium-to-complex creative collaboration system development task, including page generation, interaction logic, and data organization.
- Materials and context fixed: prompts, reference materials, and injected project context were completely identical.
- Browser environment and acceptance rules fixed: screenshot acceptance criteria and the page rendering environment were not adjusted.
The only independent variable was the underlying model. The selected models were all below the 300B parameter scale, plus one long-context model as a cross-reference. Models at this scale are currently a mainstream choice for agent applications, so the results are highly relevant.
It is important to emphasize that all four models ultimately completed the task and passed screenshot acceptance. The real difference was not “whether it could be done,” but “how it was done”—and that is the most valuable part of this experiment.
2. Four Results: Execution Pattern Differences Behind the Numbers
| Execution Path | Time Performance |
|---|---|
| Model A (DeepSeek-class) | Passed on the first attempt, completed in about 17 minutes |
| Model B (GLM-class) | First round failed; passed after a full rerun; cumulative time about 1 hour 12 minutes |
| Model C (Qwen-class) | No full rerun; 3 internal local automatic retries; execution time 88 minutes |
| Model D (Kimi-class) | Default retry policy failed twice consecutively; completed after extended backoff; cumulative time to first success about 1 hour 04 minutes |
At first glance, these numbers simply look like “fast versus slow,” but decomposition reveals distinct failure-and-recovery patterns:
1. One-Pass Completion: High Determinism Leads to Low Total Time
Model A required no retries and reached the acceptance standard directly in 17 minutes. Its value is not only speed, but also determinism of the execution path: tool-call parameters were highly accurate, planning granularity was reasonable, and frequent backtracking was unnecessary. For pipeline-style batch tasks, this determinism translates directly into cost advantages: fewer tokens consumed, fewer API calls, and less queueing time.
2. Full Rerun: Failure Cost Is Exponentially Amplified
Model B’s problem was that some stage in the first round failed acceptance, triggering a full rerun. This indicates a relatively high single-round error rate, while the harness-level recovery strategy was “start over” rather than “resume from checkpoint.” This exposes an important engineering issue: if the harness does not implement step-level checkpoints, a local failure incurs the cost of an entire round. Time that might have been 20 minutes expanded to 72 minutes, an inflation factor of more than 3x.
3. Internal Retry: Stability Traded for Time
Model C did not trigger a full rerun, but three local automatic retries occurred during execution, for a total time of 88 minutes. This is a typical “slow but stable” pattern: single-step output was slower or occasional tool-call failures occurred, but the harness’s inner retry mechanism absorbed these problems and avoided catastrophic full-round failure. Its total time was the longest, but its failure risk was the lowest.
4. Backoff Convergence: Retry Strategy Determines the Upper Bound
Model D’s behavior is the most interesting: under the default retry policy it failed twice consecutively, and only converged after switching to extended backoff (exponential backoff). This indicates that some failures were not model-capability problems, but transient server-side instability or rate limiting. Of the cumulative 1 hour 04 minutes, a considerable portion was spent waiting and retrying rather than on actual computation.
3. An Agent Efficiency Evaluation Framework Abstracted from the Experiment
Summarizing these four patterns, I believe that evaluating whether a model is suitable for agent scenarios should cover at least five dimensions, not just “how high the benchmark score is”:
- Action rounds: how many planning-execution cycles are needed to complete the same task. More rounds mean greater cumulative error and cumulative latency.
- Tool-call accuracy: the compliance rate of function parameter generation and call format directly determines retry frequency.
- Service stability: rate limiting, timeouts, and transient error rates. These non-capability factors may affect end-to-end time more than the model itself.
- Context management capability: whether the model can maintain consistency with early decisions in long tasks and avoid later steps overturning earlier results.
- Fit with harness recovery mechanisms: whether a model is better suited to full reruns or local retries should, in turn, guide the design of the harness retry strategy.
A simplified evaluation approach can be expressed in pseudocode:
def evaluate_agent(model, task, harness):run = harness.execute(model, task)return {"wall_time": run.total_time, # End-to-end time"action_rounds": run.rounds, # Number of action rounds"retry_count": run.retries, # Retry count and level"failure_mode": run.failure_pattern, # Local failure or full-round failure"passed": run.accepted # Whether acceptance passed}
The key is not a single metric, but the combination of these metrics: low time plus zero retries is the ideal state; high time plus zero full-round failures indicates stable and dependable execution; low time but a high full-round failure rate requires caution—its average time, including reruns, may not be low at all.
4. Engineering Recommendations
Combining the two experiments—swapping frameworks and swapping models—I have several recommendations for building production-grade agent systems:
- First fix the harness to test models, then fix the model to tune the harness. Isolate the two variables in turn so that the contributions of model capability and orchestration design can be seen separately.
- Implement step-level checkpoints and resume-from-checkpoint execution to prevent local failures from expanding into full reruns. This is often the largest hidden time cost.
- Design retry strategies in layers: use short-interval fast retries at the tool-call layer, exponential backoff at the service layer, and consider backtracking or replanning only when the planning layer fails.
- Establish an evaluation standard based on “average successful time” rather than “best time,” counting retries, waiting, and reruns fully into the cost.
- A low-cost or high-speed model positioning describes only part of its capability. Real agent efficiency is the result of coupling among model capability, service stability, and orchestration strategy.
Conclusion
This experiment once again confirms a conclusion: model selection cannot rely only on parameter count, price tag, or single-point benchmark scores. The same model with a different orchestration strategy can produce vastly different results; the same orchestration with a different model can produce completely different failure modes. Only by turning evaluation into controlled-variable experiments and breaking down each failure into attributable patterns can we truly answer the question, “Which model is suitable for my agent scenario?” In follow-up work, I plan to continue validating this evaluation framework on longer multi-turn tasks.
