Benchmarking Complex AI Agents: How Model Speed and Stability Affect Delivery Efficiency
This article reports a controlled comparison of four AI models on the same complex product-building task. All four completed the work, but completion times ranged from about 17 minutes to more than an hour, depending on failures, retries, and waiting time. The article explains why end-to-end delivery time, not response speed alone, should guide how teams evaluate AI agents. It also outlines a practical workflow covering task-state design, verifiable acceptance checks, retry limits, and metric tracking.
In complex software development, model parameter count, single-response speed, and price explain only part of how well an AI agent performs. When a model must repeatedly call a browser, read and write files, interpret visual assets, modify code, and fix problems based on acceptance results, delivery efficiency is usually determined by four things together: model capability, tool coordination, failure recovery, and context management.
This article compares four AI models in a horizontal experiment run under uniform conditions. The experiment kept the execution framework, task content, assets, context window, browser environment, and acceptance rules unchanged. Only the underlying model was swapped, so we could observe how the models differ in practice on a complex product-building task.
1. Experiment Goal: From Model Responses to Task Delivery
Traditional model evaluations usually focus on a few metrics: whether the answer is correct, whether generation is fast enough, and whether the code runs. In agent development, however, a model does not produce its result in one step. It goes through a continuous loop:
- Understand the requirements and constraints;
- Check the existing directories, assets, and runtime environment;
- Design the page structure and interaction flow;
- Write or modify the code;
- Start the application and verify it in a browser;
- Locate problems based on screenshots, logs, and test results;
- Apply fixes and run acceptance again;
- Output the code, screenshots, review records, and process metrics.
As a result, the total time a task takes is not the same as the time the model spends generating code. A more accurate way to calculate it is:
Total delivery time = planning time + code generation time + tool call time + waiting time + retry time + repair time + acceptance time
If a model’s first page looks good but its tool calls fail often, its real delivery efficiency may be low. Conversely, a model whose first output is unremarkable may still have an advantage on the full task if it runs stably and corrects problems quickly.
2. Unified Experiment Setup and Task Design
This experiment used the same agent development framework. The task, input assets, and acceptance criteria stayed identical, and only the underlying model changed. This design removes framework differences as much as possible, so that tool orchestration is not mistaken for model capability.
1. Task Content
The test task was a creative studio collaboration system. Models had to build a complete product interface from scratch rather than replicate an existing page. The inputs included:
- Six openly licensed images about urban culture;
- Project member information;
- Asset metadata;
- Version records;
- Initial comments and approval data;
- Product roles and interaction rules.
The final system had to cover five core states:
- Project overview;
- Inspiration asset wall;
- Asset positioning and comments;
- Version approval;
- Release preview.
The system also had to support three identities: creative director, designer, and client. Each identity had different permissions on the page, such as viewing projects, submitting versions, adding comments, approving content, or returning content for revision.
