Evaluating Model Efficiency and Stability for Complex AI Agent Tasks
A reliable evaluation of AI agents must go beyond the final screenshot. This article presents a controlled comparison of four models completing a creative studio collaboration system, examining task quality, execution time, retries, tool usage, state persistence, error recovery, delivery completeness, and human intervention. The findings show why first-success time, recovery behavior, and end-to-end delivery cost are critical when selecting agents for complex software engineering workflows.
1. Why the Final Page Is Not Enough
In agent-assisted development, a final screenshot usually shows only whether a task was completed. It does not fully represent the model’s actual capabilities. Two models may produce visually similar pages while differing significantly in execution rounds, tool-call volume, error recovery, state persistence, and context management.
Agent evaluation should therefore not focus only on outcome-based metrics such as:
- Whether the page renders successfully
- Whether core functions can be clicked
- Whether the final visual result meets expectations
It should also examine:
- The time from task initiation to the first successful result
- Whether an entire execution cycle had to be rerun
- Whether tool calls were frequent and effective
- Whether the agent could locate problems independently after an error
- Whether context growth reduced execution quality
- The number of fixes and instances of human intervention
- Whether the deliverables were complete, including code, screenshots, logs, and review records
In engineering practice, overall agent efficiency is closer to the following relationship:
Effective delivery efficiency = task completion quality ÷ total time, retry cost, and human intervention cost
This means that a low-latency model does not necessarily produce higher project efficiency. Likewise, a larger model does not automatically deliver better real-world results.
2. A Unified Test Environment and Task Definition
To prevent environmental differences from affecting the results, the experiment used the same agent execution framework for all tests and changed only the underlying model. The task, assets, context capacity, browser interaction method, acceptance criteria, and output requirements remained unchanged.
The test task was a collaboration system for a creative studio. Its input data included several openly licensed urban culture images, project member information, asset records, version history, and initial comment data. Because there was no existing page that could be directly replicated, the models had to design the information architecture, interaction model, and frontend implementation independently.
The product needed to support five core areas:
Project overview
Display project progress, participating members, the current version, pending items, and recent activity so users could quickly understand the project’s current stage.Inspiration asset board
Support image browsing, grouping, filtering, and drag-and-drop organization. Assets needed to be clearly linked to comments, projects, and versions rather than displayed in isolation.Asset-based comments
Allow users to comment on a specific asset or local region. Each comment needed to show its author, timestamp, status, and related location so that feedback remained connected to its context.Version approval
Support version switching, change review, approval, and rejection while clearly communicating the current approval status.Publishing preview
Provide a preview that closely resembled the final presentation while retaining project identity, version information, and necessary status indicators.
The system also defined three roles: creative lead, designer, and client. Each role had different permissions, with visible differences in viewing, editing, commenting, submitting, and approving actions.
Acceptance criteria covered more than visual quality. They also included interaction completeness and state reliability. For example:
- Data could not be lost after a drag-and-drop operation.
- Comments had to remain associated with the correct assets.
- Page content had to update consistently after switching versions.
- Key state had to persist after a browser refresh.
The experiment covered both desktop and mobile viewport sizes. Desktop testing focused on complex information layouts, while mobile testing examined responsive structure, touch-target sizes, and content prioritization. Each run preserved the first version of the result and allowed only one repair cycle, approximating a real development process with limited opportunities for correction.
3. Execution Performance of the Four Models
To avoid turning the evaluation into product promotion or a ranking exercise, the four test subjects are referred to as Model A, Model B, Model C, and Model D.
Model A: Fastest Completion with a Short Execution Path
Model A completed the task fastest, in approximately 17 minutes. It used a relatively restrained page structure, placing project status, asset relationships, and member information into a clear hierarchy. The interface followed a light editor-workspace style, with limited visual decoration and well-controlled information density.
Model A’s main strengths were visible in three areas.
First, its action path was short. Rather than spending many rounds refining local visual details, it completed the information architecture, primary interactions, and state flow first, followed by limited adjustments.
Second, it produced a runnable version early. For complex tasks, obtaining an operable initial system quickly allows the agent to identify real problems through browser inspection instead of remaining in a prolonged code-reasoning phase.
Third, its repair strategy was focused. Model A did not repeatedly modify unrelated areas. It concentrated validation on the core workflows, which kept total execution time low.
Model B: Strong Visual Expression but a Costly Full Rerun
Model B did not succeed on its first execution. Including the failed rerun, its cumulative execution time was approximately 1 hour and 12 minutes. Its visual design was more expressive, combining dark and warm colors in an editorial style. Typography, the permission matrix, and approval states reinforced the overall theme.
However, strong visual expression also introduced additional complexity. During the initial execution, Model B may have allocated too many resources to page decoration, hierarchy presentation, and local layout details. As a result, it did not validate the critical interaction paths early enough. Once structural issues appeared in the first run, the model had to rely on a complete rerun to recover.
This result shows that aesthetic capability and engineering delivery capability do not always progress together. A page may look polished, but if its state management, permission logic, or mobile layout is defective, the overall task should not be considered successful.
Model C: Multiple Automatic Retries and Growing Uncertainty
Model C did not perform a full rerun. Instead, it triggered three automatic retries during execution and eventually completed the task in approximately 88 minutes.
Compared with a direct failure, this approach can preserve earlier work and reduce the waste of rebuilding from scratch. However, each retry may also increase context length, duplicate operations, and state uncertainty.
When automatic retries have no clear boundaries, systems may encounter problems such as:
- Writing the same data repeatedly
- Modifying the same component multiple times and making the code structure more complex
- Keeping both old and new approaches in the context
- Making it difficult for the model to determine which issues have already been fixed
- Creating inconsistencies between the actual browser state and the expected code state
Retries are therefore not inherently equivalent to reliability. A high-quality retry should include the failure cause, current state, completed steps, outstanding items, and the next validation goal. It should not simply repeat the previous actions.
Model D: Recovery Affected by Backoff and External Tools
Under its default strategy, Model D failed twice in succession. After the backoff strategy was adjusted, it eventually completed the task, with a cumulative time of approximately 1 hour and 4 minutes to first success.
This process shows that, beyond model capability, service stability, request pacing, tool responses, and error recovery mechanisms can significantly affect final delivery time.
When an agent relies on a browser, file system, or other external tools, an execution failure may not result from a reasoning error. It may instead be caused by:
- A tool response timeout
- A page that has not finished loading
- An incomplete resource write
- A synchronization issue between frontend and backend state
- Excessive request frequency
- Retry intervals that are too short
These factors should be recorded separately during evaluation. Otherwise, infrastructure fluctuations may be incorrectly interpreted as insufficient model capability.
4. Evaluation Metrics That Matter
To make results more comparable, organizations should establish a layered metric system.
4.1 Outcome Quality
This layer includes:
- Functional pass rate
- Visual consistency
- Responsive adaptation
- Correct role permissions
- State persistence
4.2 Execution Efficiency
This layer includes:
- Total execution time
- Time to the first runnable version
- Number of tool calls
- Number of code modifications
- Number of browser inspections
4.3 Stability
This layer includes:
- First-run success rate
- Failure types
- Number of automatic retries
- Number of full reruns
- Recovery success rate
4.4 Delivery Completeness
Evaluation should cover more than the page itself. It should verify:
- Whether the code runs successfully
- Whether logs are complete
- Whether screenshots cover key states
- Whether review records explain the reasons for changes
4.5 Human Cost
Teams should record:
- Whether human intervention was required
- How many times people intervened
- What each intervention resolved
- Whether the model could continue working after receiving human feedback
Among these metrics, time to first success is particularly important. Total execution time may be strongly affected by the retry strategy, while first-success time better reflects the model’s ability to build a usable system on its initial attempt.
For production environments, teams should also track effective delivery cost per unit of quality: how much time, how many calls, and how many computational resources the model consumes to complete a task of equivalent quality.
5. Engineering Lessons from the Experiment
First, “complete the loop before polishing” should be a basic strategy for complex tasks. The agent should prioritize data structures, page routing, key interactions, and state persistence before refining visual details. If it becomes absorbed in styling early, it may not have enough time left to address core functionality.
Second, tasks should be divided into verifiable stages. For example:
- Stage 1: Establish page structure and basic data
- Stage 2: Complete role, version, and comment workflows
- Stage 3: Verify state recovery after a refresh
- Stage 4: Adapt the interface for desktop and mobile
- Stage 5: Perform final acceptance testing and archive screenshots
Each stage should have explicit checkpoints so that issues do not accumulate until the final step.
Third, retry mechanisms need context compression and state isolation. Before retrying, the system should generate a concise summary that clearly lists completed work, failure causes, and items requiring validation. Irrelevant operation history should be removed. For operations with side effects, idempotent design should be used to prevent duplicate execution from creating inconsistent data.
Finally, the evaluation environment should resemble a real production workflow as closely as possible. Generating a single static page does not demonstrate that an agent can handle complete product development. Only by evaluating browser operations, data state, error recovery, version approval, mobile adaptation, and delivery review within the same workflow can teams determine whether a model is suitable for complex engineering tasks.
6. Conclusion
The central conclusion of this comparison is not simply which model was fastest. A model’s practical value depends on several factors working together: reasoning quality, action planning, tool use, context management, error recovery, service stability, and acceptance procedures.
Under the same task and execution framework, all four models ultimately completed the objective, but their execution paths differed considerably. One model closed the loop quickly with relatively few steps. Another depended on a full rerun. A third gradually converged through multiple internal corrections. The fourth was significantly affected by backoff strategy and the stability of external tools.
Enterprise model selection should therefore not rely only on parameter scale, single-response latency, or the final screenshot. A more appropriate approach is to establish a comprehensive evaluation system covering quality, efficiency, stability, and human cost, then continuously collect data through repeatable tasks, standardized environments, and explicit acceptance rules.
For teams planning to deploy agent applications in cloud environments such as Baidu AI Cloud, it is also advisable to prioritize standardized evaluation workflows, log collection, state tracking, failure classification, and automated acceptance testing. Only when every execution is observable, reviewable, and comparable can model capability be consistently converted into engineering productivity.
