Benchmarking AI Agents in Complex Creative Collaboration Systems: Speed, Reliability, and Recovery

    How should teams evaluate AI agents on complex product work when generation speed is only one part of efficiency? This article describes a controlled benchmark in which four large-scale models used the same agent orchestration framework, task, assets, context, browser environment, and acceptance rules. All four completed a multi-role creative collaboration system, but their execution paths differed sharply: one finished in about 17 minutes, one required a full rerun and took about 72 minutes, one completed in about 88 minutes with three internal automatic retries, and one required adjusted backoff after two failed default retries, reaching first success in about 64 cumulative minutes. The article examines planning, tool use, retries, state consistency, cross-device validation, and error recovery, then proposes a reusable evaluation framework that considers delivery quality and process evidence—not only final screenshots or model speed.

    Why Model Speed Needs a New Lens

    In evaluations of intelligent coding and agent systems, the metric most likely to attract attention is “how long does it take to generate a result?” For complex product tasks, however, a single response speed is only part of efficiency. A model may generate code quickly, but if it calls tools frequently, repeatedly modifies files, fails to recognize runtime errors, or must restart the entire task after a failure, the total elapsed time can still increase significantly.

    This experiment therefore did not change the agent orchestration framework. Instead, it fixed the task, materials, context, browser environment, and acceptance rules, and changed only the underlying model. The goal was to observe how different models perform under the same execution conditions. All test subjects were current models at the commonly used large-parameter scale. The focus was not on comparing marketing claims, but on analyzing how the models behave when completing a genuinely complex task.

    The final results showed that all four model groups completed the task, but their execution processes differed significantly:

    • The first group completed in about 17 minutes.
    • The second group failed on its first execution and completed only after a rerun, for a total time of about 72 minutes.
    • The third group triggered three internal automatic retries during a single task round, for a total time of about 88 minutes.
    • The fourth group failed twice under the default retry policy, completed after increasing the backoff interval, and reached its first successful completion in about 64 cumulative minutes.

    These data show that the real efficiency of an AI agent cannot be measured only by model generation speed. It must also account for task planning, tool use, exception recovery, and execution stability.

    Test Task: Building a Multi-Role Creative Collaboration System

    The task was not simply to generate a static page. The model had to start from limited input and build a complete creative studio collaboration system. The initial input provided only a small set of open image assets, project member information, asset data, version records, and seed comment data. No ready-made page was available to copy.

    The model had to implement the following core states independently:

    1. Project overview
      Display the project name, current stage, participating members, asset count, version progress, and pending items so that users can quickly understand the overall project status.

    2. Inspiration board and asset organization
      Users need to browse image assets, add assets to an inspiration collection, and reorder them by drag and drop. The page must not only display images but also show the organizational relationships among assets.

    3. Asset-anchored comments
      Comments cannot remain at the level of ordinary text. They must be associated with a specific asset or a position on the canvas. Users should be able to tell clearly which image, region, or design element a comment targets.

    4. Version approval
      The system must support switching between multiple design versions and allow different roles to approve, return, and add feedback. State changes must be correctly read by later pages.

    5. Publish preview
      Users need to review final content before publication, confirm the current version, comment status, and approval results, and avoid exposing incomplete work directly to external parties.

    The system also includes three roles: creative director, designer, and client. Each role must reflect different priorities and permissions. For example, designers focus more on assets and version editing, clients focus more on preview and feedback, and creative directors need to track overall progress and approval status.

    The Importance of Unified Experimental Conditions

    To make the comparison as fair as possible, the test used a fixed context capacity, task budget, browser environment, and automation approach. Desktop checks used a resolution of 1440×1000, and mobile checks used 390×844. Each group was also required to preserve the initial screenshot, final screenshot, run logs, review results, and metric records.

    The task further limited each group to only one repair round. This constraint is important because it prevents a model from hiding early planning problems through unlimited iterative fixes. It also comes closer to real-world delivery conditions: the model must identify problems, determine priorities, and complete repairs within a limited time rather than trying indefinitely.

    The unified acceptance criteria mainly covered the following:

    • Whether the pages cover all core flows.
    • Whether the desktop and mobile layouts are usable.
    • Whether interactions form complete end-to-end workflows.
    • Whether data persists after the page is refreshed.
    • Whether different roles show reasonable interface differences.
    • Whether version, comment, and approval states remain consistent.
    • Whether there are obvious layout overflows, blank areas, or inoperable controls.
    • Whether code, screenshots, logs, and review materials are fully retained.

    This evaluation method is closer to practical engineering than simply observing generated code, because it focuses not on whether a page has been written, but on whether a verifiable, operable, and reproducible product can be delivered.

    Execution Differences Across the Four Model Groups

    1. Fast Finisher: Simple Planning and a Shorter Action Chain

    The first model group completed the task in about 17 minutes and used relatively few action rounds. It did not over-split pages or repeatedly consume time on local details. Instead, it first established the core data structure and then quickly completed the main pages and interactions.

    From the final result, this type of model usually has three characteristics:

    • It forms the overall page structure early.
    • Its prioritization of task goals is relatively clear.
    • When it finds a problem, it tends to make local repairs rather than broad refactoring.

    The final interface used a lightweight editing-workbench style, with project status, assets, and member relationships presented intuitively. Although the visual presentation was not the most complex, the information hierarchy was clear, and users could quickly find the current project status.

    This indicates that in complex tasks, “reducing ineffective actions” is often more important than “trying to generate every detail in one pass.” A model that can control its number of action rounds may deliver faster even if its generated content is relatively restrained, compared with a model that makes frequent modifications.

    2. First-Run Failure: Code Generation Is Not Execution Stability

    The second model group failed on its first execution and completed the task only after a rerun, for a total time of about 72 minutes. The key issue here was not whether it could eventually write the page, but that it did not form a stable execution loop during the first run.

    Common causes of such failures include:

    • Incorrect judgment of the initial directory or file structure.
    • The page was generated, but the startup process was not executed correctly.
    • The automation did not wait for asynchronous states to complete.
    • Follow-up operations that depended on page state were triggered too early.
    • After a runtime error appeared, the model did not immediately locate the root cause.

    These problems show that a model’s ability to generate code does not mean it has mature engineering execution capability. In an agent system, the model must handle files, commands, browsers, runtime states, and acceptance feedback at the same time. A lack of judgment in any one link can cause an entire task round to fail.

    If the system uses a strategy of fully rerunning after failure, the time cost will be further amplified. A rerun does not merely regenerate the page. It also reinitializes the environment, reloads context, re-executes tool calls, and takes on the risk of failing again.

    3. Internal Retry: No Full-Round Failure, but Higher Tool-Call Cost

    The third model group did not restart the entire task. Instead, it completed three automatic retries during execution, for a total time of about 88 minutes. Compared with a full rerun, this mechanism reduces environment initialization cost, but it exposes another problem: the model repeatedly attempted local steps, resulting in a longer action chain.

    Automatic retries are suitable for handling transient errors, such as a page that has not yet loaded, a delayed service response, or a single operation timeout. However, if the failure is caused by a logic error, repeating the same action usually does not solve the problem and instead increases resource consumption.

    A retry mechanism therefore needs to distinguish between two types of failures:

    Recoverable failures:

    • Brief network or service delays.
    • Page loading that has not yet completed.
    • A single operation timeout.
    • Abnormal browser focus or window state.

    Failures that cannot be resolved by repetition:

    • Incorrect judgment of a selector or page structure.
    • Inconsistent data field names.
    • Missing state-transition logic.
    • A mismatch between permission checks and page interactions.
    • Syntax or type errors in the code itself.

    A high-quality agent system should reanalyze the error information after each failure instead of mechanically repeating the previous action.

    4. Backoff Recovery: Default Policies Need Dynamic Adjustment

    The fourth model group failed twice in a row under the default retry policy and completed the task only after increasing the backoff interval, with about 64 cumulative minutes to its first successful completion. This process shows that retries require not only a limit on attempts but also consideration of time intervals and error types.

    When the underlying service experiences temporary congestion, rapid consecutive retries may cause more serious contention. Appropriately extending the waiting time can instead make stable results more likely. However, a backoff strategy cannot be extended indefinitely, or the task will fall into inefficient waiting.

    A more reasonable approach is:

    1. After the first failure, wait briefly and retry.
    2. After the second failure, check whether the error is repetitive.
    3. If it is a transient error, use an increasing waiting interval.
    4. If it is a logic error, stop retrying immediately and begin diagnosis.
    5. After reaching the limit, save the current state to avoid losing completed work.

    Such mechanisms are especially important for AI agents because agent tasks often last a long time, and a failure does not necessarily mean that all results are invalid. The system should preserve generated files, screenshots, and logs as much as possible and support continuing from an intermediate state.

    Why Final Screenshots Do Not Represent Full Capability

    If only the final screenshots of the four model groups were compared, the differences might not be obvious. All of them completed pages and had at least basic visual presentation. However, real development quality also includes many things that screenshots cannot show:

    • Whether the page underwent multiple ineffective refactors.
    • Whether data still exists after refresh.
    • Whether operations truly change business state.
    • Whether comments are associated with the correct assets.
    • Whether the mobile version is merely a scaled-down desktop layout.
    • Whether a recoverable state is preserved after failure.
    • Whether the code structure is easy to maintain.
    • Whether logs can explain why a problem occurred.

    Therefore, when evaluating complex AI agents, both process evidence and outcome evidence should be preserved. Process evidence includes tool-call records, retry counts, waiting time, error messages, and repair actions. Outcome evidence includes the final page, interaction verification, state persistence, and cross-device screenshots.

    A Reusable Evaluation Metric Framework

    To avoid drawing conclusions solely from total elapsed time, a more complete metric framework can be established.

    1. Completion Time

    Record the total time from task start to final acceptance, and break it down into:

    • Model thinking time.
    • File modification time.
    • Tool-call time.
    • Browser waiting time.
    • Retry and recovery time.

    2. Action Efficiency

    It is possible to count the total number of action rounds required to complete the task, the proportion of effective actions, and the proportion of repeated actions. Fewer action rounds are not necessarily better, but a large number of repeated operations usually indicates insufficient planning or diagnostic capability.

    3. Stability

    Focus on:

    • First-run success rate.
    • Number of full reruns.
    • Number of local retries.
    • Number of unhandled exceptions.
    • Recovery capability after task interruption.

    4. Functional Completeness

    Check whether all task requirements are covered, including pages, roles, interactions, data states, approval workflows, and publish preview.

    5. Cross-Device Adaptability

    Desktop and mobile must be accepted separately. On mobile, pay particular attention to content folding, touch-target size, horizontal overflow, bottom action bars, and usability of overlays.

    6. State Consistency

    Check whether data remains consistent by refreshing pages, switching roles, switching versions, and re-entering the project. For collaborative systems, state consistency is often more important than visual effects.

    7. Maintainability

    Evaluate whether the code contains obvious duplication, whether component boundaries are reasonable, whether the data model is clear, and whether it is easy to add new roles or states later.

    Implications for Real-World Applications

    The biggest conclusion from this experiment is that a model’s “low cost” or “high speed” describes only part of its capability and cannot directly represent final production efficiency.

    In real projects, improvements can be considered in the following directions:

    1. Define acceptance criteria before starting the model
      Write functional, visual, state, and device requirements as checkable items to reduce the model’s misunderstanding of the goal.

    2. Set stage checkpoints for the agent
      Establish checkpoints between the page skeleton, core interactions, data persistence, and final acceptance to prevent errors from accumulating until the end.

    3. Distinguish error types
      Use retries for transient failures and diagnosis for logic failures. Do not apply the same recovery strategy to every error.

    4. Save intermediate artifacts
      Code, screenshots, logs, and review results should be preserved continuously so that a failed task can resume from the most recent state.

    5. Include process metrics in model selection
      Do not look only at the final result. Also consider action rounds, first-run success rate, retry cost, and state consistency.

    6. Match models to task types
      For tasks with clear structure and tight delivery timelines, prefer models with shorter action chains. For complex exploratory tasks, focus more on diagnostic and recovery capability.

    Conclusion

    The intelligent construction of complex front-end products is not essentially a code-generation contest. It is a complete engineering process involving planning, execution, validation, repair, and delivery. A controlled cross-model test under a unified agent framework shows that what often separates models most clearly is not whether they can eventually generate a page, but whether they can complete the entire action chain reliably.

    When evaluating intelligent coding systems, the question should therefore shift from “how fast can the model write” to “can the model deliver a verifiable product reliably with fewer ineffective actions?” Only by placing speed, reliability, tool use, error recovery, and final quality in the same metric system can conclusions that are closer to real production environments be reached.