Cross-Model Benchmarking of AI Agent Task Efficiency and Stability

    Model evaluation for complex agent workflows should go beyond parameter size, response speed, and price. This benchmark compares four models in the same agent framework, browser environment, task, inputs, context, and acceptance criteria. Although all four models completed the task, their total execution times, action counts, recovery strategies, retry behavior, and resource usage differed substantially. The results show why real-world agent performance depends on planning, tool use, context management, and failure recovery—not just single-response speed.

    In agent applications, model size, response latency, and price are often the first metrics people examine. However, these indicators do not fully represent practical task efficiency. For complex tasks that require repeated browser interactions, material inspection, page modification, multi-step execution, and final acceptance checks, delivery time is often determined by whether a model can plan reliably, call tools correctly, understand context, and recover effectively after failure.

    This article presents a comparative test of four types of models in a fixed execution environment. The agent framework, task, input materials, context, browser environment, and acceptance criteria remained unchanged; only the underlying model was replaced to reduce interference from other variables.

    All four models eventually completed the task, but their execution paths, action counts, recovery strategies, and total durations differed significantly. The fastest model completed the task in approximately 17 minutes. Another model failed during its first run and required a second execution, resulting in a cumulative duration of about 1 hour and 12 minutes. A third model completed the task through built-in automatic retries in approximately 88 minutes. The fourth model succeeded after multiple default retries and backoff adjustments, taking about 1 hour and 4 minutes in total.

    These results show that model evaluation should not focus only on whether a task can be completed. It should also examine how quickly it is completed, how many actions are required, whether the model can recover from failure, and how much context and tool capacity it consumes.

    1. Why the Agent Framework Must Remain Fixed

    The final performance of an agent task is usually determined by several components:

    • The reasoning and planning capabilities of the underlying model
    • The way the agent framework decomposes tasks
    • Tool-calling protocols and parameter validation
    • The stability of the browser or operating environment
    • Context compression and history-management strategies
    • Retry, rollback, and recovery logic
    • The strictness of the final acceptance criteria

    If both the model and the execution framework are changed at the same time, it becomes difficult to determine whether a performance difference comes from the model or from the framework’s scheduling and orchestration behavior. For this reason, the test used a single-variable design: the execution framework remained unchanged while only the underlying model was replaced.

    This setup is closer to a real engineering question. Development teams typically do not ask only which model is theoretically more capable. They also need to know which model takes fewer unnecessary steps, triggers fewer invalid tool calls, and recovers more easily after failure under the same task, tools, and acceptance conditions.

    2. Why the Test Task Reflects Real Agent Capabilities

    The task was not a single-turn question-and-answer exercise. It combined creative collaboration, page operations, and result verification. The models had to perform several types of work:

    1. Read the task description and background context
    2. Understand the existing materials and their structure
    3. Plan the overall page or system solution
    4. Complete multiple interactions through a browser
    5. Continuously revise the solution based on intermediate results
    6. Handle tool-call failures, page-state changes, and input deviations
    7. Pass a final review based on consistent acceptance criteria

    Compared with traditional text generation, this type of task more closely resembles a real agent workflow. The model must not only think through the solution but also execute it. It must repeatedly cycle through planning, action, observation, and correction.

    For example, a seemingly simple page-editing operation may involve the following hidden steps:

    1. Read the current page state
    2. → Locate the target area
    3. → Determine whether existing content can be reused
    4. → Choose between editing and rebuilding
    5. → Perform the page operation
    6. → Check whether the operation succeeded
    7. → Decide the next step based on the result

    If the model skips state verification and immediately performs an edit, it may target the wrong element. If it does not verify the result promptly, later steps may be based on an incorrect state. If it cannot identify the cause of failure, it may repeatedly follow the same invalid path.

    Therefore, the core of a complex agent task is not the quality of a single response, but the effectiveness of the entire action chain.

    3. Test Results: The Final Outcomes Were Similar, but the Execution Paths Were Not

    All four models met the basic acceptance requirements. However, the process data revealed clear differences in four main areas.

    3.1 Total Execution Time

    The fastest model completed the task in approximately 17 minutes. This suggests that it established an efficient loop between task understanding, step planning, and tool execution. It was not necessarily the fastest in every individual response, but it followed a shorter overall path and required less rework.

    Another model failed during its first run and reached a cumulative duration of about 1 hour and 12 minutes after being run again. The important issue is not simply whether the second run succeeded. The first failure created additional time and resource costs. In human-assisted workflows, a failed run may require rebuilding context, rechecking the environment, or involving an operator.

    A third model took approximately 88 minutes. Rather than fully restarting the task, it performed multiple automatic retries internally. This approach improved the probability of eventual success but extended the execution time. It may be suitable for scenarios that prioritize completion probability over real-time responsiveness, but interactive development workflows may require stricter limits on ineffective retries.

    The fourth model eventually completed the task in approximately 1 hour and 4 minutes after its default retry strategy repeatedly failed. It succeeded by increasing the backoff interval and adjusting its execution pace. This indicates that retry policy is itself an important part of an agent system and should not be attributed entirely to model capability.

    3.2 Differences in Action Count

    In a complex task, each observation–decision–tool-call–verification cycle typically consumes one action round. More action rounds do not necessarily indicate stronger capabilities.

    Well-planned multi-step execution can help a model verify state incrementally and reduce operational errors. However, if many rounds consist only of repeated clicks, repeated reads of the same content, or attempts to follow the same failed path, the model may have weaknesses in planning or error recovery.

    Evaluation should therefore distinguish among:

    • Effective action rounds: Actions that genuinely move the task forward
    • Verification rounds: Actions used to confirm that a result is correct
    • Ineffective rounds: Repeated attempts, incorrect targeting, or waiting without progress
    • Recovery rounds: Corrective actions taken after an abnormal state

    Reporting only the total number of rounds can hide these differences. A more useful metric is the amount of task progress generated by each effective action.

    3.3 Tool-Calling Quality

    In browser and page-based workflows, tool-call errors often affect outcomes more directly than language quality. Common problems include:

    • Selecting the wrong page element
    • Continuing before the page has finished loading
    • Using an operation that does not match the current state
    • Failing to verify whether the previous action took effect
    • Misinterpreting page feedback as task completion
    • Continuing with the original plan after the tool returns an error

    Higher-quality models typically define the objective before calling a tool and verify the result afterward. Lower-quality execution paths often perform consecutive operations without confirming state.

    A tool call can be abstracted as follows:

    1. Identify the objective
    2. → Construct the parameters
    3. → Execute the tool
    4. → Read the result
    5. → Determine the state
    6. → Select the next action

    An error at any stage can trigger a chain of subsequent failures. In practical evaluations, teams should record the total number of tool calls, failed calls, repeated calls, and the proportion of calls followed by state verification.

    3.4 Context Management

    The longer a task runs, the more historical information accumulates. The model must retain the task objective, material constraints, completed steps, current page state, and acceptance requirements at the same time. Poor context management can lead to:

    • Forgetting important initial constraints
    • Repeating steps that have already been completed
    • Confusing an earlier page state with the current state
    • Focusing on a local change while ignoring final acceptance
    • Drifting away from the original plan late in the workflow

    Context management is not only about having a larger memory window. It also requires structuring information into layers. A more reliable approach is to divide context into four categories:

    1. Immutable task constraints
    2. Current execution state
    3. Completed actions and their results
    4. Pending items and known risks

    During long tasks, the agent should periodically generate a state summary instead of retaining all history without distinction. The summary must preserve information needed for decision-making and avoid losing important constraints through excessive compression.

    4. Retries Do Not Automatically Mean Stability

    Many agent systems use automatic retries to improve success rates. However, retries also have a cost. If the model cannot identify the type of failure, the system may simply repeat the same action.

    An effective retry should meet three conditions:

    • It can identify the cause of the previous failure
    • It changes the execution path or adjusts the parameters
    • It stops and reports the issue promptly after reaching a defined limit

    A layered retry strategy can be designed as follows:

    1. First failure: Read the error and confirm the page state
    2. Second attempt: Adjust parameters or relocate the target
    3. Third attempt: Switch to an alternative operation path
    4. Limit exceeded: Save the current state, provide diagnostics, and stop

    For temporary issues such as page loading or network fluctuations, the system can use progressively longer wait intervals. For deterministic errors such as a missing element, insufficient permissions, or a mismatched state, it should not wait blindly. Instead, it should replan.

    In this test, some models completed the task through repeated automatic retries. This shows that retries can improve eventual success rates, but it also highlights two engineering issues: failure detection may not be accurate enough, and retry limits and stopping conditions need to be defined more clearly.

    5. Building a More Reliable Model Evaluation System

    If testing records only whether the final task was completed, the conclusions will be too coarse. At minimum, evaluations should track the following metrics.

    5.1 Outcome Metrics

    • Whether the task passed final acceptance
    • Whether all hard constraints were satisfied
    • Whether manual correction was required
    • Whether the final output contained obvious defects

    5.2 Efficiency Metrics

    • Total execution time
    • Number of effective action rounds
    • Total number of tool calls
    • Average time per round
    • Task progress per unit of time

    5.3 Stability Metrics

    • First-run success rate
    • Number of automatic retries
    • Number of complete reruns
    • Number of consecutive failures
    • Recovery success rate after exceptions

    5.4 Resource Metrics

    • Input and output context size
    • Tool-call consumption
    • Page waiting time
    • Proportion of ineffective operations
    • Time spent idle or repeating actions

    A weighted score can be used for comprehensive evaluation:

    1. Overall score =
    2. Outcome quality × 0.40
    3. + Execution efficiency × 0.25
    4. + Stability × 0.20
    5. + Resource efficiency × 0.15

    The weights should be adjusted to the business context. For example, prototype generation may assign more weight to efficiency, while financial, manufacturing, and production-release workflows should place greater emphasis on stability and acceptance quality.

    6. Engineering Conclusions from the Test

    First, a model’s nominal response speed cannot be treated as its task completion speed. A fast single response may still lead to a longer total duration if extensive rework is required.

    Second, parameter size is not the only evaluation criterion. Complex agent tasks place greater emphasis on planning quality, tool-use ability, state awareness, and error recovery.

    Third, automatic retries should be combined with error classification. Retrying without diagnosis may simply delay failure.

    Fourth, agents and models should be evaluated together. The model is responsible for understanding and decision-making, while the framework handles state management, tool orchestration, timeout control, and exception recovery. Both are necessary.

    Fifth, enterprise model selection should not rely only on static question-and-answer tests. Teams should create long-running tasks that resemble real business workflows and execute them repeatedly in a unified environment to observe success rates and performance variance.

    7. A Practical Testing Process

    A complete evaluation process can be divided into five steps:

    1. Define the baseline: Fix the task description, materials, tools, browser environment, and acceptance criteria.
    2. Build process logs: Record each round’s input, model decision, tool call, returned result, and state change.
    3. Run repeated tests: Reduce the influence of random factors from a single execution.
    4. Classify failures: Distinguish model misunderstanding, tool-execution failure, environment issues, and retry-policy problems.
    5. Produce a comprehensive report: Show final quality, total duration, first-run success rate, retry count, and resource consumption together.

    The goal of model evaluation is not simply to rank models. It is to determine whether a model fits a specific workflow. For tasks that require real-time interaction, teams should prioritize solutions with short action chains and stable responses. For tasks that can tolerate longer execution times but require a high probability of completion, a more complex recovery process may be acceptable. For high-risk production workflows, explicit acceptance checks, rollback procedures, and human confirmation are essential.

    Overall, cross-model testing under a fixed agent framework reveals an important fact: real agent efficiency comes from the combined effects of model reasoning, tool execution, context management, and exception recovery. Only by evaluating these factors within a unified testing system can teams move beyond asking whether a model can complete a task and determine whether it is suitable for real production use.