Benchmarking Agent Efficiency for Lightweight Models Under a Unified Execution Framework

    This article examines agent efficiency beyond single-inference latency by fixing the execution scaffold and replacing only the underlying model. It describes a controlled evaluation design, compares four lightweight candidates across action rounds, tool calls, service stability, and context management, and derives engineering practices for retry classification, externalized state, checkpoint recovery, and scripted acceptance.

    A common misconception in agent deployment is to equate model inference speed directly with task execution efficiency. Real long-running agents often include task planning, tool calls, browser interaction, file operations, state maintenance, and result acceptance. Single-inference latency is only one part of end-to-end elapsed time. To observe model behavior in complex tasks more objectively, this article uses a controlled-variable method: keep the execution framework fixed and replace only the underlying model. It compares a group of lightweight models with similar parameter scales. This article does not recommend or review any specific product. Instead, it abstracts the problem as a general engineering question: when the execution scaffold, task materials, context budget, browser environment, and acceptance rules remain consistent, what differences appear in action rounds, tool calls, service stability, and context management across models, and how do these differences affect engineering practice?

    1. Experimental Design: Why a Fixed Execution Framework Matters

    The most common source of loss of control in agent efficiency benchmarking is a difference in the execution framework itself. Different frameworks vary in prompt organization, tool permissions, failure retries, context compression, browser control, and acceptance methods. If both the framework and the model are replaced at the same time, it is difficult to attribute the final elapsed time. Therefore, this observation follows these controlled-variable principles:

    1. Unified execution framework: All candidate models run in the same agent execution scaffold, with consistent tool registration, permission boundaries, browser isolation, and log collection.
    2. Same task and material source: Use the same set of complex tasks, with identical input materials, initial context, and available tool sets.
    3. Consistent context budget: Set the same context window budget, summarization trigger threshold, and state externalization strategy.
    4. Isolated browser environment: Each experiment uses an independent session to avoid interference from cache, login state, and residual state.
    5. Scripted acceptance rules: Result acceptance does not rely on subjective judgment. Fixed rules check artifact completeness, key fields, page state, and reproducibility.
    6. Replace only the underlying model: Except for the model, other variables are fixed as much as possible. Record time to first success, total elapsed time, retry count, tool-call success rate, invalid action rounds, peak context usage, and acceptance pass rate.

    The value of this design is that it places model capability back into system engineering for observation. Even if a model generates quickly in a single turn, end-to-end efficiency may still decline if it uses too many action rounds, has a high tool-call failure rate, or repeatedly expands context. Conversely, a model with tighter planning and more stable error recovery may outperform on total elapsed time.

    2. Results Overview: All Four Candidates Completed the Task, but the Processes Differed Significantly

    In a set of abstract comparison experiments, all four candidate models eventually completed the task, but the process differences were more valuable than the final screenshot. The four experiments can be labeled Candidate A, B, C, and D:

    Candidate Process observation Elapsed time Key observation
    A Completed in one execution About 17 minutes Fewer action rounds and a shorter tool-call chain
    B First attempt failed; completed after a full rerun About 72 minutes total Failure localization and full rerun created a large amplification cost
    C No full rerun; 3 internal automatic retries About 88 minutes total Internal retries maintained execution continuity but amplified elapsed time
    D Default retry strategy failed twice consecutively; completed after extended backoff About 64 minutes cumulative time to first success Backoff strategy and error classification directly affect success time

    The results show that a “low-cost” or “high-speed” model positioning describes only part of the capability. Real agent efficiency also depends on at least four factors: action rounds, tool-call quality, service stability, and context management. Candidate A’s advantage was not only single-turn speed but also tighter planning, which reduced invalid tool calls. Candidate B’s problem was not necessarily that the model could not perform the task; rather, after the first failure it lacked fine-grained recovery and could only rerun the entire task. Candidate C’s internal retries avoided a full restart, but three automatic retries lengthened execution time. Candidate D shows that if the default retry strategy does not distinguish error types, it may fail repeatedly during service fluctuations and recover only after extended backoff.

    3. Breaking Down the Sources of Difference

    3.1 Action Rounds and Planning Quality

    In long agent tasks, each action round may trigger tool calls, page reads, and context appends. The more action rounds there are, the faster context expands, and the higher the probability of repeated reads and invalid attempts. Candidate A completed within 17 minutes, indicating high planning compactness and tool-selection efficiency. From an engineering perspective, it is not enough to look only at whether the task was eventually completed. It is also necessary to examine the effective action ratio, that is, the ratio of effective tool-call rounds to total action rounds. If the ratio is too low, the model is spending too much effort on exploration, repeated confirmation, or error recovery.

    3.2 Tool Calls and Error Classification

    Tool-call failures can be classified at least into transient errors, semantic errors, permission errors, and state conflicts. Transient errors are suitable for retries. Semantic errors should return to the planning layer for correction. Permission errors should terminate directly and provide a prompt. State conflicts require checkpoint recovery. If a fixed number of retries is applied uniformly, non-retryable errors may be repeatedly amplified. Candidate C completed after 3 internal automatic retries, showing that an internal retry strategy can improve continuity but also hides failure cost inside a single execution. Candidate D failed twice consecutively under the default retry strategy and then completed through extended backoff, showing that backoff strategy is critical in service fluctuation scenarios.

    3.3 Context Management and Externalized State

    In long tasks, a larger context is not always better. Large page contents, logs, and intermediate artifacts returned by tools can quickly occupy the window, crowding out key state. A more reliable approach is to externalize task state, confirmed facts, and to-do lists into structured storage; summarize long logs in segments; write checkpoints after every several action rounds; and load only necessary state during recovery. In this way, even if a retry occurs, execution does not need to start from zero.

    3.4 Service Stability and Execution Isolation

    Browser sessions, network requests, external API rate limits, and concurrent resources all affect agent execution. When the execution framework is fixed, each experiment should use an independent session and timeout control, and record failure reasons and retry waiting time. Otherwise, a single network fluctuation may be misjudged as a model capability problem.

    3.5 Acceptance Rules and “False Completion”

    Unified acceptance rules are the foundation of benchmarking credibility. An agent may generate results that look complete but lack key fields. Scripted acceptance should check whether artifacts exist, whether fields are complete, whether page state is correct, and whether key steps are reproducible. Only after acceptance passes can the result be counted toward time to first success.

    4. Engineering Practice Recommendations

    First, establish an end-to-end metric system. At a minimum, collect time to first success, total elapsed time, retry count, retry reason distribution, tool-call success rate, invalid action rounds, peak context usage, acceptance pass rate, and human intervention count. Second, classify retry strategies. Use exponential backoff with jitter for transient errors, and set a maximum retry count and circuit-breaker threshold. Return semantic errors to the planning layer. Terminate directly on permission errors. Third, design state externalization and checkpoints upfront to avoid full reruns after failure. Fourth, structure execution logs by recording each action round, tool call, error type, and recovery action for easier attribution.

    Below is a neutral pseudocode example illustrating the basic idea of classified retries and checkpoint recovery:

    1. def classify_error(err):
    2. if err.is_transient():
    3. return "transient"
    4. if err.is_semantic():
    5. return "semantic"
    6. if err.is_permission():
    7. return "permission"
    8. return "unknown"
    9. def run_agent_with_retry(task, model, max_attempts=4):
    10. for attempt in range(max_attempts):
    11. checkpoint = load_or_create_checkpoint(task.id)
    12. try:
    13. return execute_task(task, model, checkpoint)
    14. except ToolError as err:
    15. kind = classify_error(err)
    16. if kind == "permission":
    17. raise
    18. if kind == "semantic":
    19. replan(task, err)
    20. continue
    21. if kind == "transient":
    22. if attempt == max_attempts - 1:
    23. raise
    24. wait = min(300, (2 ** attempt) * 30) + random_jitter()
    25. sleep(wait)
    26. continue
    27. raise

    Context management can also be expressed as a configuration policy:

    1. context_policy:
    2. max_tokens: fixed_budget
    3. summarize_threshold: 0.75
    4. external_state: enabled
    5. checkpoint_interval: every N action rounds
    6. keep_recent_steps: most recent K rounds
    7. long_log_strategy: segmented summarization

    These examples are not interfaces of any specific product. They are general engineering patterns: make error classification, backoff, checkpoints, and context compression unified capabilities of the execution framework, and then compare model differences on that basis so that conclusions are more stable.

    5. Conclusion

    The benchmarking method of fixing the execution framework and replacing only the model can bring agent efficiency back from “single-turn speed” to “end-to-end system engineering.” All four candidate models completed the task, but the differences in time to first success—17 minutes, 72 minutes, 88 minutes, and 64 minutes—show that action rounds, tool-call quality, service stability, and context management significantly affect real efficiency. For engineering teams, a more reasonable approach is to first build a unified evaluation scaffold and structured metric system, and then improve overall robustness through classified error retries, externalized state, checkpoint recovery, and scripted acceptance. A “low-cost” or “high-speed” model positioning describes only part of the capability. Truly usable agent efficiency comes from joint optimization of the model and the execution framework. With general cloud capabilities such as logging services, monitoring and alerting, and object storage, teams can also persist each action round, tool-call result, and retry reason as traceable data to support later iteration.