Agent Efficiency Benchmark: Same Framework, Different Models
This article presents a controlled cross-model benchmark that keeps the Agent orchestration framework, task input, resource limits, and acceptance rules fixed while changing only the underlying model. It compares four sub-300B models on a complex frontend generation task, examines wall-clock time, action rounds, tool-call stability, and context management, and provides a reusable checklist for evaluating real-world Agent efficiency.
Why Reverse the Variable
In the previous round of experiments, we tested the opposite direction: we fixed the same model and swapped four different Agent orchestration frameworks. The results showed that framework-level engineering capability has a huge impact on completion speed. The natural follow-up question is this: if the framework is fixed, where do differences in model capability actually show up?
That is the question this round of experiments aims to answer. Many teams look only at parameter count, price, and generation speed when selecting models. But “cheap” and “fast” describe only single-point attributes of a model as an inference engine. When a model is placed into an Agent pipeline that requires multi-round planning, repeated tool calls, and autonomous error correction, real efficiency also depends on a set of system-level factors that are easy to overlook. To see these factors clearly, the framework variable must be locked down.
Experiment Design: One Complex Product Task
What Stayed Fixed
In this round, all of the following were kept identical:
- Orchestration framework: the same in-house Agent task framework, including task decomposition, tool invocation, browser operations, and result acceptance logic;
- Task input: a frontend generation task for a “creative studio collaboration system.” The only inputs were open-licensed image assets and seed data for project members, assets, versions, and comments. There were no existing pages to reference or imitate;
- Functional requirements: the model had to autonomously implement five state pages: project overview, inspiration board, asset-anchored comments, version approval, and publish preview. It also had to support permission distinctions among three roles: creative director, designer, and client. Interactions included asset drag-and-drop, comment anchoring, version switching, and approve/reject actions. State could not be lost after refresh;
- Resource constraints: a unified 262K context window and a 200K task token budget, with browser automation tools used for self-testing;
- Acceptance rules: desktop was checked at a 1440×1000 viewport, and mobile at a 390×844 viewport. Each group first saved first-version screenshots, was allowed only one repair round, and ultimately had to deliver code, screenshots, logs, review records, and metrics data.
What Changed
Only the underlying model was replaced. All four candidate models were in the sub-300B parameter class. Three were recently released cost-effective fast models, and one served as a horizontal reference for the capability ceiling. They are referred to below as Models A, B, C, and D.
Results: Process Differences Beyond the Final Screenshots
All four models eventually completed the task, and their outputs differed in visual style and information architecture. Some took a light editorial style with a lightweight structure that was readable at a glance; some stood out with a dark workbench, achieving the highest information density and engineering completeness; some performed notably well in mobile consistency. But more valuable than the final screenshots were the differences in process data:
- Model A: passed in one attempt and completed the entire task in about 17 minutes, with no explicit retries;
- Model B: the first execution failed, and it succeeded only after reinitiating the entire round of work, for a total time of about 1 hour 12 minutes;
- Model C: did not trigger a whole-round rerun, but carried out 3 internal automatic retries during execution, for a cumulative 88 minutes;
- Model D: failed twice consecutively under the default retry policy and recovered after an extended backoff mechanism, with cumulative time to first success of 1 hour 4 minutes.
All of them “completed the task,” but the stability of the completion path varied by 3–4x.
Four Factors Affecting Real Agent Efficiency
Several conclusions can be drawn from this data that go beyond the models themselves.
1. Generation speed is not task speed. The model with the fastest single-token throughput may take longer end-to-end if it uses more planning rounds and more frequent tool calls. Evaluation must be based on wall-clock time, not generation speed.
2. Action rounds are a hidden cost. For the same task, some models complete it in fewer steps, while others repeatedly probe and revise. The number of rounds affects not only time but also token consumption and failure probability—every round is another chance to fail.
3. Tool invocation and service stability. Agent tasks depend heavily on the availability of external tools and inference services. The whole-round failures, consecutive retries, and backoff recovery observed in the experiment all show that service-side stability can significantly amplify or mask differences in model capability.
4. Context management and self-correction. In long tasks, whether a model can maintain state consistency and recover precisely from local failures instead of starting over is a key differentiator in Agent performance. Model C’s three internal retries did not cause a whole-round rerun, which is exactly evidence that its local correction mechanism was effective.
A Reusable Evaluation Checklist
If you want to reproduce this kind of controlled evaluation in your own team, follow these points:
1. Fix the orchestration framework: task decomposition, toolset, and acceptance logic remain unchanged2. Fix task input: no reference template, consistent seed data, clear interaction and permission requirements3. Fix resource constraints: unified context window, token budget, and browser/tool protocol4. Fix acceptance criteria: desktop and mobile viewports, first-version screenshots archived, only one repair round, complete deliverables5. Record process metrics: wall-clock time, action rounds, tool-call count, retries, and failures6. Repeat experiments multiple times: single runs are affected by randomness, and key conclusions require multiple rounds of validation
Conclusion
The biggest takeaway from this set of experiments is that a low-cost, high-speed model positioning describes only part of a model’s capability. Real Agent efficiency is a systems engineering problem, determined jointly by action rounds, tool-call success rate, service stability, and context management. When selecting models, do not look only at single-point scores on model leaderboards. Instead, put candidate models into your own real task pipeline and measure end-to-end performance through controlled experiments with a fixed framework—that is the evaluation approach closest to production environments.
