Evaluating Small-Parameter Agent Models Under a Fixed Execution Framework
A controlled comparison of four small-parameter models completing the same multi-state creative collaboration system shows that end-to-end Agent efficiency depends on more than generation speed. Action rounds, tool-call granularity, retry behavior, service stability, context management, and artifact completeness all materially affect first-success time and overall delivery quality. This article presents the experiment design, observed results, reusable metrics, and practical guidance for evaluating coding agents.
1. Why Keep the Execution Framework Fixed and Change Only the Model?
In Agentic Coding evaluations, model capability and execution-framework capability are often mixed together. A model may be changed at the same time as the tool protocol, context organization, browser interaction strategy, or retry logic. When the result changes, it becomes difficult to determine what actually caused the difference.
A more controlled approach is to vary only one factor: keep the task, assets, context, browser environment, acceptance criteria, and repair budget fixed, and replace only the candidate model. Each model then completes the same delivery inside the same execution framework.
This article is based on a small-scale comparative experiment. The experiment used a unified agent runtime and a task called a “creative collaboration system.” The input included only six publicly available art-image assets, along with seed data for project members, assets, versions, and comments. No existing page was provided as a visual reference.
The model had to implement five states:
- Project overview
- Moodboard
- Asset-location comments
- Version approval
- Release preview
The system also had to support three roles: creative director, designer, and client. Required interactions included asset dragging, location-based comments, version switching, approval or rejection, and state persistence after a page refresh.
The desktop viewport was checked at 1440 × 1000, while the mobile viewport was checked at 390 × 844. For each run, the first-version screenshot was saved, only one repair round was allowed, and the final output had to include code, screenshots, logs, review results, and metrics.
This type of task is closer to real-world Agent engineering than generating a static page. It evaluates page structure, state management, permission representation, interaction flows, persistence, responsive behavior, and artifact completeness at the same time.
All four candidate models eventually completed the task, but the differences in their execution processes were more informative than the final screenshots alone.
2. Experiment Design and Abstracted Results
To avoid associating the results with specific vendors, the four lower-parameter-scale candidates are referred to as Models A, B, C, and D. Only the model variable changed. The execution framework, task description, assets, context-window organization strategy, browser configuration, and acceptance rules remained the same.
| Candidate | Style and completion overview | Recorded result |
|---|---|---|
| Model A | A light editorial style with a minimal structure. Project status, assets, and member relationships were clearly represented. | Completed in approximately 17 minutes |
| Model B | A black-and-gold and off-white visual system with serif headings, a consistent permission matrix, and unified approval states. | Failed on the first attempt; approximately 1 hour 12 minutes including the rerun |
| Model C | A dark professional workspace with complete navigation, progress indicators, roles, delivery information, approvals, and an asset canvas. It had the highest information density. | No full-run restart; three internal automatic retries; approximately 88 minutes |
| Model D | An archival-system style with black serif headings, an off-white background, fine-line tables, and thumbnails forming a restrained workspace. | The default retry policy failed twice; the run completed after extended backoff, with approximately 1 hour 4 minutes to the first successful result |
Model A had a clear speed advantage, but that does not mean it is optimal for every task. Model C delivered a high level of engineering completeness and information density, but its internal retries extended the end-to-end duration.
The processes of Models B and D also show that first-attempt failure, default retry behavior, and backoff parameters can substantially affect time to first success. Therefore, labels such as “low cost” or “high speed” describe only part of a model’s capability. Real Agent efficiency also depends on action rounds, tool calls, service stability, and context management.
3. Key Observations: End-to-End Efficiency Is a Systems Engineering Problem
3.1 Action rounds matter more than single-turn generation speed
An Agent task is not completed in a single generation. It consists of a loop:
Observe → plan → call tools → verify → repair
Model A’s shorter execution time may have resulted from fewer ineffective explorations, more direct structural decisions, and a more compact tool-call chain. Although Model C produced a highly information-dense interface, its larger number of page regions, role states, and approval-flow details required more verification rounds.
Evaluations should therefore record metrics such as:
action_roundstool_callsinvalid_callsrepair_rounds
Total elapsed time alone is not sufficient.
3.2 Tool-call granularity determines recovery cost
Coarse-grained tool calls may appear to reduce the number of action rounds, but failures become more expensive to recover from. Fine-grained calls make problems easier to isolate, but they increase the number of rounds and the context burden.
A practical strategy is to use small, verifiable steps. After each write operation or interaction, perform a local check and roll back only the affected state when a failure occurs.
Interactions such as dragging, location-based comments, version switching, and persistence after refresh should have separate verification points. Otherwise, the system may pass through many actions before the final acceptance check reveals that state was lost.
3.3 Service stability and retry policy must be part of the evaluation
Model D showed that the default retry policy failed twice before the run succeeded with extended backoff. Model C continued after three internal automatic retries.
More retries are not always better. A robust retry strategy requires:
- Exponential backoff
- Jitter
- Idempotency controls
- Checkpoint-based resumption
For tool calls with side effects, the system must verify idempotency before retrying. Otherwise, a retry could create duplicate assets, submit an approval more than once, or corrupt version state.
3.4 Context management affects consistency during long tasks
The combination of five states, three roles, and refresh persistence requires the model to maintain consistent state over a relatively long execution chain.
Effective practices include:
- Externalizing task state to readable and writable storage
- Keeping only the current-stage summary and artifact index in the active context
- Referencing screenshots, logs, and reviews in layers
- Reloading the acceptance checklist before each repair round
Without these practices, a model may remember the page structure while losing track of application state. It may also improve the mobile layout in a way that damages the desktop structure.
3.5 Acceptance criteria should be executable and traceable
The experiment required the first-version screenshot to be saved, allowed only one repair round, and required code, screenshots, logs, review results, and metrics to remain available.
This design helps distinguish between a model that succeeds on the first attempt and one that depends on repeated repair. Acceptance criteria should be expressed as checks that can be verified by machines whenever possible. For example:
- Are all five states reachable?
- Can all three roles be switched?
- Is state preserved after a refresh?
- Do the desktop and mobile viewports avoid overflow?
- Does an approval action change the version state?
- Is the artifact directory complete?
4. Reusable Experiment Configuration and Metric Collection
The following product-independent pseudocode illustrates how to fix variables and collect process metrics:
experiment = {"runtime": "unified_agent_runtime","task": "creative_collab_system","fixed": ["assets", "context_policy", "browser_profile", "acceptance", "repair_budget"],"variables": ["model_a", "model_b", "model_c", "model_d"],"repair_budget": 1,"viewports": [(1440, 1000), (390, 844)]}
for model in experiment.variables:run = start_run(model, experiment)while not run.done:step = agent.step(context=run.context, tools=run.tools)run.record(step)if step.failed:run.retry_with_backoff(error=step.error,jitter=True,idempotent_check=True,max_retry=run.policy.max_retry)if run.artifact_ready("first_screenshot"):run.freeze_first_version()breakrun.collect_metrics(wall_clock=True,first_success_time=True,retry_count=True,action_rounds=True,tool_calls=True,repair_rounds=True,artifact_completeness=True)
At a minimum, the metrics should include:
- Time to first success
- End-to-end duration
- Retry count
- Invalid tool-call rate
- Number of repair rounds
- Artifact completeness
- Refresh-persistence pass rate
- Desktop and mobile acceptance results
With a small sample size, absolute rankings should be treated cautiously. The results are better understood as process diagnostics under a shared execution framework. Model selection should be based on repeated evaluations across multiple tasks and multiple runs.
5. Engineering Recommendations
First, fix the execution framework before comparing models
Version the tool protocol, context strategy, browser configuration, retry policy, and acceptance scripts. This prevents framework differences from being mistaken for model differences.
Second, manage retry behavior as an independent variable
Default retries, extended backoff, maximum retry counts, and idempotency checks should be configurable. Each failure and retry reason should be recorded in the logs.
Third, implement persistence early
For systems with multiple roles, multiple versions, and approval workflows, preserving state after a refresh is a hard acceptance requirement. It should not be left for a final patch.
Fourth, preserve complete execution artifacts
Code, screenshots, logs, review results, and metrics are all necessary for investigating why a run was slow or why it failed. Missing artifacts make meaningful diagnosis difficult.
Fifth, avoid permanent conclusions from a single experiment
Agent efficiency is strongly affected by task complexity, service stability, and context length. Evaluation should therefore use a continuously repeated benchmark set with small, incremental test runs.
6. Conclusion
The main conclusion of this experiment is not that one model is universally faster than the others. Under a unified execution framework, all four lower-parameter-scale models were able to complete a complex creative collaboration system. However, their end-to-end efficiency differed because of action rounds, tool-call granularity, service stability, retry backoff, and context management.
Model A’s 17-minute completion shows that lightweight structural tasks can be handled very efficiently. The execution paths of Models B, C, and D demonstrate that first-attempt failures, internal retries, and default retry policies can significantly change time to first success.
For real Agent engineering, model selection should consider success rate, time to first success, repair cost, artifact completeness, and operational stability—not only single-turn speed or a cost category.
