Selecting Small-Parameter Execution Models for AI Agents: Instruction Following and Throughput Efficiency

    In agent-based development pipelines, models are often split into a "capability layer" that decomposes problems and writes plans, and an "execution layer" that carries out those plans in bulk—usually with smaller models because of high call volume and repetitive tasks. This article explains why the gaps between small models shrink once the capability layer produces a clear plan, and where real differentiation actually comes from: instruction-following discipline, execution throughput and latency, and bundled capabilities such as multimodal input and image generation. It walks through four practical constraints, provides a reusable task-brief template, shows how to build a reproducible 20–30 task evaluation set with six objective metrics, explains how to account for total cost rather than just per-token price, recommends complexity-based routing between model tiers, and lists common anti-patterns with concrete engineering guardrails.

    One: Why “Picking a Small Model” Becomes an Engineering Problem

    In an agent-based development pipeline, the model usually doesn’t work alone. Instead, the workload is split into two layers: one that breaks down the problem, drafts the plan, and sets the boundaries—call it the capability layer; and another that executes the established plan in bulk—call it the execution layer. The execution layer tends to use smaller-parameter models for a straightforward reason: high call volume, homogeneous tasks, and low demands on creativity.

    Yet in practice, many teams deliberate carefully over the capability-layer model and then pick the execution-layer model almost at random. The result is that the quality of the entire pipeline gets bottlenecked by its weakest link. To judge whether a small-parameter model is good enough for your pipeline, you need to look at four independent dimensions rather than asking vaguely “which vendor is stronger.”

    Here’s the conclusion up front: once the capability-layer model has produced a clear enough implementation plan, the differences between small-parameter models shrink dramatically. The real differentiation comes from instruction-following discipline, execution efficiency (throughput and latency), and whether the model bundles “out-of-the-box” capabilities such as multimodal input and image generation.

    Two: The Four Real Constraints

    2.1 Instruction Following: Divergent Thinking Is Actually a Risk

    This point runs counter to intuition. We tend to treat “has ideas and proactively adds design” as a strength, but at the execution layer, that is exactly where defects come from.

    The execution-layer model’s job is to “land an already-decided plan accurately.” Every act of free improvisation it adds introduces three kinds of problems:

    • Out-of-scope changes: A directory you didn’t ask it to refactor gets refactored anyway; the diff grows from 30 lines to 800, and review cost goes up instead of down.
    • Silent deviation from conventions: You agreed on a particular state-management approach; it “thinks” another one is cleaner, and now the same repository has two competing styles.
    • Self-expanded acceptance scope: To “helpfully complete the tests,” it modifies modules that weren’t assigned to it, colliding with other tasks running in parallel.

    So when evaluating an execution-layer model, the thing to test is not “how smart it is” but constraint compliance: When you explicitly say “do not do X,” does it actually refrain? When you give it a required interface signature, does it change the signature on its own? When it hits a gap the conventions don’t cover, does it stop and ask, or does it decide unilaterally? That last behavior does the most damage in parallel multi-agent scenarios.

    A practical test: give it a task brief with a “Prohibited Actions” section, and deliberately place something it will likely want to optimize among the prohibitions (for example, “do not modify any generated files under build/“), then see whether it complies.

    2.2 The Agent Harness Lock-In Effect

    Comparing model capabilities side by side—detached from the specific Agent Harness (the surrounding prompt templates, tool definitions, context-injection method, retry and approval policies)—often produces conclusions that don’t hold. The same model can perform a full tier differently under two different Harnesses, because the Harness determines:

    • Whether tool descriptions are clear and error messages are readable;
    • Whether context is “stuffed in wholesale” or “retrieved on demand,” which directly affects whether the model can locate the key constraint;
    • Whether failures trigger automatic retries, graceful degradation, or handoff to a human;
    • Whether output is strictly parsed (structured protocol) or is free text that must be re-parsed afterward.

    So when discussing “generality,” the more meaningful question is: Without deep Harness customization—using only the default configuration of general-purpose tooling—which model covers the widest range of task types? This determines its “out-of-the-box usability.” Once you write a Harness specifically for a model, the side-by-side comparison is already invalid, because the cost now includes that customization work.

    2.3 Execution Efficiency: Throughput Matters More Than Time to First Token

    Batch tasks are throughput-sensitive workloads, not interaction-sensitive ones. The yardstick is output tokens per second, plus the reasoning length during inference layered on top of it.

    A typical counterexample: you just want to adjust the spacing and alignment of one UI element, so you dispatch the task—and the execution-layer model starts a long, deep chain of reasoning. It spawns a few subtasks to “research design trends,” goes back and forth asking and answering itself, and emits tokens slowly. A five-minute job balloons into half an hour, with zero marginal benefit.

    Even more painful is test-and-verification work. These tasks are characterized by many rounds with small increments per round. Once the model slips into “long reasoning every round,” total time scales up linearly:

    1. Total time ≈ Σ(output tokens per round / output throughput) + Σ(tool-call round-trip time)
    2. 40 rounds, 1500 tokens/round, 25 token/s → ~40 × 60s = 40 minutes
    3. 40 rounds, 1500 tokens/round, 100 token/s → ~40 × 15s = 10 minutes

    Notice that in the second case the model’s capability hasn’t changed at all—throughput alone produces a fourfold difference. This is why “the quota is ample, so a stronger model doesn’t matter” is a misconception: nobody is going to use a heavyweight reasoning model to fix the alignment of a single line of CSS.

    2.4 Multimodal and Image Assets: The Value of Bundled Capabilities

    Some inference platforms bundle image generation or image understanding into the model service. The engineering value of this kind of “bundling” is often underestimated.

    Take a concrete scenario. Suppose you’re building a 2D mini-game and need a batch of assets—spaceships, planets, UI icons. If the execution-layer model can only output code, its only route for the art requirements is to brute-force it with geometric primitives—piecing together a shape that “looks like a spaceship” using circles, polygons, and gradients. This burns a lot of tokens, produces a lot of hard-to-maintain code, and has a low quality ceiling.

    A model with image-generation capability, by contrast, can produce bitmap assets directly, converting “draw the art with code” into “import static resources.” There’s also an easy-to-overlook detail here: assets need a transparent background to drop straight into an engine. An image without an alpha channel has to go through an extra cutout/matting step, or it renders with a solid block behind it. Bundling this capability on the model side saves that entire step.

    From a cost perspective, the difference between these paths is clear:

    Approach Extra spend Extra ops
    Model with built-in image generation None (included in the inference service) None
    Third-party drawing credits Separate billing, needs budget approval and quota monitoring Accounts, quotas, failure retries
    Self-hosted drawing model GPU resources, VRAM, image maintenance Service availability, scaling, version upgrades

    The latter two approaches cost more than money: they also require maintaining the “CLI tools + Skill/plugin” integration layer, along with its permissions and failure handling. For a small team that just wants to get a prototype running fast, this hidden workload often exceeds the cost of the assets themselves.

    Three: The Core of Mixing Large and Small Models Is the Quality of the “Construction Plan”

    Back to the most essential point: the ceiling of the execution layer is determined by the quality of the implementation-guidance document the capability layer writes.

    If the architecture, tech stack, directory conventions, test boundaries, and acceptance criteria are all clearly specified, the gap between small-parameter models narrows significantly—because at that point they’re doing “construction from blueprints,” and what’s being tested is execution, not comprehension. Conversely, if the plan document is vague, then no small model will save you, because the uncertainty gets pushed down to exactly the stage that is least equipped to make those decisions.

    Here is a task-brief structure you can apply directly:

    1. # Task: Add unit tests for the refund scenarios in the order module
    2. ## Background and goal
    3. - Goal: cover the 5 branches of the refund flow, raise line coverage from 62% to >= 85%
    4. - Out of scope: do not refactor the class under test, do not change production code logic
    5. ## Tech stack and constraints (must follow)
    6. - Test framework: <framework name>; assertion style follows the existing test files
    7. - No third-party dependencies beyond the mocking framework
    8. - All new test files go under `tests/order/refund/`
    9. ## Boundaries and inputs
    10. - Branches involved: full refund / partial refund / reject over-refund / reject duplicate refund / order-state rollback after refund
    11. - Data source: use `tests/fixtures/refunds.json`; no new external network calls
    12. ## Acceptance criteria (auto-verifiable)
    13. 1. `make test` passes entirely
    14. 2. Line coverage of new test files >= 85%
    15. 3. No new lint warnings
    16. 4. No `skip` / `xfail` markers
    17. ## Prohibited actions
    18. - Do not modify any file under `src/order/`
    19. - Do not alter generated artifacts under `build/`
    20. - Do not weaken assertion strength just to make tests pass
    21. ## Failure handling
    22. If you judge the goal cannot be met within the above constraints, stop and explain why. Do not expand scope on your own.

    The value of this document is that it keeps “judgment” in the capability layer and leaves “execution” to the execution layer. That final “failure handling” item is especially important: it gives the small model a legitimate exit path and prevents it from forcibly breaking through the constraints.

    Beyond the task brief, three more things decide success or failure:

    1. Keep the context you feed in lean. Don’t stuff the entire repository in. Giving “the relevant files + the relevant interface definitions + the task brief” works better than giving all the code, and it saves tokens too.
    2. There must be an automated verification loop. Lint, type checks, unit tests, build—anything that can run automatically should not rely on the model’s self-report that it’s “done.” The execution-layer model’s self-assessment is unreliable; the verifier is reliable.
    3. Control the degree of parallelism. When multiple execution agents modify the same repository at once, conflicts and overwrites are the norm. Splitting tasks by directory or file boundary is safer than splitting by feature.

    As for how to pair the capability layer, there’s no single answer—it depends on what you have on hand. Common pairing approaches include: a capability-class large model plus a lightweight version from the same series; a capability-class large model plus a mid-size open-weight model; a vendor’s Pro tier paired with its Flash tier. Which one works better can only be determined by running it on your repository, your conventions, and your Harness.

    Four: Turning Selection from “Gut Feeling” into a Reproducible Evaluation

    Subjective experience will fool you, especially when your test cases number only three to five. It’s worth running a small but rigorous baseline test.

    Step 1: Build a task set. Pick 20–30 real tasks from historical tickets, covering four categories: mechanical refactoring, unit-test completion, bug localization, and small cross-file changes. Every task needs an objective pass/fail criterion (whether the tests pass, whether the diff lands only in the expected files).

    Step 2: Fix the variables. Same task brief, same Harness, same set of tool permissions, same temperature setting. Change only the model. Otherwise, you’re measuring differences in the Harness.

    Step 3: Record six metrics.

    Metric Meaning Why it matters
    First-pass rate Share that passes validation on the first delivery Rework is the biggest hidden cost
    Average rework rounds Number of human interventions Translates directly to labor
    Per-task token consumption Input + output Cost baseline
    End-to-end latency From dispatch to validation passing Combined reflection of throughput and latency
    Scope-violation rate Number of times forbidden files were changed Hard metric for instruction following
    Human rewrite rate Share of final code substantially rewritten by humans Quality floor

    Record results in a structured form like this:

    1. {
    2. "model_alias": "exec-model-A",
    3. "harness": "v3-default",
    4. "task_set": "internal-24-tasks",
    5. "first_pass_rate": 0.62,
    6. "avg_rework_rounds": 1.4,
    7. "avg_output_tokens": 18400,
    8. "avg_wall_clock_sec": 412,
    9. "scope_violation_rate": 0.08,
    10. "human_rewrite_rate": 0.15
    11. }

    A frequently overlooked conclusion: A model with a high first-pass rate but also a high scope-violation rate may actually be less usable than one with a slightly lower pass rate but strict rule-following—because the former shifts the review cost onto you.

    Five: Calculate Total Cost, Not Just Unit Price

    “This model is cheap” is often just a number on a bill. The complete cost includes at least four components:

    1. Direct call cost: input/output token price × consumption. Note that for execution-layer tasks, input is usually far larger than output (long context, short output), so the input price carries more weight.
    2. Rework and labor cost: rework rounds × labor cost per round. This item is usually an order of magnitude higher than the call cost, and it’s exactly what small-model selection most easily leaves out of the calculation.
    3. Infrastructure cost: if you self-host a model for image generation or some specific capability, you need GPU instances, image maintenance, autoscaling, and availability guarantees. In the cloud, this typically lands on container platforms and GPU compute instances, paired with object storage for models and assets, and logging plus monitoring/alerting services for observability; alternatively, you can buy a managed inference service to sidestep the ops.
    4. Integration and ops cost: the surrounding CLI tools, the plugin/Skill system, permissions and credential management, and failure-retry policies. The complexity of this chain grows non-linearly with the number of tools.

    The way to write the total-cost calculation is simple:

    1. Real cost per task = call cost + rework rounds × labor cost per round + infrastructure allocation + integration/ops allocation

    An illustrative comparison (the numbers only demonstrate the method, not real pricing): Approach A has a 60% higher per-token price, but a 30% higher first-pass rate and half the rework; Approach B has a lower unit price but needs 3× the human intervention. When volume scales to an average of 200 tasks per day, B’s total cost will very likely overtake A’s. A low unit price is only the entry ticket; the rework rate decides the outcome.

    Six: Scenario-Based Routing: Don’t Fight Everything with One Model

    The mature approach is complexity-based tiered routing, not fixing one model across the entire pipeline:

    Task type Recommended tier Rationale
    UI spacing / copy / style tweaks Execution layer (lightweight) Highly deterministic, throughput first
    Unit-test completion, log-format alignment Execution layer (lightweight) Pattern-based, validators catch issues
    Mechanical refactoring, bulk renaming Execution layer (lightweight) High volume, cost-sensitive
    Build/CI error fixing Execution layer with an escalation path Fast if simple, escalate if complex
    Architecture design, tech selection Capability layer (heavy) Requires trade-offs and judgment
    Cross-module refactoring, complex bug root cause Capability layer (heavy) Long logic chains, high cost of trial and error
    Creative work (copy, narrative, ideation) Capability layer (heavy) See the next section

    This needs to be paired with escalation and downgrade mechanisms: if the execution layer fails validation twice in a row, it should automatically escalate to the capability-layer model rather than letting it burn three rounds of tokens in the same hole.

    Seven: A Key Insight: Thinking Cannot Fill Knowledge Gaps

    Finally, creative tasks. There’s a hard constraint here: a reasoning chain can only solve logical-inference problems; the amount of internalized knowledge tied to parameter count is a separate, independent dimension.

    Reasoning can help you combine known premises into new conclusions, but it can’t conjure domain knowledge the model never encountered during training. When a model is completely unfamiliar with the conventions of a niche field, the idioms of an obscure toolchain, or the traditions of a particular writing style, telling it to “think harder” produces no improvement—it will just circle within the boundary of its own knowledge, producing output that reads fluently but is substantively spinning its wheels.

    So for creative tasks and tasks with strong domain dependence, prefer models with a larger parameter scale—this is the least-effort strategy. Correspondingly, you should explicitly feed external material in via retrieval augmentation, rather than expecting the model to fill the gaps on its own.

    Eight: Common Anti-Patterns and Engineering Guardrails

    • Anti-pattern: letting the execution layer improvise freely. Guardrail: the task brief must include both “Prohibited Actions” and “Failure Handling”; block out-of-scope changes at the CI stage with a path allowlist.
    • Anti-pattern: unbounded deep reasoning dragging down batch tasks. Guardrail: explicitly turn off long-chain reasoning for lightweight tasks; set a per-task timeout and a token ceiling.
    • Anti-pattern: judging completion by the model’s self-report. Guardrail: make every acceptance criterion automatically executable, reject anything that fails, and accept no natural-language rebuttal.
    • Anti-pattern: feeding the whole repository’s context to the execution layer. Guardrail: trim context per task, prioritizing interface definitions and adjacent files.
    • Anti-pattern: parallel agents with no isolation. Guardrail: split tasks by file boundary, give each its own branch, and run validation uniformly before merging.
    • Security guardrails (must be enforced): Execution-layer models get write access to the repository only by default and must not hold production credentials, directly modify production config, or touch live data; operations involving deletion, migration, or permission changes must go through human approval; before connecting internal code and data, confirm that the data-use and retention policies of the inference service in use meet your organization’s compliance requirements; and avoid printing secrets in logs and prompts.

    Conclusion

    Selecting a small-parameter execution model is, at its core, an engineering problem of balancing three things—constraint compliance, throughput efficiency, and the scope of bundled capabilities—not a leaderboard problem. Three things are worth doing right now:

    1. Standardize the implementation-plan document; this is the highest-return single improvement.
    2. Build a task set of your own 20–30 cases, and replace subjective impressions with six objective metrics.
    3. Set up complexity-based routing and an escalation path, so expensive models appear only where they’re genuinely needed.

    As for the specific pairing, the answer varies from person to person—it depends on your repository’s size, the maturity of your conventions, the completeness of your Harness, and your budget structure. Running the numbers on your own task set is more reliable than any side-by-side comparison conclusion.