Small Models Taking Over Terminal Execution Can Significantly Reduce Main-Model Token Usage

    A lightweight execution model can handle terminal commands, process noisy logs, and return structured summaries, allowing the primary reasoning model to focus on planning, code changes, and decision-making. This role separation can reduce token consumption by nearly 30% in certain tasks while improving context management, long-task stability, and auditability. This article explains the architecture, training strategy, context governance, security controls, and evaluation approach for deploying such a system.

    When agents handle programming, operations, and automation tasks, terminal execution is almost unavoidable. Building projects, installing dependencies, running tests, starting services, and collecting diagnostic data can all generate large volumes of raw output.

    If the primary model responsible for planning and reasoning must read every line of terminal output, its context window can quickly fill with noise. Token usage continues to rise, and long-running tasks are more likely to hit context-length limits. Public technical practices have shown that assigning terminal execution and log summarization to a lightweight model with approximately 4 billion parameters can reduce the primary model’s token consumption by nearly 30% in certain tasks.

    The key is not simply replacing one model with a smaller one. Instead, terminal execution is divided between two roles: a planner and an executor. The task-specific small model absorbs noisy logs, while the primary model consumes only high-density summaries.

    1. Why Terminal Output Becomes a Context Black Hole

    Terminal output is not naturally suited for direct inclusion in a primary model’s context. Build logs, dependency resolution output, test results, and runtime diagnostics often contain repeated lines, progress indicators, paths, warnings, and long stack traces. The information that actually affects the next decision may consist of only a few lines: an error summary, failed test cases, file locations, and the process exit status.

    When the primary model reads raw output line by line, it must handle several responsibilities at once:

    • Execute commands
    • Interpret logs
    • Locate failures
    • Plan corrective actions

    This wastes reasoning capacity and increases cost.

    Context pollution is an even greater problem. A single long test run can generate tens of thousands of tokens. As multiple repair attempts append additional logs, the context grows in a snowball effect. Before the model resolves the core issue, historical logs may consume most of the available context, while the code logic, task objective, and next-step plan become diluted.

    Simple truncation is not reliable either. The critical error may appear at the end of the output, or it may require surrounding context to interpret correctly. A mechanism is therefore needed between the terminal and the primary model to compress and structurally translate the information.

    2. Core Design: The Primary Model Plans, the Small Model Executes and Summarizes

    The responsibilities of the execution sub-agent should be clearly defined.

    The primary model should:

    • Understand the user’s objective
    • Break the task into steps
    • Decide which command to run next
    • Modify code
    • Evaluate results

    The sub-agent should:

    • Execute commands in a controlled environment
    • Read the raw output
    • Extract important events
    • Return a structured summary

    The sub-agent does not need complete general-purpose reasoning capabilities. It mainly needs to perform information extraction, error classification, status detection, and summarization consistently. A lightweight model is therefore sufficient for this role.

    The response returned to the primary model should not be an ambiguous block of natural language. It should be a machine-readable result containing fields such as execution status, key errors, failed tests, affected files or modules, areas requiring attention, references to the raw log, truncation status, and confidence.

    The primary model reads the summary and only the necessary excerpts. Raw logs can be stored on disk or in a logging service and retrieved through references when needed. This approach reduces context usage while preserving auditability and troubleshooting capabilities.

    1. # Pseudocode: return a structured summary from the terminal sub-agent
    2. def run_terminal_subagent(goal, command, policy):
    3. raw = execute_sandboxed(command, timeout=policy.timeout)
    4. compact = subagent.summarize(
    5. goal=goal,
    6. command=command,
    7. raw_output=raw,
    8. schema=EXEC_SUMMARY_SCHEMA,
    9. max_tokens=policy.max_summary_tokens,
    10. )
    11. return {
    12. "status": raw.status,
    13. "summary": compact.summary,
    14. "key_events": compact.key_events,
    15. "errors": compact.errors,
    16. "raw_log_ref": persist_log(raw),
    17. "truncated": compact.truncated,
    18. "confidence": compact.confidence,
    19. }

    3. Two-Stage SFT and RL Training: Turning a General Small Model into a Dedicated Executor

    Stage One: Establishing Basic Behavior with SFT

    The first stage uses supervised fine-tuning (SFT) to establish the model’s basic behavior. Training examples should be collected from real terminal traces and include commands, raw output, expected summaries, key errors, and structured fields.

    The dataset should cover scenarios such as:

    • Successful command output
    • Failure stack traces
    • Dependency conflicts
    • Test timeouts
    • Permission errors
    • Long logs
    • Repetitive noise

    The goal is not to reproduce the logs. The model must determine which information affects the next decision, including the error type, the first root cause, the scope of the failure, and whether the operation can be retried.

    The output format should be constrained by a schema to reduce the parsing burden on the primary model.

    Stage Two: Optimizing Tradeoffs with RL

    The second stage uses reinforcement learning to optimize the tradeoffs involved in summarization. The reward function should not reward short summaries alone, because that could cause the model to omit critical errors. It should also avoid rewarding completeness alone, which could cause the model to produce little more than a compressed copy of the original log.

    A more suitable reward function can combine:

    • Task success rate
    • Recall of critical facts
    • Format compliance
    • Token savings
    • Latency
    • Penalties for hallucinations

    Curriculum learning can gradually increase difficulty, moving from short logs to long logs and from single-command tasks to multi-step workflows.

    Reward hacking must also be addressed. If the system scores only summary length, the model may delete important stack traces. If it scores only task success, the model may rely too heavily on fallback processing by the primary model. Penalties for missed information and manual sampling should therefore be included.

    4. Context Governance and Engineering Implementation

    A layered summarization strategy is recommended:

    • Command-level summaries: Record the exit status and key errors.
    • Stage-level summaries: Aggregate build, test, and deployment results.
    • Task-level summaries: Preserve the final conclusion and unresolved issues.

    Sliding windows, error prioritization, and structured filters can remove noise from raw logs. Original logs should be stored in a logging service or object storage. The primary model can request relevant excerpts by reference instead of receiving the entire log at once.

    Security and observability are equally important. Terminal execution should use least-privilege permissions, sandbox isolation, sensitive-data redaction, command approval, and interception of high-risk operations. These controls help prevent command injection and unauthorized access.

    Monitoring should include:

    • Task success rate
    • Summary fidelity
    • Critical-error omission rate
    • Percentage of tokens saved by the primary model
    • P95 latency
    • Fallback rate
    • Raw-log retrieval rate

    In cloud environments, general-purpose logging services, object storage, container platforms, monitoring, and alerting can be combined to create an auditable operational loop.

    5. Evaluation, Gradual Rollout, and Practical Boundaries

    Evaluation should not focus on token reduction alone. It must also measure whether tasks become easier and more reliable to complete.

    A staged rollout is recommended:

    1. Run the sub-agent in parallel and compare its summaries with the results of having the primary model read logs directly.
    2. Gradually delegate high-frequency, long-log, and low-risk terminal tasks.
    3. Use confidence-based adaptive routing.

    When the sub-agent has low confidence, encounters a security-sensitive operation, or faces complex root-cause analysis, processing should fall back to the primary model.

    Small-model delegation is especially valuable for frequent build and test workflows that generate long logs. For diagnostics requiring deep reasoning across multiple files, the primary model should remain in control.

    By separating responsibilities, using a structured protocol, applying two-stage training, and retaining traceable raw logs, teams can significantly reduce primary-model token consumption without materially reducing task success rates. The result is more stable behavior for long-running agent tasks and a more manageable path to production deployment.