画板 1

    Multi-Model AI Architecture: Let Larger Models Plan and Smaller Models Execute

    4 hr ago0 views

    Learn how to divide AI workloads between frontier models and more cost-effective models. This guide compares the Advisor and Orchestrator patterns, explains how token distribution and prompt caching affect costs, and outlines when task decomposition is—and is not—worthwhile.

    Multi-Model AI Architecture: Let Larger Models Plan and Smaller Models Execute

    1. Why Use a Tiered Model Strategy?

    Many teams rely on high-capability models for a wide range of tasks, including coding, planning, research, and document review. However, flagship models typically have higher API costs, and many execution steps do not require their full reasoning capabilities. The most demanding reasoning is often concentrated in a few stages: breaking down a task, evaluating options, and making key decisions.

    A common approach is to separate planning from execution and assign each to a model suited to the work. Use a capable flagship model for planning and judgment, and a more cost-effective mid-tier model for execution, retrieval, and bulk processing. This can preserve much of the task quality while reducing costs and improving speed.

    Published benchmark results support this approach. In one software engineering evaluation, a mid-tier model working with a flagship model as an advisor retained about 92% of the performance of an all-flagship setup while using 63% of its cost. In a web research evaluation, a flagship model orchestrating mid-tier workers achieved 96% of the all-flagship result at 46% of the cost. These figures illustrate a broader principle: model capability is a limited resource, so reserve it for the parts of a task that need it most.

    2. Two Common Orchestration Patterns

    2.1 Advisor: A Worker with an Expert to Consult

    In this pattern, a mid-tier model handles the task independently and consults a flagship model when it reaches an uncertain or high-stakes decision.

    1. User task
    2.    │
    3.    ▼
    4. Worker (mid-tier model) ─── Handles most steps independently
    5.    │
    6.    ├─ Reaches an ambiguous or high-risk decision
    7.    ▼
    8. Advisor (flagship model) ─── Provides a recommendation
    9.    │
    10.    ▼
    11. Worker resumes and completes the task

    The flagship model is typically called only once per task, while most tokens are processed at the mid-tier rate. This pattern works well when the execution path is relatively clear and expert input is only needed at a few critical points—for example, code changes or bulk data processing.

    2.2 Orchestrator: A Coordinator with Multiple Workers

    In this pattern, the flagship model takes the lead. It interprets the request, breaks it into subtasks, assigns them to multiple mid-tier models running in parallel, and combines their results. The flagship model coordinates the work rather than carrying out the detailed execution itself.

    1. User task
    2.    │
    3.    ▼
    4. Orchestrator (flagship model) ─── Breaks down and assigns tasks
    5.    │        │        │
    6.    ▼        ▼        ▼
    7.  Worker   Worker   Worker   (mid-tier models working in parallel)
    8.    │        │        │
    9.    └────────┴────────┘
    10.             ▼
    11.      Orchestrator combines results

    This pattern suits tasks that can be divided into independent pieces, such as researching and cross-checking multiple web sources, reviewing a long document chapter by chapter, or classifying large volumes of logs. Since the subtasks can run concurrently, total elapsed time is often determined by the slowest worker rather than by the sum of all execution times.

    3. Why This Can Reduce Costs: Token Distribution and Caching

    3.1 Route Most Tokens to Lower-Cost Models

    The cost advantage comes from how tokens are distributed. Engineering reports on these architectures indicate that roughly 84% to 98% of input tokens can be handled by workers at mid-tier rates, with only a small portion processed by the orchestrator or advisor. Because mid-tier models generally cost a fraction of flagship models, total costs can fall to around 40% of an all-flagship setup. Parallel execution can also reduce wall-clock time, with reported speed improvements of about three times.

    3.2 Make Use of Prompt Caching

    Prompt caching is another optimization worth considering. In an orchestration setup, sub-agents often reuse the same system prompts, task instructions, and tool definitions. If the underlying API supports cached prompt billing, keeping a separate context cache for each sub-agent can reduce charges for repeated prefixes and further lower costs.

    A few practical guidelines:

    • Put stable content, such as system prompts and tool schemas, at the beginning of the prompt to make it easier to match cached prefixes.
    • Put changing, task-specific content at the end.
    • Reuse a consistent prompt structure for repeated calls to the same sub-agent rather than rebuilding it differently each time.

    4. When Task Decomposition Is Not the Right Choice

    This architecture is not suitable for every task. Consider the following constraints before adopting it:

    1. Delegation depth may be limited. Coordinators may only be able to delegate one level down, and deeper nesting may be ignored. Do not assume a multi-level hierarchy such as “orchestrator → sub-orchestrator → worker” will work.
    2. The number of concurrent agents may be capped. A multi-agent session may support only around twenty independent agents. Larger workloads may need to be divided into batches outside the agent session.
    3. Narrow, reasoning-intensive tasks may not benefit from decomposition. For a single complex mathematical proof or a detailed review of a critical legal contract, splitting the work can add coordination overhead and increase the risk of losing context. A single flagship model may be more efficient.
    4. Worker outputs need to be reconciled. Workers may differ in wording, format, or evaluation criteria. The orchestrator should normalize their results and, where necessary, run an additional validation step.

    5. Practical Guidance

    Use this decision framework to choose an approach:

    • Check whether the task can be parallelized. If it can be split into independent subtasks, use the Orchestrator pattern. If the execution path is clear and only a few decisions need expert input, use the Advisor pattern. If the task is neither decomposable nor dependent on occasional expert judgment, use a single model.
    • Define clear escalation triggers. Escalate decisions such as changes to core files, deletion operations, or judgments involving security boundaries to the flagship model. This helps prevent frequent advisor calls from eroding cost savings.
    • Instrument the workflow. Track each agent’s token usage, call count, and latency. Review the cost distribution regularly to confirm that most tokens are being processed on the lower-cost side as intended.
    • Validate with a mainstream orchestration framework. Common agent frameworks support “manager-and-worker” patterns with tool calls. Start with small workloads, measure how much quality is retained, and scale up gradually.

    One developer’s experience after a week of testing captures the trade-off well: a flagship model was unnecessary for routine coding, but exceptionally useful for planning. If you are using the most expensive model for every stage, consider a tiered approach: let stronger models focus on reasoning and more cost-effective models handle execution. It can improve both cost and speed.

    09040904

    Top articles