Division of Thoughts (DoT): Let Large Models Plan and Small Models Execute

    Division of Thoughts (DoT) is an approach that separates planning from execution in LLM applications. Instead of routing every task through a single large model, DoT uses a planner to decompose complex requests into a dependency graph, an adapter to route each subtask to a suitable model based on difficulty, and executors that run independent subtasks in parallel. This article explains the three roles, the workflow behind them, why DoT improves cost, latency, and parallelism, the key engineering considerations for building it, and the workloads it fits best.

    1. Starting from the “CEO” analogy: why a strong model shouldn’t do everything

    Imagine you are the CEO of a public company that has just landed a complex project—say, building a new game from scratch. You face two ways of working.

    The first is “getting personally involved”: as CEO, you write every line of code, draw every piece of concept art, and run every round of testing yourself. The result is usually that you are completely burned out (huge compute and effort consumption), labor costs stay high, and because one person can’t keep up, the whole project moves slowly (high response latency). This is exactly what many teams experience when they use a single large-parameter model to handle every task: capable, yes, but it wastes valuable “premium compute” on large amounts of simple, repetitive work.

    The second is “directing from the sidelines”: as CEO, you do just one thing—make a plan. For example, “this project has three steps: design the gameplay, build the engine, then finish the art.” Then you call in several specialists and hand off the concrete tasks: “You handle the code, you handle the drawings.” You keep only the core decision-making authority, while the hard, low-level work goes to cheaper, more efficient members—and several of them can even work in parallel.

    This is the core idea of Division of Thoughts (DoT). Its essence is to decouple planning of thought from execution of thought: a strong model handles high-level strategic decomposition, while lightweight models handle tactical execution, with each doing its own job in a coordinated, closed loop.

    2. The three roles of DoT and its workflow

    Drawing on the general design approach of this class of methods, DoT is typically built around three key roles that form a clear processing pipeline.

    2.1 Planner: turning a problem into a flowchart

    Faced with a complex user request (for example, “plan an anniversary dinner and work out the budget”), the planner doesn’t rush to answer. Instead, it first breaks the problem into a set of ordered subtasks, explicitly marks the dependencies between them, and ultimately forms a directed acyclic graph (DAG).

    An illustrative planner output structure looks like this:

    1. # Planner output: task orchestration (DAG structure)
    2. plan = [
    3. {"id": 1, "desc": "Decide the menu", "deps": []}, # Start point, no prerequisite
    4. {"id": 2, "desc": "Look up ingredient prices", "deps": [1]}, # Depends on step 1
    5. {"id": 3, "desc": "Calculate total budget", "deps": [2]}, # Depends on step 2
    6. {"id": 4, "desc": "Plan the ambiance", "deps": []}, # Can run in parallel with steps 1 and 2
    7. ]
    8. # Derive the batches that can run in parallel from the deps fields
    9. # Batch 1: id=1, id=4 (independent, can start at the same time)
    10. # Batch 2: id=2 (depends on id=1 completing)
    11. # Batch 3: id=3 (depends on id=2 completing)

    The key value of this stage is not just “decomposition” but also clarifying the dependencies: which step must come first and which steps can happen simultaneously are all expressed in a structured way. Dependencies determine the degree of parallelism in scheduling, and they are the basis for compressing overall latency downstream.

    2.2 Adapter: matching the “right person” to each task

    Once the tasks are broken down, who does the work? This is the responsibility of the adapter. Acting like an HR manager, it evaluates the difficulty and required capabilities of each subtask and routes a suitable execution model accordingly:

    • Simple tasks (keyword extraction, formatting, simple summarization): assigned to lightweight small models—low cost, fast.
    • Difficult tasks (logical reasoning, code architecture design, solving complex constraints): assigned to large-parameter strong models—guaranteed quality.

    The core principle here is “spend strong compute where it matters most.” The large share of simple tasks is pushed down to cheap models, and only the parts that genuinely need deep reasoning invoke the strong model, striking a balance between overall quality and total cost.

    2.3 Executor: running in parallel and aggregating results

    Following the adapter’s assignments, the executor processes each subtask. Because the planning stage has already produced the dependency graph, the scheduler can identify batches of tasks that “don’t depend on each other” and dispatch them in parallel to multiple execution instances; for tasks with dependencies, it advances strictly in topological order. The results of all subtasks are finally aggregated and integrated according to their dependencies, producing a complete answer the user can understand.

    3. Why DoT works: three optimizations in cost, latency, and parallelism

    The value of DoT is not just conceptual packaging—it can be broken down into three clear benefit dimensions.

    The first is cost optimization. The traditional approach uses a single strong model for end-to-end tasks, paying the same “premium compute” regardless of difficulty. Through difficulty-based routing, DoT sends the large share of simple tasks to cheap models and invokes the strong model only for the few genuinely hard parts, reducing overall invocation cost.

    The second is latency reduction. End-to-end serial invocation means the user must wait for all inference steps to complete one after another. DoT, based on the DAG, identifies tasks that can run in parallel and executes the steps that could “start at the same time” concurrently, significantly shortening end-to-end response time.

    The third is capability fit and controllability. Separating “planning”—a high-value, low-frequency action—from “execution”—a high-frequency action—lets each layer be independently replaced and independently optimized. Adjusting the planning strategy doesn’t affect the execution models, and upgrading the execution models doesn’t touch the planning logic, so the system is easier to maintain and iterate.

    4. Key considerations in engineering practice

    When actually building a DoT system, a few points are hard to avoid.

    1. Accuracy of the dependency graph. The entire parallel scheduling rests on a correct DAG. If the planner wrongly marks tasks that have hidden dependencies as independent, it can introduce errors during execution. So validating dependencies (cycle detection, checking the legality of topological ordering) must not be skipped.

    2. Criteria for difficulty-based routing. How does the adapter judge whether a task is “hard” or “easy”? A common approach is to score it based on features such as task type, input length, and domain tags, then set a threshold to split traffic. Routing that is too aggressive wrongly assigns hard problems to weaker models and degrades quality; routing that is too conservative loses the point of cost optimization. It needs repeated calibration in practice.

    3. Failure and retry mechanisms. Distributed execution naturally comes with uncertainty. Any subtask failure should support bounded retries, or, when necessary, escalate that subtask to a stronger model for re-execution, avoiding a single point of failure dragging down the whole flow.

    4. Result integration and consistency. How to merge the outputs of multiple parallel branches needs integration logic designed in advance, to prevent contradictory or duplicated content.

    5. Typical use cases

    DoT is especially well suited to workloads where “tasks can be decomposed, subtasks differ noticeably in difficulty, and some stages can run in parallel”—for example, multi-step data processing and report generation, complex question answering with branching decisions, tasks that combine retrieval with reasoning, and various kinds of structured content production. For these scenarios, DoT can often achieve better cost and latency performance while preserving result quality.

    6. Conclusion

    The idea of DoT can be summed up in one sentence: let the strongest mind do the planning, and let the most suitable hands do the execution. It borrows from the management wisdom of “separating decision-making from execution” in human organizations and transfers that wisdom to large-model applications. Through the closed-loop division of “the planner decomposes dependencies, the adapter matches difficulty, the executor runs in parallel,” a team can retain the reasoning depth of the strong model while also reaping the cost and efficiency benefits unlocked by lightweight models. Understanding and making good use of DoT is, at its core, about thinking through a more general engineering question—how to precisely direct limited and expensive intelligence to where it is needed most.