Tiered Model Routing: Assign Every Step of a Coding Task to the Right Model

1. The question changed: from “pick one model” to “allocate every step”

When discussing AI coding tools, the focus used to fall almost entirely on “which model is the strongest.” But once a single model family offers flagship, workhorse, and lightweight tiers, and the lightweight tier can itself handle million-token contexts, that framing breaks down: no single tier is the most cost-effective choice at every step.

A more engineering-realistic question is: into which steps has a task been broken down, what are the input shape, fault tolerance, and value density of each step, and which step is worth the most expensive inference.

The most typical shift during an update cycle like October is that vendors move their optimization focus from the mid-to-low tiers toward “supporting capabilities”: sub-agents can set their own inference-effort tier, the inference tier can be switched mid-conversation, and the usage panel points directly at which line item of spend can be cut. Each capability looks minor on its own, but together they point to the same conclusion — model tiers are shifting from a “selection parameter” to a “scheduling parameter.”

2. Capability and cost profile of the three tiers

In engineering practice, models are usually divided into three tiers. These are not a ranking of good versus bad, but a tradeoff across capability, cost, and latency:

Tier	Typical role	Suited steps	Main constraints
Flagship	Long-range planning, cross-module refactoring, complex architectural decisions	Solution design, hard-to-diagnose defect localization, high-stakes judgments in code review	Highest unit price and latency; unsuitable for high-frequency small tasks
Workhorse	Everyday coding mainstay, balancing capability and cost	Feature implementation, test writing, routine review, failure retries	Long sessions accumulate cost noticeably
Lightweight	Structured, localized, low-fault-tolerance tasks	Summarization, context compression, retrieval-hit judgment, sub-agent exploration, batch formatting	Weak on complex reasoning and long-range consistency, and carries hidden pricing conditions

A judgment that keeps being validated: the “cheapness” of the lightweight tier is usually conditional. A common industry practice is to price by the context size of each request in bands — short contexts get a low unit price, and beyond a certain threshold the unit price rises several-fold. This means putting the lightweight tier into a main conversation that has already accumulated hundreds of thousands of tokens quickly wipes out the on-paper advantage; conversely, putting it on a “single call, closed input, short output” subtask is where the cost-effectiveness genuinely holds.

3. Layering by task: a routable step list

To make routing actually land, first break a coding task into recognizable steps. Here is a fairly general layering:

1. Requirement understanding and solution planning — high value density, low fault tolerance. The input is the full context and the output determines the direction of every downstream step, so it must go to the flagship tier at the highest reasoning tier.
2. Implementation and rewriting — the standard scenario for the workhorse tier. Single-file or single-module changes with clear context, where failure cost is backstopped by tests.
3. Test writing and failure fixing — workhorse tier, but needing a relatively high reasoning tier. The fixing stage depends on cross-reasoning between error messages and code; too low a tier easily falls into an inefficient “change one line, fail, change one line again” loop.
4. Code review — workhorse tier or above. The value of review lies in catching omissions; too low a reasoning tier degrades conclusions into style hints.
5. Parallel exploration and retrieval judgment — the ideal scenario for the lightweight tier. Multiple sub-agents each look at only one candidate path, and the input is naturally isolated within a small context.
6. Summarization and context compression — must use the lightweight tier. This is a high-frequency call whose output is structured short text and depends almost not at all on deep reasoning.
7. Formatting, comment generation, commit-message generation — lightweight tier, clear rules, template-style output.

4. The cost model: why “just pick the cheap model” often saves nothing

Inference spend cannot be judged by unit price alone; you have to return to the full billing structure. A general approximation is:

Cost per task ≈ new input tokens × input unit price
             + cache-write tokens × cache-write unit price      (usually a multiple of the input unit price)
             + cache-hit tokens × cache-read unit price      (usually a fraction of the input unit price)
             + output tokens × output unit price
             + tiered surcharge (if a single request's context exceeds the threshold)

From this, three engineering conclusions follow:

First, prompt caching is the largest cost lever for long tasks. The essence of a long session is that “the same context is carried repeatedly.” If every request is billed on the full input, cost grows linearly with the number of turns; if the prefix is stable and hits cache reads, this portion of spend can drop by an order of magnitude. Many vendors have lowered their cache-read price precisely because it determines the real bill for agent-style workloads.

Second, cache hits depend on prefix stability. If anyone inserts timestamped content, a randomly ordered file list, or a changing environment variable into the system prompt, tool definitions, or the head of the conversation, the cache on that entire prefix is invalidated. The engineering practice is to front-load stable content, back-load volatile content, and monitor the cache hit ratio.

Stable prefix (cache-friendly): system prompt → tool definitions → repo structure and conventions
Volatile suffix (differs each time): current task description → recent conversation turns → live logs and errors

Third, the mainline of a long session should not be downgraded to the lightweight tier. Once context exceeds the band threshold, the lightweight tier’s unit-price advantage gets swallowed by the tiered surcharge, while its reasoning weakness gets billed a second time in the form of rework — and rework is itself token consumption.

5. Sub-agents and reasoning tiers: isolating context cuts cost

The value of sub-agents is often misunderstood as “speeding things up,” but it is equally a means of cost isolation: the main session keeps only a summary of conclusions, while the intermediate process (large amounts of file reading, logs, candidate comparisons) stays in the sub-session and is discarded after use.

Two supporting capabilities are worth using well:

• Sub-agents setting their own reasoning tier: exploratory tasks fan out in parallel at a low tier, then, once converged, the mainline makes the judgment at a high tier.
• Switching the reasoning tier mid-session: build the skeleton quickly at a lower tier first, then step up when entering tricky debugging or refactoring decisions, avoiding billing the entire task at the highest tier.

6. An actionable routing strategy

Writing these judgments as explicit configuration is far more reliable than deciding ad hoc in conversation:

# Bind each step to a tier and reasoning effort (illustrative; tier names depend on your platform)
routing:
  plan:            { tier: flagship, effort: high,   cache: prefix }
  implement:       { tier: primary,  effort: medium, cache: prefix }
  review:          { tier: primary,  effort: high,   cache: prefix }
  test_fix:        { tier: primary,  effort: high,   cache: prefix }
  subagent_search: { tier: light,    effort: low,    cache: none  }
  summarize:       { tier: light,    effort: low,    cache: none  }
  compress_ctx:    { tier: light,    effort: low,    cache: none  }
guardrails:
  max_context_per_light_call: 100k   # fall back to the workhorse tier once exceeded
  min_cache_hit_ratio: 0.6           # below threshold triggers a prefix-stability check
  escalate_on: [repeated_test_failure, cross_module_edit]

The corresponding routing entry point can be minimal:

LIGHT_CTX_LIMIT = 100_000  # Band-pricing threshold for the lightweight tier; illustrative value
def pick_model(step: str, ctx_tokens: int) -> tuple[str, str]:
    tier, effort = ROUTING[step]
    # The lightweight tier loses its advantage under long contexts due to the tiered surcharge
    if tier == "light" and ctx_tokens > LIGHT_CTX_LIMIT:
        return "primary", "medium"
    # The same test failing repeatedly signals a low reasoning tier; step up rather than retry
    if step == "test_fix" and failure_streak >= 3:
        return "primary", "high"
    return tier, effort

If a team uses model services from a cloud platform (for example, the model-service categories from Baidu AI Cloud), such strategies can usually be implemented by reusing the platform’s built-in call observability and alerting rather than building a collection pipeline from scratch — the key is instrumenting per step, not just tracking the total.

7. Measurement: making cost visible at the step level

Cost reduction without observability is just guesswork. At minimum you need to collect these metrics:

• Step-level token and cost distribution: locate the real cost sinks, usually “failed retries” and “full-context replays that miss cache.”
• Cache hit ratio: below expectation usually signals an unstable prefix, not a model problem.
• Average turns to complete a task: rising turns usually mean the reasoning tier is set too low.
• Rework rate: the proportion of the same requirement sent back to redo, a core metric for quality and safety boundaries.
• Context-length distribution of lightweight-tier calls: a large number of calls near the threshold means the band boundary needs adjusting.

A healthy target shape is: the mainline session keeps a stable cache hit ratio, repeated exploration sinks down to parallel sub-agents, summarization and compression go entirely through the lightweight tier, and retry counts stay controlled. Once these are achieved, the bill usually falls due to structural changes rather than unit-price negotiation.

8. Common pitfalls

• Treating the lightweight tier as the mainline workhorse: the bill looks cheap short-term, then gets canceled out mid-term by rework and context-compression cost.
• Ignoring channel differences: model aliases on hosted cloud, on-premises deployments, or third-party proxy channels may update on a different cadence than the vendor defaults. Validating behavioral consistency with a small regression task set before upgrading is safer than switching outright.
• Watching the model but not the context: most waste happens in repeatedly carrying long context that misses cache.
• Giving sub-agents an indiscriminate high tier: sub-agents are usually invoked concurrently many times, so tier choice should be more conservative than for the mainline.
• Using total cost as the sole KPI: it encourages over-downgrading. Quality metrics (rework rate, review defect density) must be read together with cost metrics.

9. Summary

More models do not bring decision paralysis; they bring scheduling opportunity. What truly determines cost and quality are three judgments: a task’s value density sets the ceiling on its tier, context size determines a tier’s feasibility, and cache stability determines the shape of the bill. Writing these three into configuration and monitoring, rather than weighing them ad hoc in the chat box every time, is the engineering habit most worth cultivating during an upgrade cycle like this.