LLM Tiered Routing: A Practical Guide to Controlling AI Coding Costs

    When using large language models for coding tasks or AI agents, token consumption often far exceeds intuition because a single task can involve multiple rounds of planning, tool calls, and review. This article explains the billing dimensions that drive cost—input tokens, output tokens, cache hits, and quotas—shows how to estimate per-task cost with a simple formula, and introduces a tiered routing strategy that assigns models by task difficulty. It also covers engineering techniques for reducing costs, such as improving cache hit rates, limiting output length, and compressing context, along with key metrics like cost per successful task and common pitfalls to avoid.

    LLM Tiered Routing: A Practical Guide to Controlling AI Coding Costs

    1. Why Cost Should Be Measured Per Task, Not Per Call

    When developers use LLMs to assist with coding or build agents, they tend to focus only on answer quality and rarely examine the structure of their bills. A seemingly simple code change may actually involve multiple rounds of planning, tool calls, file reads, and result reviews. Each round sends back large chunks of context, and token consumption often far exceeds intuition.

    The root of the cost problem is that billing involves more than one dimension. Common billing structures include input tokens, output tokens, and cached input tokens; some services also meter by time windows or quotas. Judging cost by a single number easily leads to wrong conclusions. Cost analysis should therefore start from “how many resources did a single task consume,” not from “whether the unit price looks high.”

    2. Three Key Dimensions of the Billing Structure

    2.1 Input vs. Output Pricing

    In most usage-based pricing systems, output tokens cost more than input tokens. Generation is autoregressive—each output token requires incremental computation—so every output token is significantly more expensive to produce. This means asking the model to write long explanations, verbose reasoning, or full blocks of repeated code directly inflates costs.

    2.2 Cache Hits

    When a request’s prefix matches a previous request exactly, the server can reuse already-computed intermediate results; this portion is called a cache hit. Cached tokens are usually priced significantly lower than regular input tokens. In agent-style applications, system prompts, tool definitions, and project context repeat in every request, so the cache hit rate has a major impact on total cost.

    2.3 Quotas and Credit Conversion

    Subscription or plan-based services typically convert consumption across different models into unified credits or quotas. The same number of requests can consume credits that differ by several times depending on the model. When evaluating costs, rely on actual consumption reported by the billing system or platform—not the nominal number of calls.

    3. Estimating Per-Task Cost with a Formula

    The cost of a single call can be expressed as:

    1. cost per call = (uncached input tokens × input price
    2. + output tokens × output price
    3. + cached tokens × cached price) / 1,000,000

    The following pseudocode implements this calculation. Here, input tokens refer only to the uncached portion; cached and uncached tokens are not double-counted.

    1. PriceTable:
    2. input_per_m // price per 1M input tokens
    3. output_per_m // price per 1M output tokens
    4. cache_per_m // price per 1M cached tokens
    5. Usage:
    6. input_tokens // uncached input tokens
    7. output_tokens // output tokens
    8. cached_tokens // cache-hit tokens
    9. function estimate_cost(usage, price):
    10. total = usage.input_tokens * price.input_per_m
    11. + usage.output_tokens * price.output_per_m
    12. + usage.cached_tokens * price.cache_per_m
    13. return total / 1000000

    3.1 Estimating Relative Cost of Mixed Tasks

    Real tasks contain both input and output. Assume an input-to-output token ratio of 4:1, ignoring caching for now. If output tokens cost 5× input tokens, the relative cost of a mixed task is:

    1. relative cost = 0.8 × 1 + 0.2 × 5 = 1.8

    If output tokens cost 8× input tokens, the relative cost becomes:

    1. relative cost = 0.8 × 1 + 0.2 × 8 = 2.4

    This shows that even small changes in the output ratio noticeably affect total cost. When designing tasks, controlling output length is just as important as controlling input length.

    Note: The numbers above only illustrate the calculation method and do not represent the pricing of any specific service.

    4. Tiered Routing: Assigning Models by Task Difficulty

    4.1 The Basic Idea

    The core of tiered routing is to assign models based on task difficulty and the cost of failure—not to always use the most capable or the cheapest model. A common approach splits work into four tiers:

    Tier Typical Tasks Selection Criteria Call Frequency
    Planning & review Requirement decomposition, solution design, final checks Strong reasoning, quality first Low
    Complex reasoning Hard problem analysis, cross-module design Sufficient reasoning depth, acceptable cost Low to medium
    Execution Code writing, tool calls, log analysis Fast, cheap, stable High
    Batch processing Format conversion, field extraction, simple classification Lowest price, high throughput Very high

    The logic behind this tiering: expensive models handle directional judgment, while cheap models handle large volumes of repetitive work. Expensive models are called rarely, keeping overall cost manageable; cheap models absorb the vast majority of tokens, amplifying their price advantage.

    4.2 Routing Example

    The routing layer selects a model based on task type. Pseudocode below—model names are placeholders and should be replaced with actual model identifiers in practice.

    1. TIER_MODELS = {
    2. "plan": "planner-model", // planning & review tier
    3. "reason": "reasoner-model", // complex reasoning tier
    4. "execute": "executor-model", // execution tier
    5. "batch": "batch-model", // batch processing tier
    6. }
    7. function route(task_type):
    8. if task_type in TIER_MODELS:
    9. return TIER_MODELS[task_type]
    10. return TIER_MODELS["execute"] // unknown tasks default to execution tier

    4.3 Fallback and Escalation Mechanisms

    Tiered design must be paired with fallback mechanisms. Typical practices include:

    1. When execution-tier output fails validation, automatically escalate to the complex reasoning tier and retry.
    2. Set a maximum retry count to prevent endless back-and-forth between two models that burns tokens without results.
    3. Define escalation triggers explicitly—for example, two consecutive test failures, or output that does not match the agreed format.

    Without fallback mechanisms, tiered routing tends to produce large volumes of incorrect output on cheap models, followed by expensive models reworking everything—raising total cost instead of lowering it.

    5. Engineering Techniques to Reduce Costs

    5.1 Improve Cache Hit Rates

    Cache hits depend on prefix stability. Recommended practices:

    • Place system prompts, tool definitions, and project conventions at the beginning of requests, and keep their content and order stable.
    • Put content that changes each round—such as the current task description or the latest error message—at the end of the request.
    • Avoid injecting dynamic fields like timestamps or random IDs into prompts; otherwise the prefix changes every time and the cache is invalidated.

    5.2 Control Output Length

    Output tokens are expensive, so actively constrain output:

    • Ask the model to return results in structured formats—fixed fields or code patches—to cut irrelevant explanations.
    • Set output limits via request parameters to prevent endlessly long responses.
    • State explicitly in the prompt that only the modified parts should be returned, not the entire file.

    5.3 Compress Context

    Long conversations make each request’s input larger and larger. Common compression approaches:

    • Rolling summaries of conversation history, keeping only key decisions and unresolved issues.
    • Pass only the file fragments relevant to the current step, not the entire codebase.
    • Preprocess long tool logs, extracting error lines and their surrounding context windows.

    5.4 Reduce Wasted Calls

    Many calls could be handled locally. For example, syntax checks, format validation, and unit tests can all run before calling the model. If you hand raw compilation errors straight to the model, it often needs multiple correction rounds—each one billed. Filtering obvious errors locally first significantly reduces the number of model calls.

    5.5 Merge Similar Tasks

    If there are multiple structurally similar small tasks—such as adding docstrings to many functions in batch—combine them into a single request. When merging, mind the context length limit to avoid failed requests.

    6. Building a Cost and Quality Measurement System

    6.1 Key Metrics

    Track at least the following:

    • Average tokens per task, broken down into input, output, and cache hits.
    • Call counts and cost share per tier.
    • Cache hit rate.
    • Task success rate and number of manual reworks.
    • Cost per successful task.

    6.2 Cost per Successful Task

    Judging by single-call price alone leads to misjudgment. A cheap model with a low success rate requires many retries, and its total cost may exceed that of a more expensive model that succeeds on the first try. The more meaningful metric is cost per successful task:

    1. function cost_per_success(records):
    2. total_cost = sum of record.cost for all records
    3. success_count = count of records where record.success is true
    4. if success_count == 0:
    5. return infinity
    6. return total_cost / success_count

    6.3 Limitations of Fitting and Extrapolation

    Some practitioners run regression fitting on historical bills and token usage to reverse-engineer consumption coefficients for each model. This can help you understand the billing structure, but two caveats apply. First, fitted results only reflect the current sample and current rules; even with high goodness of fit, they do not guarantee accurate future predictions. Second, once the provider changes its billing rules, existing coefficients become invalid. Re-validate periodically with fresh billing data.

    7. Common Pitfalls

    Pitfall 1: Comparing unit prices only. A low unit price does not mean low total cost. Judge comprehensively based on token structure, success rate, and retry counts.

    Pitfall 2: Using the strongest model for everything. Using high-capability models for simple tasks wastes resources and is usually slower to respond.

    Pitfall 3: Using the cheapest model for everything. When complex tasks are handled by low-capability models, rework increases, and the labor and time costs can wipe out the savings.