Multi-Model Layered Routing: Token Cost Modeling and Practical Strategies

    This article explains how to model token consumption across multiple AI model tiers by separating input, output, and cached token costs, and how to use the resulting cost model to design a practical routing strategy. It presents a linear regression approach based on real call logs, a sample coefficient table, a routing policy that assigns high-cost models to planning and review while offloading execution to cheaper tiers, and a closed-loop calibration mechanism with pseudocode. The methodology helps engineering teams balance quality and budget in multi-model systems without relying on gut-feel pricing comparisons.

    Multi-Model Layered Routing: Token Cost Modeling and Practical Strategies

    1. Background: Cost Differences Amplify When Model Capabilities Are Tiered

    Integrating models of different capability levels into the same business system has become a common engineering practice:

    • High-capability models handle requirement understanding, solution planning, and final review.
    • Mid-capability models handle complex reasoning and difficult problem analysis.
    • Low-cost models handle mainstream execution tasks such as code generation, document processing, and data transformation.
    • Lightweight models handle simple tasks like classification, deduplication, format validation, and batch cleaning.

    The direct benefit is a balance between quality and cost. However, the challenge is that input, output, and cache consumption coefficients across tiers can differ by orders of magnitude. Judging cost by “does a single call feel expensive?” easily underestimates the impact of output length, context length, and cache hits on the total budget.

    This article addresses a general technical question: how to derive relative token consumption coefficients for each model tier from your own call logs, build a calculable cost model, and design layered routing rules based on that model. The numerical values used here are anonymized examples and fitted results—they illustrate the methodology and do not represent any official pricing schedule. Rules may change in the future.

    2. Building a Relative Cost Model: Meter Input, Output, and Cache Separately

    The first step in cost modeling is to decompose “budget consumption” into observable variables. For a single call, at minimum record: input tokens, output tokens, cached input tokens, task type, routing tier, and final budget consumption.

    Let the cost coefficients for a tier be:

    • ci — input coefficient (budget units per million input tokens)
    • co — output coefficient (budget units per million output tokens)
    • cc — cached-input coefficient (budget units per million cached input tokens)

    The estimated cost of a call is:

    1. cost = input_tokens / 1e6 * ci
    2. + output_tokens / 1e6 * co
    3. - cached_tokens / 1e6 * (ci - cc)

    The cached term accounts for the fact that when a prefix is cached, the input cost drops from the normal input coefficient to the cache coefficient.

    Ignoring caching for a moment, if the traffic mix is 80% input and 20% output, the weighted per-million-token coefficient is:

    1. weighted = 0.8 * ci + 0.2 * co

    Given a budget B for a 5-hour window, the estimated token throughput for that tier is:

    1. throughput = B / weighted * 1e6

    As an example, with a budget of 60,000 units per 5 hours, the following anonymized fitted coefficients were obtained:

    Tier Typical Responsibility Input Coefficient Output Coefficient Cached Input Coefficient Weighted Coefficient Estimated Throughput (5h)
    L1 Planning & Review Requirement decomposition, architecture design, final review 80,000 400,000 — 144,000 ~417,000 tokens
    L2 Complex Reasoning Multi-step reasoning, difficult troubleshooting, trade-off analysis 32,000 112,000 — 48,000 ~1.25M tokens
    L3 Main Execution Code modifications, document generation, data transformation 4,000 8,000 80 4,800 ~12.5M tokens
    L4 Batch & Lightweight Classification, deduplication, format validation, field extraction 2,000 2,000 — 2,000 ~30M tokens

    This table reveals two key facts. First, output cost is a major amplifier of total cost: L1’s output coefficient is 400,000, while L4’s input and output are both 2,000—a 200× difference in output cost per token. Second, cache hits matter significantly at the low-cost execution tier: L3’s normal input coefficient is 4,000, but the cached-input coefficient is only 80, i.e., 2% of the normal input cost. Therefore, cost optimization must not focus only on input unit price; it must also control output length, improve reusable prefix cache rates, and estimate budget throughput per tier separately.

    3. Linear Fitting Is Useful but Not Absolute

    If regression on your actual call history yields R² = 1.0 and residuals near zero, it indicates that within the observed range, budget consumption has a strong linear relationship with input, output, and cache tokens. This is sufficient for engineering-level budget estimation and routing decisions. However, you must define the boundaries of the model: a linear model is only valid for the observed context ranges, output distributions, and cache policies. When model versions, pricing rules, cache policies, concurrency limits, or billing granularity change, the coefficients must be recalibrated. Do not extrapolate to untested long contexts, extremely long outputs, or multi-tenant mixed traffic.

    A more robust practice is to establish a rolling calibration mechanism:

    • Aggregate call logs daily or weekly.
    • Re-estimate the coefficients for each tier.
    • Monitor the residual between predicted cost and actual budget consumption.
    • Trigger alerts when residuals exceed a threshold.
    • Investigate whether output length drift, cache hit-rate decline, or routing ratio anomalies have occurred.

    4. Layered Routing Strategy: High-Cost Models Think, Low-Cost Models Do

    The value of a cost model lies in guiding routing decisions. A practical strategy is:

    1. L1 Planning & Review tier: Use for requirement breakdown, technical design, critical code review, and final quality gate. These calls are infrequent but high-impact, so high-cost models are justified. However, output length must be constrained, and structured conclusions should be required to prevent high-cost models from performing batch generation.

    2. L2 Complex Reasoning tier: Use for multi-step reasoning, difficult problem localization, trade-off analysis, and complex logic validation. This tier is cheaper than L1 but more expensive than execution tiers. Set confidence thresholds and escalation conditions to avoid routing all tasks to mid/high-cost tiers by default.

    3. L3 Main Execution tier: Use for high-frequency tasks such as code modification, document generation, test generation, data transformation, and summarization rewriting. This is the throughput backbone. Prioritize caching, context compression, and strict output format constraints to spend budget only on necessary content.

    4. L4 Batch & Lightweight tier: Use for classification, deduplication, format validation, field extraction, and batch cleaning. These tasks can be processed in parallel by ultra-low-cost models, with quality controlled through sampling.

    Routing decisions should not be hard-coded by task name alone. They should combine:

    • Risk level
    • Verifiability
    • Output length limits
    • Historical failure rate
    • Budget remaining

    For example:

    • High-risk tasks that cannot be automatically verified → L1.
    • Multi-step reasoning but auto-verifiable → L2.
    • Execution tasks with clear tests or format checks → L3.
    • Batch tasks with clear rules and sample-based checks → L4.

    If a low-cost tier yields low confidence, fails validation, or the user marks the task as high priority, escalate to a higher tier if budget allows. If the budget is tight, downgrade and narrow the task scope instead.

    5. Pseudocode: Cost Estimation and Routing Loop

    1. function estimate_cost(layer, input_tokens, output_tokens, cached_tokens):
    2. ci, co, cc = coefficients[layer]
    3. return input_tokens/1e6*ci + output_tokens/1e6*co
    4. - cached_tokens/1e6*(ci - cc)
    5. function route(task, budget_left):
    6. if task.risk == "high" or task.needs_final_review:
    7. layer = "L1"; max_output = 2000
    8. elif task.complexity == "high" and task.auto_verifiable == false:
    9. layer = "L2"; max_output = 4000
    10. elif task.has_tests or task.has_format_check:
    11. layer = "L3"; max_output = 8000
    12. else:
    13. layer = "L4"; max_output = 2000
    14. predicted = estimate_cost(layer, task.input_tokens,
    15. min(task.expected_output, max_output),
    16. task.cached_tokens)
    17. if predicted > budget_left:
    18. layer = downgrade(layer)
    19. return layer, max_output

    This pseudocode embodies three principles:

    1. Select the tier based on task risk and verifiability first.
    2. Control cost with an output length cap.
    3. Use remaining budget as a safeguard for downgrading.

    A production system should also record escalation, downgrade, and retry counts to avoid a situation where repeated failures at a low-cost tier result in higher total cost than a single correct high-tier call.

    6. Observability and Calibration: Keeping the Cost Model Useful

    Routing must be observable. Logs should include at minimum:

    • Task type
    • Routed tier
    • Input/output/cache token counts
    • Estimated cost
    • Actual budget consumption
    • Latency
    • Quality score
    • Validation result
    • Escalation/downgrade reason

    Aggregate by tier and task type, then periodically re-fit coefficients and monitor residuals, cache hit rates, output length distributions, and escalation rates. When deploying on Baidu AI Cloud, you can use container platforms for stateless routing services, log services for call details, and monitoring/alerting for budget consumption and residual anomalies. These general-purpose capabilities help close the loop between estimation, execution, observation, and calibration.

    7. Risks and Boundaries

    Finally, several caveats:

    • The numerical values in this article are personal fitted examples, not official pricing data. Rules may change in the future.
    • Task quality requirements, output lengths, and cache reuse rates vary widely; fixed ratios should not be applied blindly.
    • Cost optimization must not sacrifice necessary quality. Using high-cost models for planning and review, and low-cost models for execution and batch work, is an engineering trade-off, not an absolute rule.

    Only with continuous observation, rolling calibration, and clear budget/quality thresholds can you achieve both controlled cost and stable quality in a multi-model layered invocation system.