Layered LLM Routing and Token Cost Optimization in Production

    This article explains how to model LLM costs across input, output, and cache tokens, fit rates from call logs, and route work through planning, reasoning, execution, and batch tiers. It also covers caching, batching, capacity planning, and monitoring so teams can budget multi-model stacks without treating list prices as the whole story.

    Layered LLM Routing and Token Cost Optimization in Production

    1. Why costs get harder once you mix models

    As LLM apps move from a single model to a mix of cheaper and stronger tiers, cost is no longer a simple comparison of price per million tokens. Production systems usually combine input tokens, output tokens, cache-hit tokens, retries, context compression, structured output, and async batching. When input and output are priced differently—and output is often much more expensive—a “cheap” model can become costly if it writes too much, while a higher unit price can still be reasonable for rare, high-stakes calls.

    The real gap is the missing map from task to model to cost to quality. Routing is often guesswork, capacity planning is anecdotal, and the bill only gets explained after the fact. A better approach is a reusable cost model: infer consumption rates from real logs, then design layered routing on top. This article is a general method and illustrative setup, not any vendor’s official price list. Actual unit prices, quotas, and cache rules should come from platform bills and documentation.

    2. Billing abstraction: a computable cost formula

    Whether the platform bills in tokens, credits, or another unit, you can treat cost as a weighted sum:

    1. Per-call cost C =
    2. input_tokens × input_rate
    3. + output_tokens × output_rate
    4. + cache_hit_tokens × cache_rate
    5. + other fixed or tiered fees

    To compare tiers, normalize. Take the cheapest tier’s input rate as 1, define output/input ratio r = output_rate / input_rate, and cache discount d = cache_rate / input_rate. Each tier becomes a price vector:

    1. price_vector = [input_rate, output_rate, cache_rate]
    2. relative_vector = [1, r, d]

    The router only needs relative vectors and a task profile. Planning and review work is infrequent and quality-sensitive, so a higher relative price can be acceptable. Bulk cleaning, classification, and summarization are high volume and more fault-tolerant, so they belong on cheaper tiers. The formula turns “feels cheap” into something you can compare, budget, and alert on.

    3. Fit consumption rates from call logs

    When a full price sheet is missing or rules change, fit rates from production logs:

    1. Collect logs: task type, input/output/cache tokens, actual credits or spend, retry flag, latency.
    2. Clean data: drop failures, extreme context lengths, promo-period bills, and samples that cross pricing tiers so all rows sit in one billing band.
    3. Fit a linear model: express spend as a combination of input, output, and cache tokens; use least squares or constrained regression.
    4. Validate: check goodness of fit, residual distribution, and cross-validation error. Residuals near zero and R² near 1 mean internal consistency, not that the sample covers every future case.
    5. Refit on a schedule: platform rules, cache policy, quotas, and model versions change. Store fit version and validity window; never treat coefficients as a permanent price list.

    Pseudocode:

    1. def fit_cost_rates(samples):
    2. # samples: [{input_tokens, output_tokens, cache_tokens, cost}]
    3. # Non-negative least squares for the three unit rates
    4. rates = constrained_least_squares(
    5. features=[s.input_tokens, s.output_tokens, s.cache_tokens],
    6. target=[s.cost for s in samples],
    7. non_negative=True
    8. )
    9. residual = compute_residual(samples, rates)
    10. return rates, residual

    Use fitted rates for internal routing and budgets, not as public “official prices.” Keep a measured-coefficient version and keep reconciling against the invoice.

    4. Layered routing: expensive models plan, cheap models execute

    With cost vectors and task profiles, a four-tier router is practical:

    • Planning/review: high-quality, high-cost models for requirement breakdown, plans, critical code review, and final QA. Cap call volume and force structured output to avoid long, low-value text.
    • Complex reasoning: mid-to-high-cost models for multi-step reasoning, disambiguation, and hard diagnosis. Control cost with prompt templates and output length limits.
    • Main execution: mid-cost models for most code generation, document work, extraction, and tool orchestration, with an emphasis on throughput and stability.
    • Batch/simple: lowest-cost models for classification, dedup, format conversion, short summaries, and bulk labeling, plus async queues and retry.

    Decide with rules first, then optionally learn:

    1. def route_task(task):
    2. score = complexity_score(task) + risk_score(task)
    3. if task.requires_final_review or score > HIGH_THRESHOLD:
    4. return "planning_tier"
    5. if task.requires_multi_step_reasoning:
    6. return "reasoning_tier"
    7. if task.is_batch and task.tolerance_high:
    8. return "batch_tier"
    9. return "execution_tier"

    Rules are explainable and easy to roll back. After enough logs, a lightweight model on cost and quality feedback can help. Always keep an upgrade path to a stronger tier so cheap models do not fail repeatedly on critical work and inflate retry spend.

    5. Cache, batching, and capacity

    Cache is easy to undervalue. Stable prefixes, fixed system prompts, templated instructions, and versioned knowledge snippets raise hit rates and cut repeated input cost. Cache keys must be stable, with clear invalidation and version isolation to avoid pollution. Batch work should merge requests, run asynchronously, cap output length, and retry with backoff and deduplication.

    For capacity, convert quotas: estimate input/output mix, apply the relative price vector to get a weighted token cost, then estimate requests per unit time. Run sensitivity analysis on higher output share, lower cache hit rate, and higher retry rate. Monitor at least per-task cost, cost per thousand tokens, cache hit rate, routing accuracy, retry rate, quality regressions, and budget alerts. Use canaries and A/B tests to check that cost cuts do not cut quality.

    6. Conclusion

    A multi-model stack is not about finding the lowest list price. It is a loop of measured cost models, layered routing, cache and batching, and continuous monitoring. High-cost models own planning and final review; low-cost models own bulk execution. Treat every fit as an internal estimate, reconcile it with invoices, and refresh it when rules change. Cost, quality, and capacity belong in one observability system if you want durable optimization without sacrificing output.