From Tokens to Routing: Why LLM Costs Spike and How to Govern Them

1. Why Bills Get Out of Control: Root Causes of Underestimated Costs

Many teams estimate LLM costs only by assuming a single price per call. After launch, the actual bill is often far higher than expected. The root cause is that LLM billing is not one-dimensional; it is determined by several overlapping factors. Without understanding how these factors interact, costs can grow nonlinearly as usage scales.

To control costs effectively, start with the correct mental model: LLM cost is never a simple multiplication of “model unit price × number of calls.” It is the combined result of the variables below.

2. A Complete Breakdown of LLM Cost Drivers

The real cost of an LLM service is shaped by several factors. Ignoring any one of them can cause budgets to spiral.

2.1 Input Tokens and Output Tokens

These are the most basic and often the most misunderstood items. For many models, the output token price is significantly higher than the input token price, sometimes by several times. This means:

• Generating long-form content is much more expensive than reading long-form content.
• Asking the model to produce chain-of-thought or step-by-step reasoning can quickly increase output-side costs.

When designing prompts, reduce redundant instructions and avoid making the model generate more content than necessary.

2.2 Cache-Hit Tokens and Cache-Miss Tokens

This is a cost blind spot for many teams and also the biggest optimization opportunity. Most mainstream services offer prompt caching. When the same long context, such as a system prompt or knowledge-base snippet, is reused, the cached portion can be billed at a much lower rate than standard input tokens.

The key difference is:

• Cache-hit tokens: significantly cheaper and suitable for stable, reusable context.
• Cache-miss tokens: billed at the standard input price or higher, usually occurring when the cache is first written or after it expires.

The optimization strategy is to place stable content, such as system role settings and long-lived background knowledge, at the beginning of the prompt and keep it unchanged. Place frequently changing content, such as user input, at the end to improve cache hit rate.

2.3 Long-Context Multipliers

Some services apply multiplier pricing to very long contexts. When input tokens exceed a certain threshold, additional tokens may be billed at double or even higher rates. Sending entire chat histories or full documents to the model can therefore cause costs to jump in steps.

Governance measures include truncating and summarizing conversation history, using retrieval-based recall to select only the most relevant fragments instead of the full text, and managing context windows in layers.

2.4 Reasoning Tokens

Models designed for complex reasoning often generate an intermediate “thinking process” before producing the final answer. These reasoning tokens are billed as output tokens but are usually invisible to users, making them easy to overlook. For tasks that do not require deep reasoning, such as classification or format conversion, forcing a high-reasoning model is wasteful.

2.5 Tool Calls and Search Calls

In AI agent and function-calling scenarios, one user request may trigger multiple model calls:

• The model decides to call a tool, which consumes one round.
• The tool returns a result, which re-enters the model context.
• The model reasons based on the result, consuming another round.

A task that appears simple can become several times more expensive than a single call because of repeated “think-call-rethink” loops. Web-search tools may also be billed separately.

2.6 Model Routing Strategy

Different tasks have very different model capability requirements. If every request is sent to the strongest model, costs will remain high. The core of model routing is to send each request to the most appropriate model tier based on task difficulty, latency requirements, and budget constraints.

2.7 Failure Retry Rate

Network jitter, timeouts, content moderation blocks, and format validation failures can all trigger retries. If the retry strategy is poorly designed, for example without backoff or with unlimited retries, failed requests will keep consuming tokens without producing useful results, creating hidden waste.

3. Treat Current Models as Samples, Not Conclusions

Model prices and model versions change rapidly. Today’s flagship model may be replaced tomorrow by a cheaper model with similar capability. Therefore, a price list should not be treated as a permanent conclusion. Instead, teams should master the full cost-analysis method based on input, output, cache, and routing.

The table below uses illustrative prices, per 1 million tokens, to show cost differences across capability tiers. It is only meant to demonstrate the calculation method. Before production, use the official pricing of the selected service.

Capability tier	Input price (illustrative)	Cache-hit price (illustrative)	Output price (illustrative)	Typical scenarios
Flagship reasoning model	Higher	About 1/10 of input price	Highest	Complex reasoning, agents, code, critical business
Lightweight/mini model	Low	About 1/10 of input price	Low	Summaries, regular generation, medium tasks
High-concurrency/light-reasoning model	Very low	About 1/10 of input price	Medium-low	Customer support, classification, intent recognition, high-concurrency scenarios

Two patterns are clear from the table. First, output price is usually several times the input price, so controlling output length is often more effective than controlling input alone. Second, cache-hit price is commonly around one-tenth of the standard input price, making caching one of the highest-return optimization areas.

4. Building a Model Tiering and Cost Routing System

A mature LLM system should not rely on only one model; it should have a model tiering and cost routing system. The following is a practical architecture approach.

4.1 Model Tiering

Divide available models into tiers based on capability and cost:

• L1 lightweight tier: classification, extraction, format conversion, simple Q&A.
• L2 standard tier: summaries, content generation, normal dialogue.
• L3 flagship tier: complex reasoning, code generation, multi-step agent tasks.

Requests should go to a lower tier by default. Only escalate to a higher tier when a lower-tier model has insufficient confidence or the task is judged to be highly difficult.

4.2 Routing Decision Logic

Routing can be driven by rules, a lightweight classification model, or metadata. The following is an illustrative routing code sample:

def route_model(task: dict) -> str:
    # 1. Hard rule: highly sensitive or highly complex tasks go to the flagship tier
    if task.get("complexity", 0) >= 8 or task.get("is_critical"):
        return "L3_flagship"
    # 2. Cache hit priority: stable context goes to a cache-supported channel
    if task.get("cache_hit_expected"):
        return "L2_standard_cached"
    # 3. Small structured tasks: route to the lightweight tier
    if task.get("type") in {"classify", "extract", "format"}:
        return "L1_light"
    # 4. Default standard tier
    return "L2_standard"

4.3 Cost Observability

Routing and tiering only work when supported by cost observability. At the gateway layer, record for each call: model tier, input and output tokens, cache-hit amount, whether retry occurred, and whether the call ultimately succeeded. Use logging, monitoring, and alerting capabilities to build cost attribution dashboards by business, model, and scenario, so teams can identify where money is spent and why.

5. A Practical Cost-Governance Checklist

Overall, controlling LLM bills can start with the following actions:

1. Compress output: streamline prompts and limit output length, prioritizing reduction of output tokens, which are often the largest cost driver.
2. Stable prefix plus caching: place unchanged context first and keep it unchanged to maximize cache hit rate.
3. Control context length: summarize and truncate conversation history to avoid triggering long-context multipliers.
4. Choose models by task: use tiered routing instead of always using the most expensive model.
5. Constrain reasoning depth: enable high-reasoning models only for tasks that genuinely require deep reasoning.
6. Optimize tool calls: reduce unnecessary multi-round agent loops and web searches.
7. Use robust retry strategies: apply exponential backoff, set retry limits, and distinguish retryable from non-retryable errors.
8. Establish cost attribution: make every token consumption traceable to a specific business use case or scenario.

6. Conclusion

The essence of LLM cost governance is turning opaque consumption into an engineering problem that can be decomposed, attributed, and optimized. By understanding how input and output tokens, caching, long contexts, reasoning, tool calls, routing, and retries interact, and by proactively managing workloads through model tiering and cost routing, enterprises can benefit from LLM capabilities while keeping bills under control. This input/output/cache/routing analysis framework remains valid even when specific model prices change and is worth building into long-term engineering capability.