From Tokens to Routing: Diagnosing and Controlling LLM Cost Spikes

After an LLM goes live and the bill suddenly steepens, teams often suspect that “the model got more expensive.” When the call logs are laid out, the more common causes are longer single requests, retries that amplify calls, full system prompts resent on every turn, unstable cache keys, and simple classification tasks routed to high-capability models. Unit price is only a multiplier; what actually gets amplified is token volume and model selection. Cost governance has to answer three questions at the same time: how many tokens were billed, how many were billed repeatedly, and whether this model should have been used at all.

1. Break the Bill Down into Four Explainable Components

The industry generally bills by tokens. The cost of a single call can be roughly expressed as:

Cost ≈ uncached input tokens × input price + cached input tokens × cache price + output tokens × output price + other add-ons

Add-ons may include long-context surcharges, tool calls, price differences between batch and real-time channels, and similar items. The provider console’s current rules should be treated as authoritative; a sample price from one call should not be treated as a long-term conclusion. Model names and unit prices will change, but the analysis line of “input / output / cache / routing” remains valid over time.

Each of the four components maps to a different engineering lever:

• Input tokens: system prompts, conversation history, retrieved chunks, tool results, and the user’s original text.
• Output tokens: answer length, whether reasoning is billed, and whether long structured output is forced.
• Cache hits: whether a stable prefix can be recognized as the same reusable context.
• Routing: whether this task really needs a high-price model.

Optimizing only one component may still leave the bill supported by another. For example, the cache hit rate may be high, but if every answer generates thousands of tokens, the output side will still dominate cost.

2. Input Side: More Context Is Not Always Better

Input costs are often high because “all history is sent every time.” In multi-turn conversations, if the full system prompt, all history, and the entire retrieval result are concatenated as-is, tokens will grow approximately linearly—or even superlinearly—with the number of turns. A more robust approach is to manage context in layers.

1. Separate the stable prefix from the variable suffix. Role definitions, output formats, safety boundaries, and other rarely changing content should go first and remain byte-stable. User questions, retrieved chunks, and the current turn’s tool results should go after it. Prefix jitter directly breaks caching.
2. Summarize history; do not replay it indefinitely. After a fixed window, replace early turns with a short summary, keeping only the most recent turns in raw form plus key entities. The summary itself should also have a length limit.
3. Trim retrieved chunks before sending them. Truncate by relevance, remove duplicate paragraphs, navigation text, and tables unrelated to the question. Citation numbers can be kept; the full source text does not need to be pasted.
4. Structure and then compress tool returns. Extract fields from logs, web pages, and database results first, rather than stuffing raw long text back into the model.

An illustrative length budget can be expressed in pseudocode. The units are only token estimates, not an interface from any specific platform:

budget_in = 8000
prefix = stable_system_prompt()          # keep as stable as possible
recent = last_turns(n=4)
memory = summarize(older_turns, max=600)
evidence = top_k_chunks(query, k=5, max_each=400)
user = current_user_message()
prompt = join(prefix, memory, recent, evidence, user)
prompt = truncate_from_low_priority(prompt, budget_in)

Low priority is usually early retrieved chunks and old history, not the user’s current question. The truncation policy should be written into code and instrumented; otherwise, the budget may be exhausted in production without anyone noticing.

3. Output Side: Generation Length Is the Most Overlooked Multiplier

Output unit prices are usually higher than input unit prices, so “the model says one more sentence” is more expensive than “it reads one more sentence.” Common amplifiers include no maximum length, requiring long reasoning before a conclusion, full regeneration after a failure, and evaluation scripts that raise temperature and cause answers to expand.

Actionable constraints include:

• Set output caps for each task type. Classification, extraction, and scoring should use short outputs; explanation tasks can be looser.
• If a structured field can express it, do not require prose. Keep field names fixed and defaults explicit so truncation and validation are easier.
• Do not blindly replay the whole request on retry. First determine whether the issue is a timeout, rate limiting, or invalid content. If content is invalid, regenerate only the noncompliant part and lower the temperature.
• Streaming output is billed too. If cancellation still produces generated tokens, stop as early as possible on the client and record the cancellation ratio.

A simple regression check is to compare average output tokens before and after optimization on the same fixed set of questions, rather than only judging whether the answers “look good.” Quality should be protected by sampling or automated scoring; cost should use percentiles (P50, P95) to see the tail. Bill spikes often come from a small number of extremely long requests, and averages hide them.

4. Caching: A Hit Requires a Stable Prefix

The core of prompt caching is not “storing the answer.” It is that the provider or inference service recognizes a repeated input prefix, bills the matched portion at a lower unit price, and may shorten prefill time. It requires three conditions: the prefix is long enough, the content is stable, and the cache key or prefix matching rules line up. Common business-side failure causes are very specific.

• The system prompt contains timestamps, random numbers, user nicknames, or request IDs.
• Retrieval results arrive in a different order each time, or the same paragraph is rewritten with different whitespace or punctuation.
• Multi-tenancy concatenates different customers’ rules into the same prefix, leaving almost no sharing.
• The cache window is shorter than the business’s repetition cycle, so reuse between peaks is cleared.

During governance, split the prompt into a “cacheable template” and “per-request variables.” Manage the template with a version number, and keep variables only in the suffix. The same customer support or policy Q&A use case can share a template; personal data should not go into a prefix that can be reused across users. Cache hit rate should be a first-class metric: cached tokens / input tokens. Also look at the amount saved, to avoid a misleadingly high rate where the hits are all extremely short prefixes.

Caching is not a correctness solution. After a template upgrade, actively invalidate old versions to prevent mixed old and new rules. Strongly time-sensitive data such as permissions, prices, and inventory must remain in the non-cacheable suffix, and the answer should state the time range of the data.

5. Routing: Choose Models by Task Difficulty, Not by Habit

Routing answers the question “should a high-price model be used at all?” In practice, many requests are classification, rewriting, information extraction, format conversion, and simple Q&A. Only a small number require long-chain reasoning, complex code, or multi-step tool calls. If everything defaults to the flagship model, the bill rises in proportion to call volume while quality gains diminish at the margin.

A practical split can use three layers, named by capability tier rather than specific model:

• Light tier: labels, intent, sensitive-content judgment, short summaries, field extraction.
• Standard tier: ordinary generation, multi-turn customer support, routine code completion.
• High-capability tier: long tasks, cross-document reasoning, drafts for critical business decisions, complex troubleshooting.

Routing evidence should be as observable and replayable as possible: input length, whether code or tables are present, whether the user explicitly asks for deeper analysis, whether the previous turn failed, and the risk level from a rules engine. Do not rely only on a black-box classifier; otherwise a cost problem becomes an unexplainable quality problem.

if risk == "high" or task in {legal_review, incident_debug}:
    tier = "high"
elif input_tokens > 12000 or needs_tools:
    tier = "standard"
else:
    tier = "light"
answer = call_model(tier, prompt, max_output=limits[tier])
if not passes_check(answer) and tier != "high":
    answer = call_model(next_tier(tier), prompt, max_output=limits[tier])

Escalation should happen only after validation failure, and the number of escalations should be limited—for example, at most once per request. Otherwise, routing degenerates into “try cheap first, then send everything to the most expensive model,” which raises total cost. Validation can check whether the format is valid, required fields are present, and citations exist, rather than calling another more expensive model for open-ended scoring.

Batch, offline, and delay-tolerant tasks should be separated from online interactive traffic. Offline work can tolerate longer waits and higher aggregation; online traffic is where latency premiums are paid. Moving report summaries and knowledge-base pre-generation to low-peak batch processing is often cheaper than calling online item by item.

6. Observability, Budgets, and Regression

Optimization cannot be sustained without observability. At minimum, record these fields and aggregate them by application, tenant, and feature rather than only looking at the total account amount:

• Request count, failure count, retry count, and number of escalations to a higher tier
• Input tokens, cached tokens, and output tokens
• End-to-end latency and time to first token
• Estimated cost, plus the cost of P95 requests
• Feature identifier and prompt version

Budgets should be attached to features, not just a single monthly account cap. For example, set a “daily cost cap for the support assistant” and a “daily token cap per user for the code assistant.” Define the policy for hitting the cap in advance: downgrade to the light tier, shorten context, queue the request, or reject noncritical features and ask the user to try again later. Silent failure hurts the experience, while unlimited retries can blow through the bill during an outage.

Before launch, run cost regression on fixed samples: same set of requests, same prompt version, compare token distribution and sampled quality. If a prompt or retrieval change doubles P95 input, it should be treated as a change requiring review, not a casual commit. When the price table updates, replay the last seven days of usage with the new unit prices to estimate the impact; there is no need to wait for next month’s bill.

7. Common Pitfalls

Treating the strongest model as the default, then manually inspecting a few good answers to prove it is “worth it.” This ignores long-tail requests and retries. The right approach is to report cost per successful request by feature.

Compressing only input and not capping output. After the system prompt is shortened, the model may fill the space with longer explanations.

Writing user privacy into a shared prefix for caching, or freezing expired policies to improve hit rate. Cost optimization cannot break permissions or timeliness.

Judging health by a single average token count. Applications should look at percentiles, retry rate, and escalation rate together. One outage retry can cost an order of magnitude more than a normal request.

Hardcoding a vendor’s current unit price and model name into an article or plan. A sample price only illustrates the algorithm; the production environment should follow the console, and unit prices should be configurable rather than hardcoded in business code.

8. Implementation Order

Prioritize by benefit and risk rather than changing every link at once. First, complete logging so input, output, and retries are distinguishable. Second, add output caps and prompt versions for each task type. Third, stabilize the system prompt prefix and observe cache hits. Fourth, move clearly light tasks out of the high-capability tier and escalate only after validation failure. Fifth, set feature-level budgets and alerts. Each step can be verified independently, avoiding a large change after which quality degradation cannot be attributed.

Rising LLM bills are usually an architecture problem, not a finance problem. Tokens determine scale, caching determines whether repeated work is paid for repeatedly, and routing determines whether high-price capability is used to solve low-price problems. Make these three things regression-testable engineering constraints, and the cost curve will remain controllable as the business grows, instead of jumping after a prompt change or an outage retry.