LLM Token Cost Optimization: A Practical Guide to Tiered Routing
This article explains why token billing can silently drive up AI application costs, breaking it down into three components — input tokens, output tokens, and cached tokens — and showing how to optimize their type and distribution rather than simply using fewer tokens. It then lays out a four-tier framework for classifying tasks by difficulty and matching them with models of different capability and cost levels, so that strong models handle planning and final review while cheaper models absorb high-volume, low-difficulty execution work. Readers will find a pseudocode example of a routing layer, a four-step method for estimating cost from real usage data, a checklist of engineering techniques including prompt caching, history trimming, output length limits, batching, and cost monitoring, plus a step-by-step set of recommendations for implementing the approach.
1. Why Token Cost Becomes a Hidden Threshold for AI Applications
Once large language models (LLMs) move into the engineering phase, many teams hit the same wall: the feature works, but the bill does not. The reason is that LLM billing is not measured per call — it is measured per token. Tokens are the smallest unit of text that a model processes, and both input and output are accumulated item by item.
An easily overlooked fact: a seemingly simple conversation can consume far more tokens than expected because of long context, multi-turn history, and repeated system prompts. When an application moves from a demo to production, and from one user to many, token cost grows linearly and sometimes faster. Understanding the cost structure and designing a sound invocation strategy have therefore become core parts of AI engineering capability.
This article does not discuss any specific product. It describes a general methodology for cost optimization that applies whether you consume model services from a public cloud or run your own inference cluster. Taking a general-purpose model service offered by a cloud provider as an example, users can typically choose models at different capability levels on demand, and build cost observability and governance on top of infrastructure such as monitoring alerts and logging services.
2. Breaking Down Token Cost into Three Dimensions
You cannot optimize a cost structure without first seeing it clearly. LLM token billing is usually decomposed into three categories.
2.1 Input Cost (Input Tokens)
Input covers all text sent to the model by the user and the system: system prompts, conversation history, retrieved knowledge snippets (RAG), and the current question itself. The longer the input, the higher the cost. In many real-world scenarios, verbose context and steadily accumulating multi-turn history are the main sources of runaway input cost.
2.2 Output Cost (Output Tokens)
Output is the text the model generates. A general rule applies here: the unit price of output is usually noticeably higher than that of input. This is no accident — generation is decoded token by token, which is computationally denser and puts more load on server-side resources. Consequently, making a model produce longer text is typically more expensive than making it read longer text.
2.3 Cache Hit Cost (Cache Tokens)
When parts of a request repeat — for example a fixed system prompt or the prefix of a long document — and those parts hit the cache, those tokens can be billed at a price far below the regular input rate, sometimes only a small fraction of it. Prompt caching is therefore a key lever for controlling cost in long-context scenarios.
Once these three differences are clear, the first conclusion follows: optimizing cost is not simply about “using fewer tokens” but about optimizing the type and distribution of tokens. More cache hits, compressed ineffective input, and controlled redundant output usually pay off more than cutting call volume alone.
3. Model Tiering: A Rational Trade-off Between Capability and Cost
Models at different capability levels correlate positively across three axes: reasoning ability, generation quality, and unit price — the stronger the model, the higher the price. This creates a core engineering judgment: not every task needs the most capable model.
Using the strongest model at every step inflates cost rapidly, and most of that spend is wasted on low-value tasks. Conversely, running everything on the cheapest model causes quality to collapse on complex tasks. The sound approach is to build a model tiering system so that models at different capability levels each do what they are suited for.
It helps to think of a two-dimensional map: task difficulty (complexity) on one axis and model capability (cost) on the other. Placing tasks on that map and matching each quadrant with an appropriate model is the underlying logic of tiered routing.
4. A Framework for Task Tiering and Model Matching
Drawing on common AI application and Agent engineering practice, tasks can be roughly divided into four layers, each matched with a model at a different capability level:
- Planning and final review layer: requires strong reasoning, global oversight, and quality control. These calls are infrequent but high in value, and suit the most capable, highest-priced model.
- Complex reasoning layer: multi-step logic, code comprehension, difficult problem solving — needs a fairly capable model.
- Main execution layer: handles the vast majority of everyday work such as coding, writing, and structured output. It sits in the sweet spot between cost and quality.
- Batch simple-task layer: formatting, classification, summarization, field extraction — high volume, low difficulty, best served by the most economical model.
One sentence captures this strategy: the expensive model thinks; the cheap model does. Concentrate your scarce strong-model capacity on the few steps that genuinely require thinking, and let low-cost models absorb the bulk of repetitive, low-difficulty work.
The following is a simplified sketch of a routing layer (pseudocode) showing how to dispatch tasks to models of different capability tiers based on task type:
def route_and_call(task: Task, context: Context) -> Result:# 1. Classify the task: tag it with a capability tier based on complexitytier = classify_task(task) # plan / reason / execute / batch# 2. Pick a model tier according to the level (capability level placeholder, not a specific product)model = MODEL_REGISTRY[tier]# 3. Reuse cacheable prefixes where possible to reduce input costrequest = build_request(system_prompt=SHARED_PROMPT, # fixed prefix, helps cache hitshistory=trim_history(context), # trim multi-turn history to control input sizecontent=task.content,)# 4. Cap output length to avoid unnecessary output tokensresponse = model.generate(request, max_tokens=OUTPUT_LIMITS[tier])# 5. Record the tokens consumed by this call for later cost accountingmetrics.record(tier, response.usage)return response
The critical part is the accuracy of classify_task: the more accurate the classification, the more visible the savings. Start with rules (task type, input length, whether tool calls are needed) as a first pass, then gradually introduce a lightweight classifier to refine it.
5. How to Estimate Costs Scientifically
Before a formal selection, teams usually need to forecast cost. A reusable approach is to derive a per-unit consumption rate from actual usage records:
- Collect real data: record call volume, token consumption, and billed amounts over a period to form a dataset.
- Set reasonable assumptions: for example, assume an input/output ratio (say 80% input, 20% output), and initially ignore caching for a conservative estimate.
- Fit a consumption rate: regress token consumption (dependent variable) on call volume (independent variable) linearly to obtain the token consumption rate per call. A goodness-of-fit (such as R²) closer to 1 indicates a more reliable model.
- Extrapolate the budget: multiply the fitted consumption rate by the expected call volume to obtain a budget range.
One point deserves emphasis: these figures are estimates for a specific scenario and will shift with prompt length, task distribution, and caching strategy — they should not be treated as fixed standards. For actual rollout, always rely on your own monitoring data, and leave room for the fluctuations caused by caching and rule adjustments.
6. A Checklist of Engineering Techniques for Cost Optimization
Beyond tiered routing, there is a set of directly applicable optimization techniques:
- Prompt caching: place stable content such as system prompts and long documents at the front of the request to maximize the cache hit rate and significantly cut the cost of repeated input.
- History trimming: do not blindly concatenate the full conversation history; summarize it or apply a sliding window as needed to keep input tokens in check.
- Output constraints: set a reasonable
max_tokenslimit for each task to avoid verbose, rambling output from the model. - Batching and async processing: merge and process simple tasks asynchronously to cut both cost and improve throughput.
- Cost observability: integrate logging and monitoring with alerts, instrument token usage and cost for every call, set budget thresholds, and configure alerts to prevent bills from spiraling out of control.
7. Practical Recommendations
Overall, the key to reducing the cost of LLM applications is not to be stingy but to differentiate. Build a clear tiered routing system so that strong models focus on high-value thinking while low-cost models take on scaled execution work, then layer on engineering measures such as caching, trimming, rate limiting, and monitoring. That yields a sustainable balance between quality and cost.
Three steps are a good starting point: first, build cost observability (know how much you are spending); second, tier your tasks (know which steps need a strong model); and third, continuously optimize routing and caching strategies (know how to save more as you scale). Once this mechanism is running, an AI application’s cost curve shifts from “linear growth with scale” to “continued decline through optimization.”
