LLM Token Cost Optimization: A Practical Guide to Tiered Model Routing
LLM token spend rarely grows linearly once models sit in your core request path, and a simple average price will not get you a reliable budget. This article breaks down how token billing is actually structured — asymmetric input and output pricing, wide gaps between model tiers, and the impact of prefix cache hits. It then shows how to reverse-engineer your platform's effective cost rates from your own call logs using a least-squares regression, and explains how to evaluate the fit. From there, it lays out a tiered routing strategy that sends planning, complex reasoning, routine execution, and high-volume simple tasks to the model tier whose cost matches, with estimation functions and budget guardrails to keep spend predictable. You will come away with a measurement-first workflow, a routing pattern you can implement, and a clear view of why routing can matter before you send a single request.
1. Why LLM Costs Are Harder to Control Than They Look
As a project moves from prototype to production, call volume often doesn’t scale linearly. It steps up as business modules are layered on. Once you embed LLM capabilities in the core request path, token consumption becomes a real and recurring cash expense. A trap many teams fall into: the feature works, but the month-end bill far exceeds expectations.
The root cause is that token cost is not as simple as “one unit price times the number of calls.” It is shaped by three dimensions:
- Input and output pricing are asymmetric. On most platforms, the output rate is significantly higher than the input rate. It is often several times higher, and sometimes an order of magnitude higher.
- The gap between tiers is enormous. The unit price of a flagship reasoning model and a lightweight model can differ by roughly two orders of magnitude.
- Cache hits can reshape the cost structure. Inputs that hit a prefix cache can drop to just a few percent of the normal input rate.
If you estimate with a single “average unit price,” your budget will almost certainly be off. The controllable approach is to first build a way to quantify actual consumption, then design your routing strategy around it.
2. How Token Billing Actually Works
Before optimizing, you need a clear cost model. The token bill for a single call typically consists of the following:
total cost = uncached input x input rate+ cached input x cache rate (cache rate is far below input rate)+ output tokens x output rate
Several key conclusions are worth emphasizing:
1) Why output is expensive. Output tokens are generated one at a time through autoregression, so the compute and scheduling cost is inherently higher than reading the input in a single pass. That is why “making the model say less” is often a more effective lever than “feeding it less context.” Controlling output length is a high-value optimization.
2) Caching is the biggest lever on the input side. When multiple requests share the same prefix (a system prompt, a long document, fixed context), prefix caching can push the rate for that repeated input down to a very low level. In some solutions, cached input costs only about 2% of what regular input costs. Putting stable content first and volatile content last is therefore an engineering habit that saves money directly.
3) Tier gaps are what make tiering possible. When the output rates of the most and least expensive tiers differ by roughly two orders of magnitude, “using the same model for everything” is clearly wasteful. Most tasks do not need the capability of the strongest model.
3. Reverse-Engineering Your Real Cost Rates from Your Own Logs
Billing rates are rarely handed to you directly, but you can solve for them from your own call logs. This is the most practical methodology in this article.
The idea is to treat each call as a sample. Record its input tokens, output tokens, and cached tokens, along with the credits deducted from your balance for that call. With enough samples, credit consumption is approximately linear in the token counts. Fit the data with least-squares regression, and the regression coefficients correspond to the effective cost rate of each component.
Illustrative Python implementation:
import numpy as np# Each row: [uncached input, cached input, output] -> credits deductedX = np.array([[12000, 8000, 500], # input, cache hits, output[3000, 15000, 120],[20000, 0, 900],# ... more real call samples], dtype=float)y = np.array([320, 180, 760], dtype=float) # corresponding credits# Least-squares solution; coefficients are each tier's "credits per 1K tokens"rates, residuals, *_ = np.linalg.lstsq(X, y, rcond=None)input_rate, cache_rate, output_rate = ratesprint("input:", input_rate, "cache:", cache_rate, "output:", output_rate)
To judge whether the fit is trustworthy, look at two metrics:
- Coefficient of determination (R²): The closer it is to 1, the better the linear model explains actual consumption.
- Residuals: The closer they are to 0, the more closely predicted spend matches what was actually deducted.
When you get R²≈1 and residuals≈0, you have captured the platform’s actual billing structure rather than noise. At that point, you can plug the coefficients into any call scenario to estimate the credits that call will cost offline, before you send the request, without waiting for billing to settle. This is the dividing line between regretting the bill after the fact and staying in control beforehand.
4. Tiered Routing: Let Strong Models Think and Light Models Do the Work
Once you have the real cost rates for each tier, the core strategy comes down to one sentence: classify tasks by difficulty and route each to the model tier whose cost matches.
A practical tiering:
| Task tier | Typical work | Recommended model |
|---|---|---|
| Planning and final review | Breaking down requirements, designing solutions, sign-off | Flagship high-capability model |
| Complex reasoning | Multi-step derivations, tricky code, logic verification | Reasoning-enhanced model |
| Main execution | Routine generation, code completion, text processing | Standard execution model |
| High-volume simple | Classification, extraction, formatting, summarization | Lightweight model |
The key insight here is that the marginal return on capability is nonlinear. The hardest 20% of tasks may need the strongest model. For the other 80% of day-to-day work, a lightweight model is sufficient and roughly two orders of magnitude cheaper. Spend your expensive reasoning budget on the few steps that genuinely need to “think,” and hand the bulk of the “doing” to cheaper tiers. Your overall cost structure will improve immediately.
You can implement the routing logic with a task-feature scoring function:
def route(model_rates, task):"""Pick the cost-matched model tier based on task complexity."""# Roughly assess the required capability level from task featuresif task["needs_planning"] or task["needs_final_review"]:tier = "flagship" # planning / final review -> flagshipelif task["steps"] > 5 or task["needs_deep_reasoning"]:tier = "reasoning" # complex reasoning -> reasoning-enhancedelif task["batch"] and task["simple"]:tier = "lite" # high-volume simple -> lightweightelse:tier = "standard" # main executionreturn {"model": tier, "est_rate": model_rates[tier]}
5. Cost Estimation and Budget Guardrails
After tiering, turn cost into something you can monitor and get alerts on. Three things are recommended:
1) Build an estimation function. Turn the fitted coefficients into a utility function that returns the estimated credits for a call before it is sent:
def estimate_cost(rates, in_tok, cache_tok, out_tok):"""Estimate credits per call offline from fitted rates."""miss = max(in_tok - cache_tok, 0)return miss * rates["input"] + cache_tok * rates["cache"] + out_tok * rates["output"]
2) Set budget guardrails. Set credit quotas for each business window (e.g., every 5 hours, each week). When the estimated cost approaches a threshold, automatically downgrade to a cheaper tier or trigger rate limiting.
3) Re-fit periodically. Billing rules or platform policies can change, so re-fit with the latest logs on a regular basis. If R² drops noticeably and residuals grow, the cost structure has changed and the coefficients need recalibrating.
A concrete comparison: under the assumption of “80% input + 20% output, no caching,” the same token budget spent on the most expensive tier might produce only a few hundred thousand tokens. Spent on a lightweight tier, it can produce tens of millions. The same credits, roughly two orders of magnitude difference in output. That is the value of tiering.
6. A Few Practical Reminders
- Measure before you guess. Base every rate on a fit of your own real logs. Don’t go by gut feeling. R²≈1 from your fit is the signal you can trust.
- Don’t save in the wrong place. Cutting costs on planning and final review can lead to rework, which ends up costing more. Save on the repetitive labor of bulk execution.
- Caching has to be earned through coding habits. Put stable prefixes first and volatile content last to keep your cache hit rate as high as possible. This is a zero-cost structural optimization.
- The figures here are methodological examples. The price tiers and multiplier relationships above illustrate the analysis method and the tiering approach. They are not the official prices of any real platform. Rely on your chosen service’s published billing documentation and your own fitted results.
In one sentence: measure first, then tier, then add guardrails. Fit your real cost rates so that high-capability models focus on thinking while lightweight models handle execution, and use budget guardrails to cap the downside. Together, these steps can bring LLM call costs down to a reasonable level without sacrificing quality.
