How to Reduce LLM Inference Costs: Tiered Routing and Model Selection in Practice
LLM inference costs can spiral quickly if you only look at input pricing. This article explains why unit inference costs tend to decline over time, highlights common billing mistakes, and presents a practical three-tier routing strategy that matches task complexity to model capability. It includes a cost calculation formula, sample routing code, and additional techniques—caching, context compression, output limits, and batching—to keep your AI bill under control.
1. Will Inference Costs Keep Falling?
Most observers expect unit inference costs to trend downward over the long term. Four main forces drive this: more efficient inference chips and server designs, maturing optimization techniques such as KV cache reuse, quantization, batch scheduling, and speculative decoding, model architectures that require less compute, and intensifying price competition among vendors.
There is no single answer to how much costs will fall. Estimates vary widely depending on the model, the time period, and the measurement methodology, so this article does not quote a fixed reduction factor. When judging costs, rely on the vendor’s public pricing page and your own actual bills.
2. Common Cost Miscalculations
Looking only at unit prices makes it easy to get your bill wrong. Three pitfalls are common:
- Looking only at input pricing and ignoring output tokens. Many models charge significantly more per output token than per input token, and long responses can make your bill grow fast.
- Ignoring long contexts. If every request carries large chunks of conversation history or documents, input token counts will far exceed expectations.
- Ignoring retries and failure fallbacks. Retries on timeouts and re-calls after invalid output formatting mean the same task may be billed multiple times.
3. Older Models: Low Unit Prices for Simple, High-Volume Tasks
Previous-generation models have been out for a while, their capability boundaries are well understood, and vendor competition is intense, so unit prices are often far lower than at launch. For high-volume tasks such as text classification, sentiment analysis, structured information extraction, and first-pass content moderation, these models are often good enough.
A low unit price does not mean negligible cost. You still need to account for call volume, retry counts, cache miss rates, and escalation calls when quality falls short.
4. Newer Models: Higher Unit Prices for Critical Tasks
The prices of the latest flagship models tend to drop more slowly. Possible reasons: they sit at the capability frontier with few alternatives, high-quality demand is concentrated in a small number of scenarios, and training and inference require heavy resource investment. Check the vendor’s pricing page for current prices, which change over time.
Flagship models suit complex reasoning, agent planning, high-precision generation, and fine-grained understanding of long documents. These tasks carry high costs when they fail, so the extra spend typically buys better results.
A word of caution: in long-context, high-output-token scenarios, flagship model costs can far exceed expectations. Run A/B tests before launch and set daily or per-task budget caps.
5. Tiered Selection: Three Task Tiers, Three Model Classes
Divide business tasks into three tiers by complexity, each with its own model class, quality threshold, and monitoring metrics.
| Task Tier | Typical Tasks | Candidate Model Class | Quality Threshold | Cost Monitoring Metrics | Escalation Rule |
|---|---|---|---|---|---|
| Simple | Classification, extraction, template-based generation | Mature small models or older models | Meets baseline accuracy on your evaluation set | Cost per call, retry rate | Escalate when metrics fall below threshold repeatedly |
| Medium | Summarization, multi-turn dialogue, routine coding assistance | Mid-tier models | Pass rate from automated scoring or manual sampling | Average token count, escalation rate | Escalate when automated scoring drops below threshold |
| Hard | Complex reasoning, agent planning, high-precision generation | Flagship models | Task success rate, factual error rate | Total cost per task, daily budget consumption | No downgrade; alert when over budget |
This table is a starting point; calibrate the thresholds against your own business evaluation set. If you don’t have one, start by manually labeling a few dozen real samples.
6. Routing Implementation: Calculate Costs First
The core of routing logic is cost calculation. Below is illustrative code—replace the unit prices with actual values from the vendor’s pricing page, and implement call_model and eval_score yourself.
from dataclasses import dataclass# Unit prices in USD per million tokens. Replace with actual prices from the vendor's pricing page.PRICE = {"legacy-small": {"in": 0.0, "out": 0.0},"mid-tier": {"in": 0.0, "out": 0.0},"flagship": {"in": 0.0, "out": 0.0},}QUALITY_THRESHOLD = 0.8 # Calibrated against your evaluation setdef call_cost(model: str, in_tokens: int, out_tokens: int, attempts: int = 1) -> float:"""Cost per task = (input tokens x input price + output tokens x output price) / 1e6 x number of calls"""p = PRICE[model]per_call = (in_tokens * p["in"] + out_tokens * p["out"]) / 1_000_000return per_call * attemptsdef route(task, eval_score):if task.complexity == "low":return call_model("legacy-small", task.text)if task.complexity == "mid":draft = call_model("mid-tier", task.text)if eval_score(draft) >= QUALITY_THRESHOLD:return draft# Quality below threshold, escalate to the flagship modelreturn call_model("flagship", task.text)return call_model("flagship", task.text)
Do the math on the “start low, escalate later” approach. Let the per-call cost of the mid-tier model be c_m, the flagship cost be c_f, and the escalation rate be p. The expected cost per task is c_m + p × c_f. When the escalation rate is high, this approaches c_m + c_f—more expensive than calling the flagship model directly. This strategy only pays off when the escalation rate is low and the mid-tier cost is significantly lower than the flagship cost. Measure your escalation rate before launch, then decide whether to adopt this approach.
Two additional caveats:
- Scoring itself costs money. If you use a large model as the scorer, the scoring calls are billed too.
- Thresholds need ongoing calibration. A threshold set too high triggers frequent escalation; one set too low lets quality problems slip through.
7. Other Cost-Reduction Techniques
Beyond tiered routing, the following practices typically reduce your bill directly:
- Caching. Cache results for repeated or highly similar requests to avoid paying twice.
- Compress context. Keep only conversation history relevant to the current question, and periodically summarize older content.
- Limit output length. Specify output format and length in the prompt, and set a maximum output token count.
- Batching. For tasks that don’t need real-time responses, use batch processing APIs if your vendor offers them.
- Set retry limits. Define a maximum retry count per task type, and log token usage per call so you can investigate abnormal consumption.
8. Continuous Re-Evaluation
Older models get cheaper quickly, and new models keep launching. Re-run your evaluation set every quarter and ask: can the mid-tier model now pass the quality threshold for tasks that previously required the flagship? If so, move that task to a lower tier, and confirm with one to two weeks of production data that quality hasn’t degraded.
9. FAQ
How can I reduce LLM inference costs? Start with task tiering, handing simple tasks to mature, low-cost models. Then cut token consumption through caching, context compression, output length limits, and batching.
How should I route between models? Split tasks into three tiers by complexity and set quality thresholds for each using an evaluation set. Escalate only when thresholds aren’t met, and track the escalation rate to confirm total costs actually fall.
What are the common mistakes? Three main ones: looking only at input pricing and ignoring output tokens; overlooking the extra calls caused by long contexts and retries; and running “start low, escalate later” with a high escalation rate, making total cost exceed direct flagship usage.
10. Summary
Inference costs will likely keep falling over the long term, but the rate varies by model and task. Older models are already cheap and fit simple, high-volume workloads; newer models cost more and suit critical tasks where quality matters and errors are expensive.
In practice, start by tiering your tasks with an evaluation set, then assign a model class and quality threshold to each tier. Verify your routing logic against the cost formula, including retries and escalations. After launch, monitor both cost and quality continuously, and re-evaluate your model choices quarterly. Follow this process, and the savings from falling inference prices will actually show up in your bill.
