Understanding LLM Inference Pricing Tiers and Cost Optimization Practices
LLM inference pricing is not a single number but a system of tiers and their ratios. This article explains the three core billing tiers—input on a cache miss, input on a cache hit, and output—traces their price differences to the technical divide between the prefill and decode phases, and shows how the ratios between tiers reveal a provider's pricing strategy. It then offers practical engineering techniques for lowering cost: building cache-friendly prefixes, constraining output length, tracking cost per tier, and choosing services by tier ratios rather than output price alone.
1. Understand the Tier Structure Before Looking at Unit Prices
When a model provider changes its pricing, the easiest mistake developers make is to draw conclusions from a single number—how much output now costs. In reality, mainstream inference pricing is never a single price; it is a combination of several “tiers.” Without understanding the tier structure, you can neither estimate the cost of a real call nor optimize at the architecture level.
Typically, billing splits into three categories:
- Input on a cache miss: the full input goes through prefill and is billed at the highest input rate;
- Input on a cache hit: the prefix computation is reused, so you only pay for the new suffix, at a rate far below the full-input rate;
- Output: the tokens the model generates one by one, usually at the highest rate.
These three rates often differ by an order of magnitude. So understanding the ratio between them is more useful than memorizing the absolute prices.
2. The Technical Roots of Three-Tier Pricing: Prefill vs. Decode
To understand the price structure, first understand what happens under the hood during an inference request. A generation typically has two phases:
Prefill phase: the model processes all input tokens in parallel and computes their attention states (the KV Cache) in a single pass. This phase is compute-dense and highly parallelizable, so hardware utilization is relatively high.
Decode phase: the model generates output tokens one at a time, and each token requires reading the full KV Cache and running one forward pass. This phase is serial and memory-bound, so compute is hard to saturate.
This fundamental difference explains why output is inherently more expensive than input:
Prefill: [t1 t2 t3 t4 ... tn] ── parallel ──> KV Cache (compute-bound, cheap)Decode: KV Cache ─> o1 ─> o2 ─> o3 ... (serial memory access, expensive)
Output token generation is strictly serial and cannot be parallelized, so the marginal cost per token is higher; input tokens can be “digested in batches” during prefill, which lowers their unit cost. That is why nearly all mainstream providers price output at several times the input rate—not arbitrary pricing, but a reflection of real hardware utilization.
3. Cache Hits: Turning Repeated Computation into a Single Computation
During prefill, if multiple requests share the same prefix (the same system instructions, the same few-shot examples, the same long document), the computed result for that prefix (the KV Cache) can be reused. This is the core value of context caching / prompt caching.
A cache-hit input only needs to pay for the “newly added suffix”; the prefix is billed at a rate far below the full-input rate. This forms the second tier: cache-hit input.
It is worth noting that providers often actively adjust the ratios between tiers. As an illustrative example of tier ratios: if a provider widens the “cache-miss input : cache-hit input” ratio from about 6× to about 10×, it is essentially using a price signal to strongly encourage “cache-friendly” designs—amplifying the payoff of hits while raising the cost of wasting the cache.
4. What the Moving Tier Ratios Reveal About Pricing Strategy
Looking at the ratios across tiers reveals more about intent than absolute prices do. Using the same illustrative figures:
- Cache-miss input : cache-hit input ≈ shifting from 6× → 10×
- Output : cache-hit input ≈ shifting from 25× → 50×
- Cache-hit tier: relatively modest increase
- Output tier: the steepest increase
These changes show that while overall prices are being adjusted, the relative cost of the cache-hit tier is deliberately kept low, and most of the price pressure lands on the output tier. The logic is usually this—output is the genuinely “expensive” part of inference (the serial memory-access cost of decode), while the cache-hit tier is very cheap for the provider (it skips repeated prefill). Maintaining or even strengthening its value is essentially a way to steer users toward a “high cache hit, low output” call pattern.
The direct takeaway for developers: when the price structure changes, re-evaluate your application shape and call patterns rather than simply extrapolating old unit prices linearly into new costs.
5. Practical Cost Optimization Practices
Based on the mechanisms above, here are a few engineering recommendations.
1. Front-load stable content to actively create a cache-friendly prefix
Put unchanged or infrequently changing content (system role, tool/function definitions, long-term domain knowledge, few-shot examples) at the very front of the prompt, and the per-turn user input at the end. Caching usually matches on the “longest common prefix,” so the more stable the prefix, the higher the hit rate:
[Stable prefix — aim for a high hit rate]System role definitionTool / function descriptionsDomain knowledge / long documentsFew-shot examples[Variable suffix — once the cache hits, you only pay for this part]This turn's user input
2. Constrain output length precisely
Since output is the most expensive tier, limit output size at the design level: use explicit formatting requirements, maximum-length caps, and structured output (such as a JSON Schema) to avoid runaway costs from a “chatty” model.
3. Build per-tier cost observability
Instrument and track the three token counts separately, instead of looking at a single “price per million tokens” total:
def estimate_cost(usage, price):return (usage.input_tokens * price.input_miss # cache-miss input+ usage.cached_tokens * price.input_hit # cache-hit input+ usage.output_tokens * price.output # output)
Per-tier tracking shows you the real cost composition: often what looks like “expensive output” is actually inflated by a large volume of uncached repeated prefixes.
4. Choose services by ratios, not unit prices
The three-tier ratios can vary widely across generations and providers. When choosing, estimate your total cost based on your workload’s “input/output ratio” and “expected cache hit rate” rather than comparing output price alone—a service with a higher output price but excellent cache-hit-tier value may actually be cheaper in your scenario.
6. Conclusion
The price of LLM inference is never “a single number,” but a signal system built from a set of tiers and their ratios. Once you understand the technical difference between prefill and decode, and the economics of cache hits, you can translate a headline like “output price rose several times” into a judgment of its real impact on your business—and make smarter decisions in architecture design, prompt organization, and budget control.
