LLM Pricing Tiers: How Cache Hit Rate Is Reshaping the Cost Curve

The Real Story Behind Rising Prices

When a major LLM provider released its latest model version, the output price jumped from 27 yuan per million tokens to 100 yuan, while cache-miss input rose from 6.5 to 20 yuan and cache-hit input climbed modestly from 1.1 to 2 yuan. On the surface, this looks like an across-the-board price hike—roughly 4x for output and 3x for input. But comparing headline prices alone misses the more significant signal: the ratios between different billing tiers have shifted structurally.

Specifically:

• The price ratio between cache-miss and cache-hit input widened from roughly 6:1 to 10:1.
• The ratio between output and cache-hit input widened from about 25:1 to 50:1.

These ratio changes reveal the true intent: the cache-hit tier saw the most moderate increase, while output and cache-miss input were raised more aggressively. The pricing strategy is moving from a uniform rate model to a tiered incentive system—rewarding high-frequency reuse while penalizing low-reuse behavior.

Why Cache Hit Rate Has Become the Cost Divide

To understand this pricing structure, we need to look at the inference architecture. Modern high-performance inference deployments commonly use disaggregated architectures that decouple the prefill and decode phases, while reusing previously computed KV caches across requests. When multiple requests share the same prefix—such as identical system prompts, long-context documents, or code repository summaries—the computations for that prefix can be cached and served directly without recomputation.

In common industry implementations, this type of optimization can push cache hit rates above 90% in programming workloads. This means: for workloads with highly reused prefixes, the vast majority of input tokens are billed at the cache-hit price, not the cache-miss price.

The logic behind the pricing structure becomes clear:

• Cache-hit input rose the least (1.1 → 2 yuan), because this is the cheapest path.
• Cache-miss input rose moderately (6.5 → 20 yuan), because it requires full prefill computation.
• Output rose the most (27 → 100 yuan), because decoding is token-by-token generation with the highest compute density and latency sensitivity.

In other words, what increased in price is the low-reuse path, while the high-reuse path saw a comparatively controlled increase.

Real Cost Calculations for Two User Types

To validate this analysis, let’s examine two typical scenarios.

Scenario A: Enterprise Agent—High Reuse, High Input-to-Output Ratio

Assume a long-context coding agent scenario:

• Input-to-output ratio: 20:1 (20 input tokens consumed per output token generated)
• Cache hit rate: 90%
• Effective input price: 2 × 0.9 + 20 × 0.1 = 3.8 yuan per million tokens
• Output price: 100 yuan per million tokens

Combined cost per million tokens:

(20 × 3.8 + 1 × 100) / 21 ≈ 8.4 yuan per million tokens

Scenario B: Individual Developer—Low Reuse, Low Input-to-Output Ratio

Assume an occasional caller with different prompts each time:

• Input-to-output ratio: 5:1
• Cache hit rate: 20%
• Effective input price: 2 × 0.2 + 20 × 0.8 = 16.4 yuan per million tokens
• Output price: 100 yuan per million tokens

Combined cost per million tokens:

(5 × 16.4 + 1 × 100) / 6 ≈ 30.3 yuan per million tokens

Comparison

With the same model, the enterprise agent scenario achieves a unit cost roughly 1/3.6 of the individual developer’s cost. The gap does not come from the model itself—it comes from cache hit rate and input-to-output ratio, two controllable levers.

Dimension	Scenario A (Enterprise Agent)	Scenario B (Individual Developer)
Input-to-output ratio	20:1	5:1
Cache hit rate	90%	20%
Effective input price	3.8 yuan	16.4 yuan
Output price	100 yuan	100 yuan
Combined cost	≈8.4 yuan	≈30.3 yuan

How to Proactively Reduce Per-Token Costs

Understanding the pricing structure, developers can optimize costs in three directions.

1. Increase Prefix Reuse

Keep stable system prompts, tool definitions, domain knowledge, and coding conventions fixed at the beginning of the request prefix, avoiding re-organization with every call. The more stable the prefix, the higher the cache hit rate, and the closer the effective input price moves to the cache-hit tier.

2. Optimize the Input-to-Output Ratio

Long context does not mean dumping everything into each request. By using retrieval augmentation, summarization, hierarchical memory, and similar techniques, you can reduce redundant context without sacrificing quality. For the output side, setting appropriate max_tokens limits and using structured output constraints directly reduce decode overhead.

3. Route Tasks to Appropriate Model Tiers

Not every task requires the most powerful model. High-frequency, low-difficulty tasks can be routed to lighter models, reserving expensive models for genuinely complex reasoning. This tiered routing strategy is especially effective in agent systems.

Architecture Implications of Tiered Pricing

This pricing change sends a clear signal: the unit cost of LLM inference is no longer a constant—it depends on your usage pattern. For system architects, this means cost optimization is no longer just “pick a cheaper model.” It requires actively designing reuse paths at the architecture level.

Specifically:

• In agent frameworks, strictly separate invariant and variable parts, placing the invariant portion first to maximize cache hits.
• In multi-turn conversations, maintain a stable prefix structure for historical messages, avoiding frequent rewrites of earlier context.
• In batch workloads, structure requests so that items in the same batch share common prefixes, increasing overall hit rates.

These practices are general engineering best practices, independent of any specific vendor. But under a tiered pricing system, their cost benefits are significantly amplified.

Summary

Headline price increases are real, but the listed price is not the actual cost. When cache hit rates reach 90% and input-to-output ratios reach 20:1, the combined cost drops to the range of 8 yuan per million tokens. When hit rates fall to 20% and the ratio is 5:1, costs approach 30 yuan. The same model, two different usage patterns, a cost gap of more than 3x.

Therefore, when faced with a pricing table like “output 100 yuan, input 20 yuan, cache-hit 1 yuan,” the truly important question is not the absolute number—it is which tier you land on. For enterprise customers running long-standing system prompts with highly reused prefixes, the effective price is a discount. For individual developers who change prompts every time, the cost is close to the full list price. The essence of tiered pricing is using price signals to guide users toward higher-reuse, more efficient inference patterns—which happens to be exactly the direction engineering should pursue anyway.