How to Interpret LLM Inference Price Increases: A Cache Hit Rate and Unit Cost Analysis

1. Surface Prices: The Increases Are Indeed Significant

Recent price adjustments for LLM inference services have sparked considerable discussion. Looking at list prices alone, the output side rose from roughly 20+ yuan per million tokens to 100 yuan, cache-miss input from just over 6 yuan to 20 yuan, and cache-hit input from around 1 yuan to 2 yuan. Every line item increased roughly three to five times, with output seeing the largest jump.

If you compare only the “output unit price,” it is easy to read this as a major price hike. But LLM inference cost is not determined by a single price. Actual spending depends on input tokens, output tokens, cache hit rates, context length, and calling patterns. In particular, once a provider treats cache hit rate as a key optimization lever, the gap between list price and actual cost can widen considerably.

The more worthwhile question, then, is not “how much did one line item go up” but “what happened to the real blended cost per million tokens in actual usage scenarios.”

2. Breaking Prices into Three Categories: Output, Cache-Miss Input, and Cache-Hit Input

LLM inference billing typically includes at least three categories:

1. Output tokens
   These correspond to model-generated content and usually consume the most compute. The generation phase requires step-by-step decoding and cannot reuse context at scale the way the input side can.

2. Cache-miss input tokens
   These are newly appearing prefix contents in each request that cannot be reused. The system must re-encode these tokens, so the unit price is noticeably higher than for cache-hit input.

3. Cache-hit input tokens
   When a request prefix matches or heavily overlaps with previous requests, the system can directly reuse previously cached intermediate representations, significantly reducing compute cost. Cache-hit prices are typically far lower than cache-miss prices.

In this pricing change, output rose the most and cache-hit input rose the least. At the same time, the price ratio between cache-miss and cache-hit input widened further. This means the pricing system is more strongly encouraging users to raise cache hit rates rather than simply reducing input or output.

3. Why Cache Hit Rate Becomes the Key Variable

In scenarios such as long-context coding agents, document analysis, and multi-turn conversations, input token counts often far exceed output token counts. An enterprise coding agent, for example, may carry a large amount of project context per request, with an input-to-output ratio of twenty to one or even higher.

If these inputs hit the cache heavily each time, actual input cost moves toward the cache-hit price. Conversely, if request prefixes differ substantially each time, most input is billed at the cache-miss price, and actual cost rises noticeably.

Consider an input-to-output ratio of twenty to one with a 90% cache hit rate:

• Effective input unit price: 2 × 0.9 + 20 × 0.1 = 3.8 yuan per million tokens
• Output unit price: 100 yuan per million tokens
• Blended cost per million tokens: (20 × 3.8 + 1 × 100) / 21 ≈ 8.4 yuan

Now consider another scenario: an input-to-output ratio of five to one with a 20% cache hit rate:

• Effective input unit price: 2 × 0.2 + 20 × 0.8 = 16.4 yuan per million tokens
• Output unit price remains: 100 yuan per million tokens
• Blended cost per million tokens: (5 × 16.4 + 1 × 100) / 6 ≈ 30.3 yuan

In these two scenarios, the latter’s effective unit cost is about 3.6 times that of the former. This shows that cache hit rate can influence real cost even more than list prices do.

4. Actual Costs in High-Cache-Hit Scenarios May Not Be High

When a system can raise cache hit rates above 90% through disaggregated inference architectures, prefix caching, scheduling optimizations, and similar techniques, the actual cost of long-running enterprise scenarios is significantly diluted.

This is because long-running systems often exhibit highly repetitive context structures. For example:

• The same system prompt is used across multiple calls;
• The same project files are reused repeatedly in coding tasks;
• Historical messages are retained as prefixes in multi-turn conversations;
• Public knowledge bases or document fragments are shared across batch requests.

In these scenarios, as long as the platform can identify and reuse prefixes, the vast majority of input tokens fall into the cache-hit tier. Even if output prices rise, output tokens account for a small share, so their impact on total cost is relatively limited.

Therefore, the blended cost in high-cache-hit scenarios is not equivalent to the surface-level “output price quadrupled” impression.

5. Low-Frequency, Weak-Cache Scenarios Face Greater Cost Pressure

By contrast, scenarios where request prefixes differ greatly and cache reuse is difficult will bear greater cost pressure. Examples include:

• Individual developers whose prompt changes noticeably each time;
• Tasks without a fixed system prompt or project context;
• Short-term trial users;
• Low-frequency calls with discontinuous context;
• Simple single-turn Q&A with only basic input.

These users typically have low effective cache hit rates, and much of their input is billed at the cache-miss price. Meanwhile, if the output share is high, output price increases will directly push costs up.

The new pricing system can be understood as rewarding reusable, long-stable, highly engineered scenarios while being less friendly to occasional, short-chain, fragmented calls.

6. Look at Unit Task Cost, Not Just Output Price

To judge whether a price adjustment is reasonable, you should not look only at the output token unit price but return to unit task cost. How much it costs to complete a code generation, a long document summary, or a multi-turn task depends on multiple variables:

• Input token count;
• Output token count;
• Cache hit rate;
• Whether batch processing is used;
• Whether repeated prefixes can be reduced on the client or middle layer;
• Whether context can be shortened without hurting task quality.

For developers, the key to cost optimization is not necessarily choosing a cheaper model but engineering improvements that raise cache hit rates, reduce redundant input, and control wasteful output.

7. How to Improve Cache Hit Rates in Engineering Practice

Major cloud providers and inference platforms typically offer prefix caching, session caching, or context reuse capabilities. Developers can improve hit rates in several ways:

1. Put Stable Content First

Caching is usually based on prefix matching. If fixed content such as model role descriptions, tool definitions, and project structure is placed at the very front of the request, with variable content after it, it is easier to hit a common prefix.

2. Reduce Meaningless Prefix Jitter

Avoid placing timestamps, random IDs, request sequence numbers, and other volatile information at the beginning of each request. Once such content sits in the prefix position, it directly breaks cache hits.

3. Use Fixed System Prompt Templates

In multi-turn or batch tasks, using the same system prompt, the same tool descriptions, and the same knowledge base structure improves reuse probability.

4. Reuse Sessions or Threads Where Possible

For agents, coding assistants, document Q&A, and similar scenarios, try to complete related tasks within the same session rather than frequently creating requests with no conversation history.

5. Control Context Length

Even with high cache hit rates, input tokens still affect total cost. Removing irrelevant files, compressing historical messages, and keeping only necessary context can reduce overall spending.

8. How to View the Deepening of Price Tiers

This price adjustment carries another clear signal: the price ratio between cache-miss and cache-hit input has widened further, and the ratio between output and cache-hit prices has also widened. This structural change can be understood as a form of cost redistribution.

Platforms are no longer simply billing by “how much compute is consumed” but using price signals to guide users toward optimizing their calling patterns. For enterprise users who can reliably reuse context, the actual cost increase may be lower than the list price increase; for scenarios with low cache hit rates, cost pressure may feel higher.

This change is not simply “more expensive” or “cheaper” but makes actual prices depend more on usage patterns and engineering capability.

9. A Rough Cost Estimation Example

The following pseudocode can be used to evaluate blended costs across different scenarios:

def estimate_cost(input_tokens, output_tokens, cache_hit_rate,
                  price_miss=20, price_hit=2, price_output=100):
    input_price = (
        input_tokens * cache_hit_rate * price_hit +
        input_tokens * (1 - cache_hit_rate) * price_miss
    ) / 1_000_000
    output_price = output_tokens * price_output / 1_000_000
    return input_price + output_price
scenario_a = estimate_cost(20_000, 1_000, 0.9)
scenario_b = estimate_cost(5_000, 1_000, 0.2)
print("Estimated cost in high-cache scenario:", scenario_a)
print("Estimated cost in low-cache scenario:", scenario_b)

This example is intended only to illustrate the calculation method and does not represent the actual billing rules of any platform. Real usage should also account for free quotas, batch discounts, resource packs, concurrency limits, and other factors.

10. Conclusion: Understanding Prices Requires Returning to Usage Structure

After LLM inference price adjustments, judging expensive or cheap by output unit price alone is insufficient. Actual cost is determined jointly by cache hit rate, input-to-output ratio, prefix stability, context length, and other factors.

For enterprise scenarios with high cache hit rates, long contexts, and high input shares, effective unit cost may remain acceptable. For scenarios with low cache hit rates, fragmentation, and high output shares, cost pressure will be greater.

Therefore, rather than focusing only on list prices, developers should turn their attention back to usage structure. Improving cache hit rates, optimizing context design, and reducing wasted input and output may be more effective than simply hunting for a lower-priced model. This also shows that LLM inference cost is moving from “static list pricing” toward “dynamic structural cost”—understanding this matters more than memorizing any particular price number.