Why Inference Pricing Is Moving Toward Cache-Aware, Tiered Pricing

1. The Trend: Inference Pricing Is Not a Single Number

In large language model inference services, pricing is rarely expressed as a single “price per million tokens.” It is usually split into at least three tiers:

• Input price: the cost of tokens that the model reads from user input.
• Output price: the cost of tokens that the model generates.
• Cache hit price: the cost of reading prefix tokens that are already cached and do not need to be recomputed.

A recent pricing change from a major model provider is worth examining. Compared with the previous generation, the output price rose from about twenty-something yuan per million tokens to one hundred yuan. The non-cached input price rose from a little over six yuan to twenty yuan. The cache hit price rose from around one yuan to two yuan.

At first glance, this looks like an across-the-board increase. But comparing the three listed prices alone is not enough to determine how real user costs changed. Inference cost is not simply the listed price multiplied by call volume. It also depends on how much of each request hits the cache and on the actual input-output ratio in a given application.

2. Breaking Down the Ratios Between Pricing Tiers

To compare changes across tiers more effectively, the cache hit price can be used as a baseline.

In the previous generation, a common ratio structure was roughly:

• Non-cached input price ≈ 6× the cache hit price.
• Output price ≈ 25× the cache hit price.

In the new generation, the gaps have widened:

• Non-cached input price ≈ 10× the cache hit price.
• Output price ≈ 50× the cache hit price.

This suggests that providers are strengthening the incentive for cache-friendly usage patterns. The more a request reuses cached prefixes, the lower its unit cost. Conversely, requests that frequently change context or depend heavily on newly generated output bear a higher marginal price.

This shift is not arbitrary. It is closely related to how inference systems compute attention internally.

3. Why Cache Hit Rate Matters More Than Before

LLM inference has a key computational characteristic: for the same input prefix, the attention computation is highly similar across requests. In traditional serving approaches, the prefix portion is recomputed for every request, which creates large amounts of redundant computation.

To address this, the industry has gradually adopted prefix caching. The core idea is:

1. Split input text into prefix segments.
2. Retain the key-value cache for prefixes that have already been computed.
3. When a new request arrives, first match it against the cached prefix.
4. Reuse cached results for the matched portion and compute only the remaining part.

This can significantly reduce time to first token and total computational load. In long-context scenarios, system prompts, instructions, conversation history, and codebase context are often highly repetitive, so cache hit rates can reach high levels.

According to publicly available technical information, the newer model generation can achieve cache hit rates above 90% in some programming scenarios by using a disaggregated inference architecture. This means most input tokens no longer need to be recomputed, which substantially lowers the unit serving cost.

Therefore, looking only at the output price increase from twenty-something yuan to one hundred yuan may overstate the actual cost impact. If the cache hit rate is high enough, real input-side spending can remain relatively low.

4. Estimating Real Costs with Scenario Models

To understand the impact of pricing changes, it is necessary to model specific usage patterns. The key variables are:

• Input-output ratio.
• Cache hit rate.
• Unit prices for each pricing tier.

Two typical scenarios are analyzed below.

Scenario 1: Enterprise Long-Context Coding Agent

Assume an enterprise coding agent with the following characteristics:

• Each request carries a large amount of system prompts, repository context, and change history.
• The input-output ratio is about 20:1.
• The cache hit rate reaches 90%.

The effective input price can be calculated as:

effective input price = cache hit price × hit rate + non-cached input price × (1 - hit rate)

Using the current prices:

effective input price = 2 × 0.9 + 20 × 0.1
                      = 1.8 + 2.0
                      = 3.8 yuan per million tokens

The output price remains 100 yuan per million tokens.

The blended cost per million tokens can be expressed as:

blended cost = (input tokens × effective input price + output tokens × output price) / total tokens

With a 20:1 input-output ratio:

blended cost = (20 × 3.8 + 1 × 100) / 21
             = (76 + 100) / 21
             ≈ 8.38 yuan per million tokens

Although the output unit price is high, output accounts for only a small share of total tokens, and the input-side cache hit rate is high. As a result, the final unit cost remains relatively low.

Scenario 2: Low-Frequency Individual Developer with Highly Variable Prompts

Now consider the opposite case: an individual developer who calls the model occasionally, with tasks that vary significantly and low prompt reuse.

Assume:

• The input-output ratio is 5:1.
• The cache hit rate is only 20%.

The effective input price is:

effective input price = 2 × 0.2 + 20 × 0.8
                      = 0.4 + 16.0
                      = 16.4 yuan per million tokens

The blended cost is:

blended cost = (5 × 16.4 + 1 × 100) / 6
             = (82 + 100) / 6
             ≈ 30.33 yuan per million tokens

In these two scenarios, the first has a unit cost roughly one-third to one-fourth that of the second.

This shows that the model may appear more expensive, but the real driver of cost is how it is called. Enterprise applications that reuse long, stable prefixes can achieve an actual cost far below the nominal price increase. Scattered calls with low cache reuse will experience a cost increase closer to, or even higher than, the headline price change.

5. How Cache Hits Affect Architecture and Engineering Practice

The pricing structure described above directly influences engineering decisions. To reduce inference cost, more teams are focusing on the following practices.

1. Keep System Prompts and Fixed Prefixes Stable

System prompts should remain as stable as possible. Avoid regenerating or recombining different content for every request. The longer and more stable the fixed prefix, the easier it is for the caching system to recognize and reuse it.

For example, in multi-turn tasks, all shared context can be placed in the prefix area:

[system role description]
[task rules]
[tool definitions]
[environment information]
[current user input]

As long as the prefix remains consistent, subsequent requests can reuse the cache.

2. Limit Context Rewrites

In long conversations or complex tasks, context length can grow quickly. If history is reordered or rewritten each time, prefix reuse is disrupted. A better approach is to append new information rather than rewrite existing content so that cache continuity is preserved.

3. Split Requests When Appropriate

If a task naturally contains both stable and variable parts, consider handling them separately. Stable background information can serve as a fixed prefix, while variable content follows as later input. This avoids invalidating an entire cached segment because one small part changed.

4. Pay Attention to Call Patterns and Capacity Planning

When evaluating model cost, do not simply multiply the output price by estimated output volume. Build a model that incorporates the input-output ratio and cache hit rate. It is better to measure the real hit rate during testing and include the results in capacity planning and budget estimation.

6. Industry Trends Behind the Pricing Shift

This pricing change reflects several important trends.

Trend 1: Inference Pricing Is Moving from Linear to Tiered

Early LLM inference pricing was often based on a single input-output token price. Over time, input and output were priced separately, and the structure has since become more granular with cached and non-cached input tiers. This tiered approach reflects real differences in inference system costs and gives users more room for cost optimization.

Trend 2: Caching Is Becoming a Core Competitive Capability

Cache hit rate directly affects both user spending and provider resource efficiency. By improving cache hit rates, providers can serve more requests without a proportional increase in compute. Users benefit from lower unit costs. As a result, cache architecture and hit rate optimization are becoming key competitive factors in inference services.

Trend 3: Cost Evaluation Must Be Scenario-Based

Different business scenarios have very different price sensitivity. Enterprise long-session applications may care more about stability and total cost, while low-frequency developers may focus on per-call price. Comparing output prices alone no longer reflects the real burden, making scenario-based cost modeling essential.

7. Practical Recommendations

For developers and enterprise users facing changing inference pricing structures, the following steps can help:

1. Separate cost evaluation dimensions: track input volume, output volume, and cache hit volume separately instead of looking only at total spend.
2. Optimize prefix design: keep system prompts, tool descriptions, and environment context stable.
3. Measure real cache hit rates: test caching behavior in your own workload rather than relying only on official examples.
4. Build a cost model: combine input-output ratio, hit rate, and tiered unit prices into an explainable cost estimate.
5. Track long-term trends: treat cache friendliness as part of model selection and architecture design, not just benchmark scores.

8. Summary

Changes in LLM inference pricing cannot be reduced to simply “up” or “down.” Output prices in the newer generation have risen significantly, while cache hit prices have increased more moderately. At the same time, the gaps between pricing tiers have widened. This means inference cost is shifting from pure volume-based billing toward a model that depends heavily on context reuse efficiency.

For enterprise scenarios with high cache hit rates and large input-output ratios, actual unit costs may remain low. For scenarios with low cache reuse and scattered calls, cost increases will be more noticeable. Understanding this logic helps developers make better decisions about technology selection, architecture design, and cost optimization.

As cache architectures, disaggregated inference, and long-context capabilities continue to evolve, pricing structures may become even more granular. For developers, the key factor is not any single price tier. It is whether their system design allows their usage pattern to benefit as much as possible from caching.