LLM Inference Costs Are Diverging by Tier: A Pricing Structure and Cache Hit Rate Analysis

One: Price Changes Are Not Uniform Across All Tiers

In a recent model version update, output-side pricing was revised from 27 CNY per million tokens in the previous generation to 100 CNY per million tokens, an increase of nearly 4x. Input-side miss pricing moved from 6.5 CNY to 20 CNY per million tokens, while cache-hit input pricing went from 1.1 CNY to 2 CNY per million tokens—both roughly 3x increases.

At first glance, this looks like a broad price increase. But comparing the three tiers side by side reveals structural shifts that matter more than average growth:

• The ratio of miss-input to hit-input widened from about 6x to 10x.
• The ratio of output to cache-hit input widened from about 25x to 50x.
• The cache-hit tier saw the smallest absolute increase, while the output tier saw the largest.

This suggests the pricing strategy is not simply “raising prices across the board,” but rather concentrating cost pressure on the paths that consume the most compute resources, while using a lower cache-hit price to steer callers toward better context reuse.

Once this is understood, comparing list prices in isolation loses meaning. The real cost drivers are three variables: cache hit rate, input-to-output ratio, and how friendly the workload is to context reuse.

Two: Why Cache Hit Rate Becomes the Critical Variable

In LLM inference, processing input tokens costs far less than generating output tokens—but input volumes are typically much larger than output volumes. To avoid redundant computation, most production systems use prefix caching: if an input shares a prefix with a previous request, the intermediate results can be reused instead of recalculated from scratch.

Cache-hit pricing is usually far lower than miss pricing. Using the updated prices as an example: miss-input at 20 CNY per million tokens versus hit-input at 2 CNY per million tokens—a 10x difference. This means that for the same input volume, moving the hit rate from 0% to 90% can reduce input-side costs by over 80%.

Cache hit rate therefore becomes the first-order variable in actual inference cost. And the achievable hit rate depends almost entirely on the shape of the workload.

Three: Cost Differences Across Two Typical Scenarios

To illustrate how real-world costs can diverge by several multiples under the same price sheet, here are two hypothetical scenarios. Prices follow the updated structure: miss-input at 20 CNY/M tokens, hit-input at 2 CNY/M tokens, output at 100 CNY/M tokens.

Scenario One: Enterprise-Grade Long-Context Agent

This is the classic case of a continuously running system with highly repetitive context. Consider a coding agent: each request carries large project files, conversation history, and tool outputs. These contents repeat heavily across requests, so the cache hit rate can reach 90% or higher.

At the same time, this workload has far more input than output—an input-to-output ratio of roughly 20:1, meaning 20 input tokens are processed for every 1 output token generated.

Effective input unit price:

Effective input price = Hit price × Hit rate + Miss price × Miss rate
                     = 2 × 0.9 + 20 × 0.1
                     = 3.8 CNY per million tokens

Output price remains at 100 CNY per million tokens. With a 20:1 input-to-output ratio, each 20 input tokens correspond to 1 output token, so the blended cost per million tokens is:

Blended cost = (20 × 3.8 + 1 × 100) / 21
            = (76 + 100) / 21
            ≈ 8.4 CNY per million tokens

This figure is far below what the list price alone would suggest. The reason is that a high hit rate compresses most of the input cost down to near the cache-hit price level.

Scenario Two: Low-Frequency Personal Use with Highly Variable Prompts

Now consider the opposite case: an individual developer calling the model occasionally, with different prompt content each time, no fixed prefix to reuse, and a very low cache hit rate—say 20%. This type of workload also tends to have a smaller input-to-output ratio, assumed here as 5:1.

Effective input price:

Effective input price = 2 × 0.2 + 20 × 0.8
                     = 0.4 + 16
                     = 16.4 CNY per million tokens

Blended cost per million tokens:

Blended cost = (5 × 16.4 + 1 × 100) / 6
            = (82 + 100) / 6
            ≈ 30.3 CNY per million tokens

Under the same price sheet, these two scenarios show a real cost gap of roughly 3.6x. The entire difference comes from the combined effect of cache hit rate and input-to-output ratio.

Four: The Technical Logic Behind the Pricing Structure

In this adjustment, the ratio of miss-input to hit-input widened from 6x to 10x, and output-to-hit widened from 25x to 50x. Why widen these gaps?

The answer lies in the inference architecture. Current mainstream LLM serving systems have adopted disaggregated inference architectures, where the inference process is split into separate phases—prefix processing and token generation can run in different resource pools. This design naturally supports prefix reuse: identical context is computed once, and subsequent requests reuse the result directly.

Public technical materials indicate that in coding scenarios, this architecture can achieve cache hit rates above 90%. If real hit rates are indeed at that level, the logic behind the pricing structure becomes clear:

• For high-hit-rate users, most input costs only 2 CNY per million tokens, so the price increase is heavily diluted.
• For low-hit-rate users, input costs approach 20 CNY per million tokens, and the price signal pushes them to improve cache reuse.
• The higher output price reflects the compute cost and resource exclusivity of the generation path, while also supporting overall gross margins.

In short, this is a “pay for actual consumption” approach: the more compute a request consumes, the more it costs; the more reuse, the lower the cost.

Five: A General Method for Cost Estimation

The calculation above can be generalized into a simple formula for estimating costs in any scenario.

Let:

• P_in_miss be the miss-input price per million tokens.
• P_in_hit be the cache-hit input price per million tokens.
• P_out be the output price per million tokens.
• H be the cache hit rate (0 to 1).
• R be the input-to-output ratio (input tokens / output tokens).

Effective input price:

P_in_eff = P_in_hit × H + P_in_miss × (1 - H)

Blended cost per million tokens:

P_total = (R × P_in_eff + 1 × P_out) / (R + 1)

Substituting any model’s list prices along with your workload parameters gives a realistic estimate of actual inference cost.

Six: Practical Implications for Developers

For teams that call models frequently with long contexts, the following points are worth attention.

First, prioritize cache hit rate optimization. Keep stable, unchanging context in the request prefix and place variable content in the suffix. This significantly improves prefix reuse. In system design, avoid inserting random timestamps, dynamic IDs, or other prefix-breaking elements at the start of every request.

Second, distinguish evaluation scenarios from production scenarios. Testing cost with one-off, scattered requests will overestimate real production spending. Evaluate under a load that closely resembles actual business patterns.

Third, treat cache hit rate as a selection criterion. When comparing models or deployment options, list price is only a baseline reference. Convert it using your own cache hit rate and input-to-output ratio. For businesses with high hit rates, a lower cache-hit price matters most; for workloads with naturally low hit rates, miss-input price and output price have a larger impact.

Fourth, pay attention to the nonlinear relationship between cost and latency. Higher hit rates not only reduce cost but also cut latency from repeated computation. When designing request strategies, cache optimization often improves both cost and response time—a benefit that pure model performance tuning cannot easily achieve.

Seven: What This Price Adjustment Signals

The change in pricing structure reflects a broader shift in inference services from “per-token billing” toward “per-resource-consumption billing.” Raising prices for output tokens and miss-input tokens, while adjusting the cache-hit tier more gently, is essentially using price signals to steer callers toward optimizing context reuse.

For inference providers, this structure helps improve overall cluster utilization: when more requests hit the cache path, the same compute capacity can serve more calls. For developers and enterprise users, understanding and adapting to this pricing structure is not about passively accepting a price increase—it is about actively optimizing system costs.

Once input-output ratio, cache hit rate, and request frequency are incorporated into the cost evaluation framework, decision quality around model selection and system design improves significantly. The price tag is just the starting point. The real cost lives in every request and every piece of context.