Tiered LLM Pricing: How Cache Hit Rates Are Reshaping Cost Structures

1. What the Headline Price Increases Actually Reveal

A recent pricing adjustment to a next-generation LLM API has sparked extensive discussion in developer communities. On the surface, output pricing jumped from the previous generation’s mid-to-low tier to roughly RMB 100 per million tokens, while input pricing for cache misses rose to RMB 20 per million tokens. The overall increases look dramatic. But comparing headline prices alone easily leads to a simplistic “everything got more expensive” conclusion, while missing the deeper structural shift in how pricing is organized.

Let’s first map the key price tier changes:

• Output side: From roughly RMB 27 per million tokens in the previous generation to the RMB 100 level—a substantial absolute increase;
• Input side (cache miss): From roughly RMB 6.5 to RMB 20—about a 3x increase;
• Input side (cache hit): From roughly RMB 1.1 to RMB 2—a comparatively modest increase.

Notably, the price ratios between tiers have been structurally adjusted. In the previous generation, the cache-miss input price was about 6x the cache-hit price; in the new generation, that ratio widens to 10x. The ratio between output price and cache-hit input price expands from about 25x to 50x.

This reshaping of ratios indicates that the pricing strategy is no longer a simple “across-the-board adjustment” but rather an attempt to steer specific usage patterns.

2. Cache Hit Rate: The Overlooked Cost Lever

To understand the intent behind these ratio changes, it helps to first understand the caching mechanism in LLM inference.

In a typical chat or Agent scenario, each request usually contains a system prompt, historical conversation context, and the current user input. The system prompt and part of the historical context are highly repetitive across requests. The core idea of caching is to retain the KV Cache for this repeated content in the inference infrastructure, so subsequent requests that hit the cache avoid recomputation, substantially reducing compute overhead.

This means the cache hit rate directly determines the actual billed cost on the input side. And across different usage scenarios, cache hit rates can differ by orders of magnitude.

Among common industry approaches, some teams use disaggregated inference architectures to achieve cache hit rates above 90% in coding scenarios. These architectures decouple the prefill and decode stages, making cache reuse more efficient.

3. Real Cost Modeling Across Two Typical Scenarios

To illustrate how cache hit rate affects overall cost more intuitively, let’s model two typical scenarios.

Scenario A: Enterprise Agent (High Cache Hit Rate)

Assumptions:

• Input-to-output ratio of about 20:1 (common for long-context coding Agents)
• Cache hit rate of about 90%

Effective input unit price:

effective input price = cache_hit_price × hit_rate + cache_miss_price × (1 - hit_rate)
                      = 2 × 0.9 + 20 × 0.1
                      = 1.8 + 2.0
                      = 3.8 RMB per million tokens

Blended cost per million tokens:

blended cost = (input_tokens × effective_input_price + output_tokens × output_price) / total_tokens
             = (20 × 3.8 + 1 × 100) / 21
             ≈ 8.4 RMB per million tokens

Scenario B: Individual Developer (Low Cache Hit Rate)

Assumptions:

• Input-to-output ratio of about 5:1 (short conversations, different prompt each time)
• Cache hit rate of about 20%

Effective input unit price:

effective input price = 2 × 0.2 + 20 × 0.8
                      = 0.4 + 16.0
                      = 16.4 RMB per million tokens

Blended cost per million tokens:

blended cost = (5 × 16.4 + 1 × 100) / 6
             ≈ 30.3 RMB per million tokens

Cost Divergence Comparison

Metric	Scenario A (Enterprise Agent)	Scenario B (Individual Developer)
Cache hit rate	90%	20%
Input-to-output ratio	20:1	5:1
Effective input price	3.8 RMB per million tokens	16.4 RMB per million tokens
Blended cost	~8.4 RMB per million tokens	~30.3 RMB per million tokens
Cost multiple	Baseline	~3.6x

With the same model, unit costs differ by about 3.6x between the two scenarios. This means the real impact of headline price increases varies dramatically across user segments.

4. The Technical Rationale Behind Tiered Pricing

From an inference infrastructure perspective, this tiering is not arbitrary.

In high-cache-hit scenarios, compute utilization across the inference cluster is higher. Redundant computation in the prefill stage is largely eliminated, and GPU time per token drops significantly. Service providers can serve these requests at lower marginal cost, leaving more room to pass savings on.

In low-cache-hit scenarios, every request requires full prefill computation, and KV Cache is frequently created and destroyed, placing greater pressure on memory bandwidth and scheduling systems. From a resource-consumption standpoint, charging more has a reasonable cost basis.

This effectively creates differentiated pricing based on resource consumption: the higher the reuse, the lower the unit price; the lower the reuse, the higher the unit price.

5. Cost Optimization Guidance for Developers

Once the pricing structure is clear, developers can optimize their usage patterns accordingly to reduce overall costs.

5.1 Engineering Practices to Improve Cache Hit Rates

Stabilize the system prompt: Design the system prompt as fixed content and avoid dynamic concatenation on every request. For Agent scenarios, place fixed elements such as tool definitions and role settings at the front and keep them unchanged.

Design for prefix reuse: In multi-turn conversations, keep prefix content as consistent as possible. For example, do not modify the format or order of historical messages midway, as this invalidates all subsequent cache entries.

Sensible context management: Not all historical conversation needs to be retained in full. Use summarization, key-information extraction, and similar techniques to control context length while keeping the prefix stable.

# Recommended: fixed prefix, dynamic suffix
messages = [
    {"role": "system", "content": FIXED_SYSTEM_PROMPT},  # never changes
    {"role": "user", "content": historical_summary},      # stable summary
    {"role": "user", "content": current_query}            # dynamic
]
# Not recommended: rebuilding the system prompt every time
messages = [
    {"role": "system", "content": f"Current time {now}, user {name}..."},  # changes every time
    ...
]

5.2 Optimizing the Input-to-Output Ratio

Output tokens are priced far higher than input tokens, so controlling output length is the most direct way to reduce costs. Specific measures include:

• Explicitly constrain output format and length in prompts;
• Use structured output (such as JSON Schema) to reduce redundant text;
• For classification, extraction, and similar tasks, prefer short output templates.

5.3 Matching Scenarios to Models

Not every task requires the most capable model. For simple classification, formatting, and information extraction tasks, lightweight models are sufficient, and costs can differ by an order of magnitude. Consider building a task complexity assessment mechanism that dynamically routes to models of different tiers.

6. The Right Way to Evaluate Costs

Faced with any pricing adjustment, developers should build a weighted cost model based on their own usage rather than simply comparing headline prices. Core evaluation steps:

1. Measure your actual input-to-output ratio: Analyze historical request logs for the average input/output token ratio;
2. Measure your cache hit rate: Pay attention to cache-related fields returned by the API and calculate the actual hit proportion;
3. Compute a weighted unit price: Plug your numbers into the formulas above to derive the blended cost per million tokens;
4. Compare against business value: Weigh cost together with task quality, development efficiency gains, and other factors.

7. Summary

LLM API pricing is moving from “single headline prices” toward “structured tiers.” The widening ratio between output price and cache-hit price is essentially rewarding high-reuse, high-cache-hit usage patterns while charging a higher resource premium for low-reuse scenarios.

For enterprise Agent developers, if architecture design can keep cache hit rates at a high level, the real cost increase may be far smaller than the headline increase. For individual developers whose prompts differ every time, the focus should shift to output length control and model selection optimization.

Understanding the pricing structure is the key to navigating pricing changes. We hope the modeling framework and optimization suggestions in this article help you find the cost strategy best suited to your business amid a new round of price adjustments.