How Cache Hit Rates Are Reshaping Large Language Model Pricing and Cost Structure

When a new version of a large language model is released, pricing changes often attract more attention than capability improvements. When output prices rise sharply, the immediate reaction is usually to assume an across-the-board increase. But if you examine the pricing tiers closely and calculate weighted costs based on real usage patterns, the conclusion can be very different.

This article analyzes how LLM pricing is evolving through three dimensions: pricing gradients, cache mechanisms, and call patterns. It also provides a quantifiable method for evaluating actual costs.

1. Pricing gradients are more than just headline increases

You cannot understand pricing changes by looking at a single number. Current mainstream LLM pricing is usually divided into three core tiers:

• Uncached input price: tokens that appear for the first time in each request
• Cached input price: tokens that share a prefix with historical requests
• Output price: tokens generated by the model

Take the transition from one model generation to the next as an example. Suppose the previous generation charged 27 yuan per million output tokens, while the new generation charges 100 yuan per million output tokens. That is roughly a 3.7x increase on the output side. On the input side, prices changed from 6.5 yuan per million uncached tokens and 1.1 yuan per million cached tokens to 20 yuan and 2 yuan respectively, roughly a 3x increase.

What deserves more attention is the change in price ratios:

Ratio	Previous generation	New generation	Change
Uncached input / cached input	~6:1	~10:1	Wider
Output / cached input	~25:1	~50:1	Wider

The widening of these two ratios reveals a key signal: cache hits are becoming the central lever in the pricing system. Cached input prices have risen relatively moderately, while uncached input and output prices have increased more sharply. This means that whether you can use the cache efficiently will directly determine your actual cost.

2. Cache hit rate is an underappreciated cost variable

Why does cache hit rate matter so much? The answer starts with inference architecture.

A common optimization approach in the industry is to use a disaggregated inference architecture that decouples the prefill and decode stages, while caching frequently reused prefixes. In scenarios such as coding assistance, system prompts, project context, and code templates tend to be highly repetitive. Cache hit rates can exceed 90% in these cases.

This explains the underlying logic of the pricing adjustment: structurally discount high-reuse workloads while maintaining standard pricing for random-call workloads.

In other words, the price increase is not a uniform change. By widening the gap between cached and uncached pricing, providers are effectively applying differentiated pricing to different usage patterns.

3. Scenario-based cost estimation: the same model, different bills

To make the difference more concrete, let us calculate weighted costs for two typical scenarios.

Scenario A: enterprise-grade agent with high reuse

Assumptions:

• Input-to-output ratio: 20:1, typical for a long-context coding agent
• Cache hit rate: 90%

Calculation:

Actual input unit price = 2 × 0.9 + 20 × 0.1 = 3.8 yuan per million tokens
Output unit price = 100 yuan per million tokens
Blended cost per million tokens = (20 × 3.8 + 1 × 100) / 21 ≈ 8.4 yuan

Scenario B: individual developer with random calls

Assumptions:

• Input-to-output ratio: 5:1
• Cache hit rate: 20%

Calculation:

Actual input unit price = 2 × 0.2 + 20 × 0.8 = 16.4 yuan per million tokens
Output unit price = 100 yuan per million tokens
Blended cost per million tokens = (5 × 16.4 + 1 × 100) / 6 ≈ 30.3 yuan

Result comparison: Scenario A has a unit cost about 3.6 times lower than Scenario B. The same model can produce nearly a 4x difference in actual cost depending on how it is called.

4. Engineering practices for cost optimization

Based on the analysis above, developers and enterprises can optimize costs in the following areas.

1. Maximize prefix reuse

Put stable system prompts, role definitions, and project background at the beginning of the request to increase cache hits. Avoid inserting timestamps, random IDs, or other frequently changing content into the system prompt.

# Recommended approach: stable prefix + dynamic suffix
system_prompt = """
You are a senior programming assistant responsible for code review and optimization.
Project stack: Python 3.11 + FastAPI + PostgreSQL
Code standards: PEP8, complete type annotations
"""  # This part remains stable over time and is cache-friendly
user_query = f"Please review the following function: {code_snippet}"  # Dynamic part goes at the end

2. Manage the input-to-output ratio

In agent scenarios, you can control the input-to-output ratio by:

• Streamlining context and keeping only relevant code snippets
• Passing summaries instead of full historical text
• Setting reasonable maximum output lengths to avoid unnecessary long generations

3. Design call patterns

For high-frequency repetitive tasks, consider batch processing or asynchronous queues to group similar requests and improve cache reuse. For occasional random queries, evaluate whether the engineering investment in cache optimization is worthwhile.

5. The product logic behind the pricing strategy

From a product design perspective, this pricing structure sends clear signals:

• Predictable, structured call patterns are encouraged: enterprise applications typically use stable prompt templates and workflows that can fully leverage caching
• Resource-intensive workloads are priced higher: uncached input requires full computation, while output requires token-by-token decoding, both of which cost more
• Price levers guide users toward more efficient call patterns: when cache hit rate becomes a core cost variable, developers are incentivized to optimize prompt structure

This also aligns with the physical cost structure of inference services. The prefill stage can be processed in parallel, and cache reuse can greatly reduce computation. The decode stage, however, is serial, and every output token requires a full forward pass. Therefore, output prices being higher than input prices, and uncached prices being higher than cached prices, reflects engineering reality.

6. Summary and recommendations

To evaluate the real cost of using a large language model, you cannot rely on the headline price alone. Use the following formula instead:

Actual unit cost = f(input-to-output ratio, cache hit rate, pricing tier rates)

For developers and enterprises, the recommendations are:

1. Measure the cache hit rate in your own scenario: this is the key variable that determines cost
2. Optimize prompt structure: stable prefixes first, dynamic content later
3. Choose models based on call patterns: high-reuse scenarios can fully benefit from cache discounts, while random-call scenarios require broader evaluation
4. Build a cost monitoring system: track actual token consumption and cache hit rates, then optimize continuously

Pricing adjustments are ultimately guiding users toward more efficient resource usage. Understanding the logic behind them helps you make more rational technology choices and cost plans.