Why Cache Hit Rate Is Becoming the Key Variable in LLM Inference Cost Control

1. Looking at List Prices First: Which Side Increased the Most

If you only compare base list prices, the output-side price of new-generation reasoning models is close to four times that of the previous generation. Input-side cache miss and cache hit prices also generally increased by about three times. On the surface, this looks like an across-the-board price increase.

But the list prices themselves reveal a new cost structure. What deserves the most attention is not the increase itself, but the changing ratios between pricing tiers.

In the previous generation of models, the cache miss input price was about six times the cache hit price. The new generation has pushed that ratio to ten times. The output price compared to the cache hit price has also expanded from about twenty-five times to fifty times.

This means two things:

1. Cache miss input has become more expensive. If every request carries a completely new context, costs will rise significantly.
2. Reusable context has become cheaper. As long as the cache is hit, input costs can be pushed to an extremely low level.

Therefore, the real question is no longer “Is the model expensive?” but “Can the task enable context reuse?”

2. Why Cache Hit Rate Has Become So Important

From a technical perspective, the cost of long-context inference is not evenly distributed. The input side requires encoding the prefix, while the output side continuously consumes memory and bandwidth during decoding. As context length increases, KV cache storage pressure and the bandwidth consumption of attention computation both rise significantly.

If the same prefix is encoded repeatedly, computing resources are consumed redundantly. For this reason, inference services usually introduce a prefix caching mechanism: the same prefix is encoded only once, and subsequent requests directly reuse the existing KV state.

A high cache hit rate means that a large number of input tokens do not need to be recomputed. A low cache hit rate means every request must bear the full cost of forward encoding.

As a result, pricing strategies have begun to favor cache-friendly tasks. Cache hit prices have increased moderately, while cache miss prices have risen sharply. In essence, the pricing signal is guiding developers to optimize context reuse.

3. The Same Model, but Very Different Actual Costs

To understand this difference more intuitively, we can estimate costs for two typical scenarios.

Scenario 1: Enterprise Agent with High Reuse

Assume an enterprise coding agent task with an input-to-output ratio of about 20:1 and a cache hit rate of about 90%. This type of task typically includes a large amount of stable system prompts, tool definitions, project context, and rule descriptions, making the prefix highly reusable.

The actual input-side unit price can be calculated as a weighted average of cache hits and cache misses, which comes to about 3.8 yuan per million tokens. The output side is calculated at 100 yuan per million tokens. The combined cost is about 8.4 yuan per million tokens.

Scenario 2: Individual Developer with Low Reuse

Assume an individual developer uses different prompts each time, with a cache hit rate of only 20% and an input-to-output ratio of about 5:1. In this case, the actual input-side unit price is about 16.4 yuan per million tokens, while the output side remains at 100 yuan per million tokens. The combined cost is about 30.3 yuan per million tokens.

For the same model, the unit cost in the high-reuse scenario is less than one-third of that in the low-reuse scenario.

This shows that the new pricing system is not simply raising prices for everyone. Instead, it shifts costs from tasks that can reuse context to tasks that cannot.

4. The Cost Logic Has Changed in the Agent Era

Conversational applications typically involve only dozens to thousands of tokens per interaction, and users are not highly price-sensitive. Even if unit prices rise, the absolute cost may increase by only a fraction of a cent.

Agent tasks are different. A single complex task may involve thousands of tool calls, hundreds of thousands of tokens, or even multiple instances running in parallel for days. These tasks demand far more inference resources than traditional conversations, and any unit price fluctuation is sharply amplified.

If the previous generation of pricing were maintained, high-concurrency, long-context agent tasks could easily cause inference services to operate at a loss. The reason is that memory usage, KV cache storage, and decoding-stage bandwidth pressure all grow linearly or even superlinearly with context length.

Therefore, raising cache miss input prices and widening the gap between cache hits and misses is effectively establishing a new cost recovery mechanism:

• Tasks with stable prefixes amortize computing costs through caching.
• Tasks with unstable prefixes pay for additional encoding computation.
• Higher output-side prices reflect the resource pressure of long decoding tasks.

5. How Developers Should Respond to the New Pricing Structure

After understanding this mechanism, developers designing agent systems should no longer treat “cost per call” as the only metric. Instead, they should focus on three variables:

1. Cache hit rate
2. Input-to-output token ratio
3. Prefix stability

Common ways to improve cache hit rate include:

• Place stable content such as system prompts, tool descriptions, and environment context at the very beginning of the request.
• Avoid dynamically inserting changing content into the prefix area for every request.
• Standardize the unchanged parts of a session as much as possible and reduce meaningless formatting differences.
• For high-concurrency tasks, keep request prefixes as consistent as possible.

For example, when constructing a request:

[Fixed system prompt]
[Fixed tool definitions]
[Fixed project description]
[Dynamic user input]
[Dynamic execution trace]

This structure maximizes prefix reuse and reduces the probability of cache misses.

At the same time, rising output-side costs require developers to control unnecessarily long outputs. For example, reducing unnecessary explanatory text, returning structured results, and setting a reasonable maximum output length can all help lower output costs.

6. Conclusion

The pricing adjustments for new-generation reasoning models essentially reflect the different resource consumption structures of long-context, high-concurrency agent tasks.

Looking only at list prices leads to the conclusion that prices have risen across the board. But if cache hit rate, input-to-output ratio, and context reuse are taken into account, high-reuse tasks may actually cost far less than low-reuse tasks.

For developers, the new pricing system brings not only cost pressure but also a clear architectural signal: put stable content first, put changing content later, and reuse context as much as possible.

In the future, evaluating the cost-effectiveness of a model service cannot rely on comparing a single pricing tier. It requires building a complete cost model based on the characteristics of your own tasks. Those who better understand and leverage caching mechanisms will gain greater cost advantages as agent applications scale.