LLM Inference Costs Are No Longer About List Price: Why Cache Hit Rate Is the Real Pricing Lever

From Price Table to Bill: The Gap Between List Price and Real Cost

When a leading model provider priced its new K3 model at 100 yuan per million output tokens, 20 yuan per million input tokens, and only 1 yuan per million cache-hit tokens, the reaction was predictable: “A 5x price increase? How is anyone supposed to use this?”

But judging by headline prices alone misses the real intent behind this pricing change.

Compare this with the previous-generation K2.7 Code model: output was 27 yuan per million tokens, input with a cache miss was 6.5 yuan, and input with a cache hit was 1.1 yuan. On the surface, output prices nearly quadrupled and input prices roughly tripled. But a closer look at the price structure reveals a subtler shift: the ratio between cache-miss input price and cache-hit input price widened from roughly 6:1 in K2.7 to 10:1 in K3. Meanwhile, the ratio between output price and cache-hit price jumped from 25:1 to 50:1.

The same model can cost different users up to 3.6x more depending on how they use it. This is not a pricing gimmick. It is the result of a deliberate strategy that couples pricing with underlying architecture, using price signals to segment users while locking in high-value workloads through architectural advantages.

Why Cache Hit Rate Has Become a Hidden Discount

Caching is a straightforward concept in LLM inference. When multiple requests share the same prefix—such as system prompts, tool definitions, or conversation history—the system can reuse previously computed KV caches and skip redundant prefill computation. This reduces both latency and compute consumption.

The K3 model takes this mechanism further through a disaggregated inference architecture. Reports suggest that cache hit rates in coding scenarios can exceed 90%. For enterprise Agent applications with long contexts and high prefix reuse, the vast majority of input tokens flow through the cache-hit path.

A simple quantitative comparison makes the impact clear:

Enterprise Agent scenario (20:1 input-to-output ratio, 90% cache hit rate):

• Effective input price = 1 yuan × 0.9 + 20 yuan × 0.1 = 2.9 yuan per million tokens
• Output price = 100 yuan per million tokens
• Blended cost per million tokens = (20 × 2.9 + 1 × 100) / 21 ≈ 7.6 yuan

Individual developer with sporadic calls (5:1 input-to-output ratio, 20% cache hit rate):

• Effective input price = 1 yuan × 0.2 + 20 yuan × 0.8 = 16.2 yuan per million tokens
• Output price = 100 yuan per million tokens
• Blended cost per million tokens = (5 × 16.2 + 1 × 100) / 6 ≈ 30.3 yuan

Both scenarios use the same K3 model, yet the former pays roughly one quarter of the latter’s per-token cost. Cache hit rate is effectively an invisible discount card issued to high-value customers.

How Disaggregated Architecture Reshapes the Cost Structure

To understand the logic behind K3’s pricing, it helps to examine the technical foundation: a disaggregated inference architecture.

Traditional LLM inference combines prefill and decode on the same machine. Cache reuse efficiency is limited, and scaling is rigid. In a disaggregated architecture, prefill and decode are deployed separately. Prefill nodes process inputs and store KV states, while decode nodes focus on token generation. The two communicate over a high-speed network.

This design brings several direct benefits:

1. Higher cache hit rates: A unified cache management pool enables efficient reuse of long-running system prompts, tool calls, and multi-turn conversation history.
2. Lower costs for long-context workloads: The longer the context and the more stable the prefix, the greater the value of caching.
3. Better resource utilization: Separate deployment allows each node type to scale independently, reducing wasted compute.

In this light, K3’s seemingly steep price increase is not a simple markup. Rather, it transmits a lower-cost structure to different user segments through price differentiation. Users who can leverage caching effectively receive what amounts to wholesale pricing, while those who cannot pay retail rates.

Cost Scenarios Across Different Usage Patterns

To make this more concrete, consider three typical scenarios:

Scenario 1: High-Frequency Enterprise Agent (about 95% cache hit rate, 30:1 input-to-output ratio)

• Effective input price ≈ 1 × 0.95 + 20 × 0.05 = 1.95 yuan per million tokens
• Blended cost per million tokens = (30 × 1.95 + 1 × 100) / 31 ≈ 6.9 yuan
• Compared with the previous-generation K2.7 Code, the blended cost is higher, but the capability gains in code generation accuracy and long-context understanding outweigh the incremental cost.

Scenario 2: Mid-Frequency Tool-Calling Application (60% cache hit rate, 10:1 input-to-output ratio)

• Effective input price ≈ 1 × 0.6 + 20 × 0.4 = 8.6 yuan per million tokens
• Blended cost per million tokens = (10 × 8.6 + 1 × 100) / 11 ≈ 16.9 yuan
• This is the middle ground. Developers need to actively design prompt reuse strategies, such as fixed tool descriptions and templated conversation prefixes, to lower actual spending.

Scenario 3: Low-Frequency Exploratory Calls (below 10% cache hit rate, 3:1 input-to-output ratio)

• Effective input price ≈ 1 × 0.1 + 20 × 0.9 = 18.1 yuan per million tokens
• Blended cost per million tokens = (3 × 18.1 + 1 × 100) / 4 ≈ 38.6 yuan
• For these “pure retail” users, K3’s cost-effectiveness drops noticeably. Lighter-weight models or open-source alternatives may remain a better fit.

The Agent Era Has Rewritten the Economics of Inference

In the chat era, a single interaction consumed anywhere from dozens to a few thousand tokens, and users were largely price-insensitive. That assumption no longer holds. The rise of Agent and workflow-based applications has fundamentally changed token consumption patterns:

• Prompt inflation: A mature Agent application may include tens of thousands of tokens in system instructions, tool definitions, and enterprise knowledge base summaries. This portion can dominate the request.
• Multi-turn loops: Agents require multiple rounds of tool calls and reasoning loops, so input consumption far exceeds that of a single-turn chat.
• Batch processing: Enterprise workloads such as code review and document analysis routinely process millions of tokens.

In this new paradigm, cache hit rate directly determines whether an application is economically viable. K3’s pricing is effectively rewarding developers who understand cache optimization and pushing the industry toward more efficient calling patterns.

Developer Strategies: Designing for Cost Instead of Accepting It

Under the new pricing system, several practices are worth adopting:

1. Systematically separate prompt prefixes: Isolate fixed content such as system instructions, tool descriptions, and role definitions from dynamic content like user input and changing context to maximize cache reuse.
2. Use context caching interfaces actively: Manage conversation history deliberately, avoid carrying redundant information, and periodically compress or truncate earlier content in long sessions.
3. Match evaluation workloads to the right model: For sporadic or exploratory calls, consider lighter-weight models. For stable, high-frequency business logic, choose flagship models with stronger reasoning capabilities.
4. Build batch processing mechanisms: Aggregate delay-tolerant requests under fixed prefixes and submit them in batches to improve hit rates.
5. Monitor and analyze cost metrics: Track actual cache hit rates through logs and continuously refine prompt structure and calling logic.

The Inflection Point Has Arrived: Inference Costs Now Require Fine-Grained Operations

K3’s pricing adjustment sends a clear signal: LLM inference cost is no longer a flat rate. It is now a detailed pricing system deeply coupled with infrastructure capabilities and user behavior.

For enterprise users, this can be an advantage. With proper architecture adaptation and prompt engineering, they can access top-tier model capabilities at costs well below the published prices.

For individual developers and early-stage teams, the challenge is sharper: without the ability to exploit caching at scale, marginal costs remain high. This pressure is also pushing the industry to explore more efficient inference techniques, from quantization and speculative decoding to novel hardware architectures. The competitive frontier in AI engineering is shifting from “how strong is the model” to “how low is the cost per unit of effective intelligence.”

Going forward, evaluating a model provider will require looking beyond benchmark scores. The more important question is how much effective intelligence can be delivered per unit of cost. For developers and enterprises alike, the task is to find their own optimal position within this new cost coordinate system.