LLM Inference Pricing Divergence: How Cache Hit Rates Reshape Cost Models and Architecture Decisions

1. Why Price Adjustments Demand a Fresh Cost Look

A general-purpose coding model recently revised its inference pricing: output-side rates rose from ¥27 to ¥100 per million tokens (nearly 4×), uncached-input rates from ¥6.5 to ¥20, and cached-input rates from ¥1.1 to ¥2. Three tiers, three different multipliers.

The gap between uncached and cached input prices widened from roughly 6× to 10×; the gap between output and cached-input prices doubled from about 25× to 50×. Cached input saw the mildest absolute increase, while output and uncached input jumped far more sharply. This is not random noise—it mirrors a shift in the underlying inference architecture.

To make sense of the divergence, you need to move beyond static per-token comparisons and factor in cache hit rates, input-output ratios, and request patterns. This article offers a reusable framework for estimating real inference cost across different usage scenarios.

2. The Three Standard Billing Tiers

Most LLM inference services break charges into three categories.

Uncached input (prefix not reused). When a request arrives, its prompt prefix cannot reuse prior computations, so the system runs a full forward pass. This tier is usually the most expensive because it involves the heaviest computation, plus distributed communication, GPU-memory reads/writes, and scheduling overhead.

Cached input (prefix reused via prefix caching). If the prompt prefix overlaps with a previous request and the platform supports prefix caching, the shared portion is skipped and only the new tokens are computed. Unit prices here are typically far lower, though the exact discount depends on cache design, matching granularity, and eviction policy.

Output generation. Each new token is produced by auto-regressive decoding—one full forward pass per token, memory-intensive, and poorly parallelizable compared with prefill. Scheduling is harder because output length is unpredictable, and output tokens are almost never cacheable. In long-generation or extended-thinking modes, output token counts can multiply further.

3. Cache Hit Rate: The Decisive Variable

Cache hit rate is the single most important factor in real cost. A technical blog reports that, in a coding-agent scenario using a disaggregated inference architecture, hit rates can exceed 90%. From a neutral engineering standpoint, that figure is plausible.

Disaggregated inference splits prefill and decode onto separate compute pools and adds a dedicated caching layer. Prefill nodes handle heavy prompt computation; decode nodes run auto-regressive generation. The cache layer persists key-value (KV) tensors across requests and sessions so subsequent calls can reuse already-computed prefixes.

Coding agents naturally have high hit rates. Between tool calls, the system prompt, repository context, environment info, and conversation summaries change little. Even across different users, requests that hit the same codebase or share tool definitions have substantial prefix overlap. A unified cache serving many similar prefixes cuts redundant compute dramatically.

At 90% hit rate, with uncached input at ¥20/M tokens and cached input at ¥2/M tokens, the effective input price is:

2 × 0.90 + 20 × 0.10 = ¥3.8 per million tokens

That is far below the ¥20 uncached rate and well below a naïve average of ¥11. Most input tokens land in the cheap tier.

At 20% hit rate, the math flips:

2 × 0.20 + 20 × 0.80 = ¥16.4 per million tokens

More than four times higher. Identical nominal pricing, vastly different real cost—driven entirely by workload pattern.

4. Cost Estimation Across Scenarios

Accurate cost estimation requires a composite model. A common formula:

Effective cost per million tokens = (effective input price × input share) + (output price × output share)

Enterprise-grade long-context coding agent

These agents repeatedly read repository files, conversation history, and tool results. Input-output ratio can reach 20:1 (twenty input tokens per output token), with a 90% cache hit rate.

• Effective input price: ≈ ¥3.8 / M tokens
• Output price: ¥100 / M tokens
• Composite cost: (20 × 3.8 + 1 × 100) ÷ 21 ≈ ¥8.4 per million tokens

This is dramatically below the ¥20 you would get by assuming every input token is uncached.

Low-frequency individual developer

Prompts vary widely each time, context reuse is minimal, and the cache hit rate may be only 20%. Assume a 5:1 input-output ratio.

• Effective input price: ≈ ¥16.4 / M tokens
• Output price: ¥100 / M tokens
• Composite cost: (5 × 16.4 + 1 × 100) ÷ 6 ≈ ¥30.3 per million tokens

Nearly four times the enterprise scenario’s unit cost.

The contrast reveals the pricing strategy’s intent: by widening the gap between cached and uncached rates, providers incentivize workloads that favor cache reuse. High-frequency, long-context, highly structured agent traffic benefits; low-frequency, short-context, independent-request traffic pays more.

5. Why Cached Input Is So Much Cheaper: A Compute Perspective

The price gap has a real computational basis. An uncached input requires full prefill—massive matrix multiplications, attention calculations, and intermediate-result storage. A cached-input request reads existing KV tensors from cache and only computes the new portion. The difference in FLOPs, memory bandwidth, and network traffic can be an order of magnitude or more.

Disaggregated architectures amplify this. A shared cache layer lets prefill and decode nodes access the same KV data, avoiding per-node cache fragmentation that would lower hit rates. Schedulers can use prefix hashing to locate cache entries quickly, reducing wasted data movement.

But these gains depend on sufficient prefix overlap. If most requests share little, the cache layer adds complexity and latency without payoff. The cached-input discount is therefore both a reflection of real compute savings and a signal that steers traffic patterns.

6. Practical Cost-Optimization Levers

Under this model, developers can reduce inference spend in several ways.

Boost cache hit rates.

• Place stable content (system instructions, tool descriptions, repo documentation) at the front of the prompt.
• Put highly variable user input at the end.
• Avoid inserting timestamps, random values, or session-unrelated dynamic content into the prefix.
• Use prefix hashing to merge requests that share a repository or task.
• In agent loops, reuse the same context skeleton rather than rebuilding the full prompt each turn.

Control the input-output ratio.
Output is usually far pricier than input, so trim unnecessary output: ask for concise structured results in tool calls, limit code generation to changed segments rather than full files, and compress or summarize conversation history in multi-turn dialogs to prevent unbounded context growth.

Assess the marginal value of long context.
More input tokens can improve quality but scale cost linearly. If some context contributes little, consider retrieval-augmented approaches that load relevant passages on demand instead of stuffing everything into the prompt.

A simple script helps compare strategies:

def estimate_cost(input_tokens, output_tokens, cache_hit_rate,
                  input_price_miss, input_price_hit, output_price):
    """
    Returns the composite cost per million effective tokens.
    All price arguments are in the same currency unit per million tokens.
    """
    input_avg = (input_price_hit * cache_hit_rate +
                 input_price_miss * (1 - cache_hit_rate))
    total = input_tokens * input_avg + output_tokens * output_price
    return total / (input_tokens + output_tokens) * 1_000_000

Teams can run sensitivity analyses across hit rates, input-output ratios, and pricing tiers to find the most cost-effective prompt organization.

7. Long-Term Implications for Developers

This pricing divergence is not a one-version or one-provider phenomenon. As disaggregated inference, prefix caching, and global scheduling become standard, cache hit rate will carry even more weight in cost structures. Developers will need to factor request-pattern design into engineering decisions, not just model capability.

For enterprise applications—long-context coding agents, knowledge-base QA, multi-turn automation—high cache reuse delivers a clear cost edge. For individual developers and prototyping, low hit rates mean per-token costs can be surprisingly high.

Inference cost is no longer a static number set solely by the provider. It is a dynamic outcome shaped by system architecture, request patterns, and pricing policy together. Understanding this mechanism helps developers strike a more rational balance between performance and spend.