LLM API Pricing: Why Cache-Hit, Cache-Miss, and Output Tokens Have Three Tiers

    A new flagship model's API price change looks like a broad increase across all tiers, but the relative price ratios reveal a deeper shift from token-count pricing toward context-reuse pricing. This article breaks down the cost structure of prefill, KV cache reuse, and decode; compares effective unit prices for high-reuse enterprise agents and low-reuse individual developers; explains why agent workloads make cache economics critical; and outlines engineering practices developers can use to reduce their real cost at the same list price.

    1. Event Review: A Price Change That Looks Like a Broad Increase

    Recently, an LLM provider announced API pricing for a new flagship model. On the output side, the price rose from about CNY 27 per million tokens for the previous programming-specialized version to CNY 100 per million tokens, an increase of nearly 4x. On the input side, the cache-miss price rose from CNY 6.5 to CNY 20, and the cache-hit price rose from CNY 1.1 to CNY 2, both roughly 3x higher. On the surface, this looks like a comprehensive price increase.

    But when the ratios among the three tiers are examined, the pricing structure has changed fundamentally:

    Price ratio Previous generation New generation
    Input cache miss / cache hit ~6x ~10x
    Output / cache-hit input ~25x ~50x

    In other words, the cache-hit tier had the mildest increase of all tiers, while the output tier rose the most sharply. This scissors-like adjustment is not simply a pass-through of higher costs; it is a重构 of pricing strategy. Understanding it requires returning to the cost structure of LLM inference itself.

    2. Inference Cost Structure: Why Output Is the Most Expensive and Cache Hits Are the Cheapest

    From the perspective of an inference system, the cost of an LLM request is mainly composed of three parts, and their cost curves have completely different shapes.

    1. Prefill stage (input processing): The input prompt is computed in parallel to generate the KV Cache. This stage is highly parallel and achieves high GPU utilization, so the per-token compute cost is relatively low. But if every request has a completely new prompt, the KV Cache is discarded after use, and compute resources are wasted repeatedly.

    2. KV Cache storage and reuse: In long-context scenarios, KV Cache memory usage grows linearly with context length. Mainstream inference optimization approaches, such as disaggregated inference architectures, separate KV Cache from compute nodes and place it in a dedicated cache layer, so subsequent requests with the same prefix can hit the cache directly and skip most prefill computation. Data disclosed in vendor technical blogs shows that in coding scenarios such architectures can achieve cache hit rates above 90%, because a coding agent’s system prompt, codebase context, and conversation history are highly reused across multiple interactions.

    3. Decode stage (output generation): Tokens are generated autoregressively one by one. Each step must read the full KV Cache, bandwidth consumption grows linearly or even superlinearly with context length, and unlike prefill, it cannot be parallelized in the same way. This is the highest-cost and hardest-to-optimize part of the three tiers.

    So the pricing gradient is effectively a mirror of the cost structure: cache-hit input is the cheapest because it reuses prefill cost that has already been paid; cache-miss input is next because it requires recomputation; output is the most expensive because every step must pay the bandwidth cost for the entire context. The new generation widening the output / cache-hit input ratio to 50x reflects precisely the sharp rise in decode-stage cost pressure in long-context agent scenarios.

    3. Two Calculations: Same List Price, Different Real Unit Prices

    Once the pricing structure is understood, it becomes clear who is paying full price and who is getting a discount. Consider two typical scenarios:

    Scenario A: Enterprise agent, input/output ratio 20:1, cache hit rate 90%

    1. Effective input price ≈ 2 × 0.9 + 20 × 0.1 = CNY 3.8 per million tokens
    2. Output price = CNY 100 per million tokens
    3. Blended cost ≈ (20 × 3.8 + 1 × 100) / 21 ≈ CNY 8.4 per million tokens

    Scenario B: Individual developer, different prompt every time, input/output ratio 5:1, cache hit rate 20%

    1. Effective input price ≈ 2 × 0.2 + 20 × 0.8 = CNY 16.4 per million tokens
    2. Output price = CNY 100 per million tokens
    3. Blended cost ≈ (5 × 16.4 + 1 × 100) / 6 ≈ CNY 30.3 per million tokens

    For the same model from the same provider, the unit cost in Scenario A is only about 1/3.6 of that in Scenario B. What looks like a broad price increase is in fact precise customer segmentation:

    • Enterprise customers that run the same system prompt for long periods and reuse prefixes heavily receive a real discount, and may even see a lower actual bill than with the previous generation, because the new model is more capable and may require fewer total tokens to complete a task.
    • Individual developers who change prompts every time and make low-frequency calls pay list price or even a premium.

    4. Why Now: The Agent Era Changes the Cost Equation

    In the chat era, an interaction involved dozens to thousands of tokens. Users were not sensitive to unit price, did not spend much, and providers could simply offer flat pricing.

    The agent era is completely different. Take a typical long-horizon task disclosed by the industry: a single task can involve thousands of tool calls and consume hundreds of thousands of tokens, with hundreds of agents running autonomously in parallel for days. Under such workloads:

    1. KV Cache storage becomes a rigid cost: the longer the context, the more memory and storage the cache layer occupies, and it must remain resident for active sessions.
    2. Decode bandwidth cost rises linearly with context: every generated token requires scanning the KV Cache of the entire context.
    3. The marginal benefit of cache hit rate is extremely high: for every 10 percentage points of hit-rate improvement, prefill computation drops by nearly the same proportion.

    Running such tasks under the old pricing would most likely lose money on a single call. The essence of the new pricing is therefore to turn “degree of context reuse” into a pricing dimension: workloads with high reuse are subsidized, while workloads with low reuse bear the real cost. This is not discrimination against users; it internalizes externalities. Cache-layer resources are limited. Without a gradient, high-frequency reusers and low-frequency users would cross-subsidize each other, and the end result would be either a price increase for everyone or an unsustainable service.

    5. Developer Strategies: Turn Your Bill into a “Discount Price”

    Whichever model service you choose, this kind of pricing structure is becoming an industry trend. Developers can adapt proactively on the engineering side:

    1. Maximize prefix reuse. Place stable system prompts, tool definitions, and codebase summaries at the very front of the prompt and keep them byte-for-byte identical. Place dynamic content such as user input and tool results at the end. Even a one-character difference can break the cache.

    2. Actively manage session context. In an agent loop, periodically compact the growing conversation history into summaries, control the context length of each session, and reduce bandwidth cost in the decode stage.

    3. Use batching and concurrency to amortize cost. For tasks without real-time requirements, use batch inference endpoints, which usually offer significant discounts. For multi-agent parallel scenarios, evaluate whether multiple agents can share the same long-context prefix to improve the overall hit rate.

    4. Build token-level monitoring. Record the distribution of cache-hit tiers for each request at the application layer, and regularly calculate the real blended unit price. Here is a minimal calculation example:

    1. def effective_price(hit_price, miss_price, out_price, hit_rate, io_ratio):
    2. in_price = hit_price * hit_rate + miss_price * (1 - hit_rate)
    3. return (io_ratio * in_price + out_price) / (io_ratio + 1)
    4. # Enterprise agent scenario
    5. print(effective_price(2, 20, 100, 0.9, 20)) # ≈ 8.4
    6. # Low-frequency individual developer scenario
    7. print(effective_price(2, 20, 100, 0.2, 5)) # ≈ 30.3

    5. Choose model tiers based on workload characteristics. High-reuse, long-context agent tasks are suitable for the new flagship model. For low-frequency, independent simple tasks, the previous generation or a lightweight model may be more cost-effective.

    6. Conclusion

    The most valuable insight from this price change is not how much prices rose, but what it reveals about the next phase of LLM commercialization: the pricing unit is shifting from “token count” to “context reuse efficiency”. For providers, cache discounts are a pricing lever for selecting high-value workloads. For developers, the ability to maximize prefix reuse and control context growth in engineering will directly determine the real cost gap at the same list price, and that gap can exceed 3x. The price list is public, but the real unit price is something you calculate yourself.