Why LLM Price Increases Do Not Tell the Whole Story: Cache Hit Rate Determines Your Actual Cost

    List prices for large language models can rise sharply, but the real impact on your bill depends on more than the advertised input and output rates. By combining cache hit rate, input-to-output ratio, and output volume, you can estimate effective cost more accurately. This article explains the pricing structure, provides a practical cost formula, and shows why enterprise workloads with reusable prefixes may be far cheaper than low-reuse developer workloads.

    1. Distinguishing List-Price Increases from Effective Unit Cost

    The most common mistake when discussing LLM pricing is to look only at the absolute input or output rate and assume that a price increase means every user’s year-over-year cost has increased by the same amount.

    Modern LLM services are usually billed across at least three dimensions:

    • Input tokens that do not hit the cache
    • Input tokens that hit the cache
    • Output tokens

    The increase in each category can be very different. More importantly, the share of each category in the total bill depends entirely on the workload.

    Using a commonly discussed example for illustration, an earlier-generation code model charged approximately RMB 27 per million output tokens, while a newer model charged approximately RMB 100 per million output tokens. The price for uncached input increased from about RMB 6.5 to RMB 20 per million tokens. Cached input increased more moderately, from roughly RMB 1.1 to the RMB 1–2 range per million tokens.

    These figures are examples used to explain cost structure. They do not represent the current pricing of any specific product. Actual charges should always be checked against the provider’s published pricing rules.

    Billing tier Example: previous generation Example: new generation Change pattern
    Uncached input About RMB 6.5/million tokens About RMB 20/million tokens About 3×, a significant increase
    Cached input About RMB 1.1/million tokens About RMB 1–2/million tokens Relatively moderate
    Output About RMB 27/million tokens About RMB 100/million tokens Nearly 4×, the largest increase

    The table shows that output pricing increased the most, while cached-input pricing increased relatively modestly.

    The ratios are even more revealing:

    • The ratio of uncached input to cached input increased from approximately 6× to approximately 10×.
    • The ratio of output to cached input increased from approximately 25× to approximately 50×.

    This sends a clear signal: the pricing structure encourages prefix reuse, cache hits, and reduced repeated prefill computation, while placing a higher cost on workloads with large output volumes and little reuse.

    Therefore, “prices increased across the board” is only a description of list prices. The actual bill depends on several variables:

    • How long the requests are
    • The input-to-output ratio
    • The cache hit rate
    • The total output volume

    Comparing only the price per million tokens while ignoring these variables can lead to the wrong conclusion.

    2. A Blended Cost Formula

    Let:

    • (h) be the cache hit rate
    • (p_{miss}) be the price of uncached input
    • (p_{hit}) be the price of cached input
    • (p_{out}) be the output price
    • (r) be the input-to-output ratio

    Here, (r) means that for every unit of output, the workload generates an average of (r) units of input.

    The effective input price is:

    [
    p{in}^{eff}=p{hit}\times h+p_{miss}\times(1-h)
    ]

    The blended cost per million mixed tokens can then be calculated as:

    [
    C=\frac{r\times p{in}^{eff}+p{out}}{r+1}
    ]

    This formula converts the three separate billing tiers—input, output, and cache status—into a single comparable metric.

    The logic can be represented in pseudocode:

    1. function effective_input_price(hit_price, miss_price, hit_rate):
    2. return hit_price * hit_rate + miss_price * (1 - hit_rate)
    3. function blended_cost(hit_price, miss_price, output_price, hit_rate, input_output_ratio):
    4. input_price = effective_input_price(hit_price, miss_price, hit_rate)
    5. return (input_output_ratio * input_price + output_price) / (input_output_ratio + 1)
    6. enterprise_workload = blended_cost(2, 20, 100, 0.9, 20)
    7. individual_developer_workload = blended_cost(2, 20, 100, 0.2, 5)

    Using the example prices:

    • Enterprise long-context workload: With an input-to-output ratio of approximately 20:1 and a 90% cache hit rate, the effective input price is (2\times0.9+20\times0.1=3.8) RMB per million tokens. The blended cost is approximately ((20\times3.8+100)/21\approx8.4) RMB per million mixed tokens.
    • Individual developer workload: With a 5:1 input-to-output ratio and a 20% cache hit rate, the effective input price is (2\times0.2+20\times0.8=16.4) RMB per million tokens. The blended cost is approximately ((5\times16.4+100)/6\approx30.3) RMB per million mixed tokens.

    For the same model, the first workload costs roughly one-third to one-quarter as much per unit as the second.

    If the cached-input price is RMB 1 instead of RMB 2, the enterprise workload falls further to approximately RMB 7.5 per million mixed tokens. The conclusion remains the same: cache hit rate is a cost lever, while the input-to-output ratio amplifies cost.

    3. Why Enterprise Customers Can Achieve Lower Effective Costs

    Enterprise applications with long contexts—particularly coding agents, knowledge-base question answering, and automated operations assistants—often have highly stable prefixes.

    These prefixes may include:

    • A fixed system prompt
    • Stable tool descriptions
    • A codebase summary
    • A consistent conversation-history structure

    Only the user’s question and a small amount of dynamic context change from one request to the next.

    This type of workload is naturally suitable for prefix caching. The same prefix needs to be computed only once, while subsequent requests can reuse the cached content and pay a lower input rate.

    Technical discussions commonly describe techniques such as separating prefill from decoding, reusing KV caches across layers, and optimizing request scheduling. In programming-related workloads, these approaches may raise cache hit rates above 90%. However, this figure should not be treated as a default for every application. It represents a possible range after engineering optimization.

    If an enterprise increases its cache hit rate from 20% to 90%, the effective input price falls from RMB 16.4 to RMB 3.8 per million tokens—a reduction of more than 70%. Even if the output price remains RMB 100 per million tokens, the blended cost can still be reduced substantially.

    However, enterprises do not automatically receive low costs simply because they are enterprise customers. A high cache hit rate requires engineering discipline, including:

    • Stable prefixes
    • Session stickiness
    • Cache lifetime management
    • Careful context organization
    • Monitoring of the proportion of hit tokens

    If an application continuously reorders its tool list, inserts a changing timestamp, or randomizes the system prompt, its cache hit rate can fall significantly. In that case, the so-called enterprise discount disappears.

    The lower effective cost of enterprise workloads is therefore the result of reusable traffic patterns and engineering optimization—not merely customer identity.

    4. Why Individual Developers Are More Likely to Pay a Premium

    Individual developers and lightweight applications typically have workloads with the following characteristics:

    • The prompt changes on every request
    • The context is relatively short
    • The input-to-output ratio is lower
    • The cache hit rate is low

    For example, one-off questions, temporary script generation, and small-scale text rewriting provide limited opportunities for prefix reuse. The system prompt may be short, while the user input changes substantially between requests. As a result, cache hits are difficult to achieve.

    In these scenarios, much of the input is billed at the uncached rate. At the same time, output accounts for a larger share of total usage, so the effective cost naturally approaches—or may even exceed—the advertised rate.

    With a 5:1 input-to-output ratio and a 20% cache hit rate, the blended cost in the example is approximately RMB 30.3 per million mixed tokens.

    If output volume increases, the RMB 100 output price has a stronger impact. If prompts change frequently, the RMB 20 uncached-input price becomes more important as well.

    The pricing structure therefore suggests several practical principles for these users:

    • Avoid repeatedly sending long inputs
    • Control output length
    • Remove unnecessary long-context content
    • Reuse stable prefixes whenever possible

    Without these optimizations, the bill can rise because of workload structure even if the provider’s list price remains unchanged.

    5. Cache Hit Rate Should Be Observable, Not Blindly Maximized

    Cache hit rate is a critical metric, but maximizing it at any cost is not always the right goal.

    Caching introduces storage, indexing, expiration, and scheduling overhead. Overly rigid prefixes may reduce application flexibility. In addition, when a cache expires, traffic can suddenly fall back to the uncached price, creating a billing spike.

    Enterprises should monitor cache hit rate together with:

    • Uncached token volume
    • Output token volume
    • Request distribution
    • Session stickiness

    On general-purpose cloud platforms such as Baidu AI Cloud, billing analysis, monitoring alerts, and log services can be combined to build a cost dashboard. Regardless of the model service being used, the important point is to keep measurement consistent.

    For each request, record:

    • Input tokens
    • Output tokens
    • Hit tokens
    • Missed tokens
    • Model pricing tier
    • Cache lifetime

    Then use the formula above to calculate the effective unit cost.

    Only after converting list prices into effective costs can procurement and architecture decisions be based on actual usage patterns.

    6. Conclusion: A Price Increase Is a Pricing Signal, Not the Final Cost Result

    New-generation models may have significantly higher list prices, with output rates increasing the most and cached-input rates increasing more moderately. The ratios between uncached and cached input, as well as between output and cached input, may also become much larger.

    This is not simply a case of every workload becoming more expensive. It is a pricing structure designed to differentiate workloads:

    • Enterprise customers running stable system prompts with highly reusable prefixes can achieve lower unit costs through high cache hit rates.
    • Individual developers who frequently change prompts and have little prefix reuse are more likely to pay close to the list price or experience a premium.

    When evaluating model costs, do not ask only, “How much does the model charge per million tokens?” Also ask:

    1. What is the input-to-output ratio?
    2. What cache hit rate can the workload achieve?
    3. Can output volume be reduced?
    4. Are the prefixes stable and reusable?

    Once these variables are quantified, the blended cost formula can reveal the actual bill behind the advertised price.

    For developers, improving cache friendliness and context organization may reduce costs more effectively than simply comparing list prices. For enterprises, building an observable cost-management system is more important than pursuing a single low-priced billing tier.