LLM API Price Hikes: Cache Hit Rate Determines Real Cost

    This article explains why higher LLM API list prices do not necessarily translate into proportional cost increases for every workload. It models three-tier billing—uncached input, cached input, and output—and shows how cache hit rate and input-to-output ratio determine effective input price and blended cost. It compares an enterprise agent scenario with a low-frequency developer scenario, examines why cache hit rate can create several-fold differences in real cost, and outlines engineering practices for incorporating cache hit rate into TCO planning.

    Recent increases in LLM API pricing have triggered considerable discussion. On the surface, output prices have risen several-fold from one generation to the next, while both uncached input and cached input tiers have also increased. It may appear that all callers will face the same proportional cost increase. From an engineering and billing-structure perspective, however, the list price is only a “nominal unit price.” What really determines the bill is the characteristics of the workload: the input-to-output ratio, cache hit rate, output-length distribution, and the degree of request-prefix reuse. Once these factors are included in the model, the actual impact of the same “price increase” can differ by several times across users.

    This article does not focus on a specific product. Instead, it abstracts the discussion into a general technical question: when a model service uses three-tier billing for uncached input, cached input, and output, how should effective cost be calculated? Why does cache hit rate become a key lever? Why do enterprise agents and low-frequency developers see different real prices? And how can engineering teams optimize for this?

    1. Higher List Prices Do Not Mean Proportionally Higher Effective Costs

    LLM inference cost can be broadly divided into two stages: prefill and decode. The input side mainly occurs during prefill, where the model encodes prompts, system instructions, tool definitions, and context documents. The output side occurs during decode, where the model generates tokens autoregressively and is usually more sensitive to latency and compute occupancy. As a result, output unit prices being higher than input unit prices is common in the industry.

    The point of cache billing is that if multiple requests share the same prefix—such as the same system prompt, tool schema, codebase description, or multi-turn conversation history—the computation result for that prefix can be reused, reducing repeated prefill cost. The cache-hit price is usually significantly lower than the uncached input price and also lower than the output price. Therefore, the input cost of a request is not simply calculated at the “uncached input unit price.” Instead, it is weighted by hit and miss:

    Effective input price = cached input price × hit rate + uncached input price × (1 - hit rate)

    Further, if the number of input tokens is N_in, the number of output tokens is N_out, and the input-to-output ratio is r = N_in / N_out, then the blended cost per million tokens can be written as:

    Blended cost = (r × effective input price + output price) / (r + 1)

    This formula shows that the final bill is determined by three variables together: the three unit prices, the cache hit rate h, and the input-to-output ratio r. Focusing only on the output price or on headlines saying “prices are rising across the board” can easily lead to misjudging the real cost.

    2. Using Illustrative Prices to See Cost Stratification

    For illustration, assume a model generation uses the following indicative prices: uncached input at 20 CNY per million tokens, cached input at 1 CNY per million tokens, and output at 100 CNY per million tokens. Note that these are used only for cost-modeling demonstration and do not represent any specific vendor’s billing commitment. The previous generation can be assumed to have prices of approximately 6.5 CNY for uncached input, 1.1 CNY for cached input, and 27 CNY for output. The comparison shows that the output tier has increased more noticeably, while the cached tier has remained relatively moderate.

    Scenario A: an enterprise long-context coding agent. The input-to-output ratio is about 20:1, and the cache hit rate is 90%. Effective input price = 1 × 0.9 + 20 × 0.1 = 2.9 CNY per million tokens. Blended cost = (20 × 2.9 + 1 × 100) / 21 ≈ 7.52 CNY per million tokens.

    Scenario B: a low-frequency developer whose prompts differ each time. The input-to-output ratio is about 5:1, and the cache hit rate is 20%. Effective input price = 1 × 0.2 + 20 × 0.8 = 16.2 CNY per million tokens. Blended cost = (5 × 16.2 + 1 × 100) / 6 ≈ 30.17 CNY per million tokens.

    Under the same illustrative prices, the unit blended cost in Scenario A is about one quarter of that in Scenario B. If the cache-hit price is set to 2 CNY instead of 1 CNY, the enterprise scenario’s blended cost is about 8.38 CNY, while the individual scenario’s is about 30.33 CNY; the ratio is still about 3.6 times. This shows that beneath an apparent “across-the-board increase,” enterprise customers that repeatedly reuse the same system prompts, tool definitions, and prefix context are actually paying a weighted “discounted price.” By contrast, callers that change prompts every time and have low cache hit rates are closer to paying the “original price” formed by uncached input plus a high output unit price.

    3. Why Cache Hit Rate Can Create Several-Fold Differences

    Workloads with high cache hit rates usually have stable prefixes: fixed system instructions, fixed tool schemas, fixed codebase index descriptions, and multi-turn conversation histories that are continuously appended rather than reordered. Coding agents, customer service quality inspection, document Q&A, and fixed workflow automation naturally involve substantial prefix reuse. Common industry technical approaches use disaggregated inference architectures to decouple prefill and decode resources and reuse prefix caches across requests, thereby achieving higher hit rates in long-context scenarios.

    Conversely, if every request inserts timestamps, random IDs, dynamically sorted retrieval results, or frequently reorders system prompts and tool definitions, the cache prefix is broken and the hit rate drops quickly. In this case, even if the input unit price appears lower than the output unit price, a higher share of uncached input will significantly push up effective input cost. More importantly, output cost accounts for a large share of blended cost: at a 20:1 input-to-output ratio, output still contributes about 100/21 ≈ 4.76 CNY, more than 60% of total cost; at 5:1, output contributes about 16.67 CNY, more than half. Therefore, an output price increase is more unfavorable for low-frequency, short-input, long-output workloads.

    4. Engineering Practice: Incorporate Cache Hit Rate into the TCO Model

    First, stabilize prefixes. Place system prompts, tool definitions, and output-format constraints at the very beginning and keep them byte-level stable; put dynamic time, user IDs, random seeds, and similar elements later. Second, reduce prefix reordering. For multi-turn conversations, append messages as much as possible rather than reordering history every turn. Third, use chunked summaries for long documents. For codebases or knowledge bases that appear repeatedly, cache stable summaries and index descriptions instead of concatenating the full original text each time. Fourth, monitor hit rate. Track cache hit rate and input-to-output ratio by tenant, endpoint, and scenario to build a cost dashboard. Fifth, control output. Reduce ineffective decoding through structured output, maximum output limits, deduplication, and truncation strategies. Sixth, route models. High-frequency reuse and long-context tasks can prioritize cache-friendly services; for low-frequency random prompts, compare output unit price and uncached input price as the main focus.

    Below is vendor-neutral pseudocode for cost modeling:

    1. function effective_input_price(p_in, p_cache, hit_rate):
    2. return p_cache * hit_rate + p_in * (1 - hit_rate)
    3. function blended_cost(p_in, p_cache, p_out, hit_rate, input_output_ratio):
    4. e = effective_input_price(p_in, p_cache, hit_rate)
    5. return (input_output_ratio * e + p_out) / (input_output_ratio + 1)
    6. // Example: enterprise agent
    7. cost_enterprise = blended_cost(20, 1, 100, 0.90, 20)
    8. // Example: low-frequency random prompts
    9. cost_individual = blended_cost(20, 1, 100, 0.20, 5)

    During budget reviews, teams should not compare only the “output price per million tokens.” They should collect the input-output distribution and cache hit rate of real requests and calculate blended cost weighted by scenario. For high-frequency reuse workloads, a higher output unit price may still be offset by a high hit rate; for low-frequency random workloads, output unit price and uncached input price are the main risks. When selecting model services on Baidu AI Cloud, teams should also include cache hit rate, input-to-output ratio, and output-length distribution in the TCO model rather than looking only at the price list.

    5. Conclusion

    The core of this round of LLM API price adjustments is not that “all tiers rose by the same proportion,” but rather that the billing structure re-stratifies workloads. The cached tier is relatively moderate, while the output tier has risen more noticeably. Enterprise agents with high hit rates and high input-to-output ratios receive a weighted discount, while developers with low hit rates and random prompts bear a price close to uncached input plus a high output unit price. Engineering teams should treat cache hit rate as a first-class metric when designing context, organizing requests, and monitoring bills. Only by modeling with the effective input price and blended cost formulas can teams answer the truly important question: how large is the real impact of the price increase?