How LLM API Price Increases Reshape Costs: The Impact of Cache Hit Rates

    LLM API pricing is not determined by headline token rates alone. The effective cost depends on how uncached input, cached input, and output are priced, as well as on a workload’s input-to-output ratio and cache hit rate. This article uses normalized prices rather than any specific provider’s rates to explain the billing model, compare different workload profiles, and outline practical ways to reduce costs.

    1. From a Single Token Rate to Three-Dimensional Billing

    Early LLM APIs typically used a simple pricing model: input tokens and output tokens were billed separately. As long-context applications, multi-turn conversations, code agents, and tool calling became more common, billing evolved into three categories:

    • Uncached input
    • Cached input
    • Output

    Each category has a different cost basis, which explains the differences in pricing.

    Input processing is generally dominated by prefill computation, which can be handled in parallel and therefore has a relatively manageable per-token cost. When the same prefix is reused, a platform can cache and reuse intermediate computation results, offering a discount for the cached portion.

    Output generation is autoregressive: each token depends on the tokens generated before it. It is affected by memory bandwidth, batching efficiency, and decoding latency, so its per-token cost is typically higher. This is why output pricing is often higher than input pricing.

    When a new API price list shows a multiple-fold increase, looking only at the change in output pricing can be misleading. If cached input prices rise less than uncached input and output prices, the actual cost impact depends heavily on the workload. An enterprise agent with long, reusable prefixes and a high cache hit rate may achieve a lower effective unit cost. By contrast, one-off prompts, short sessions, and frequently changing low-volume requests will be closer to uncached input pricing and may absorb a larger premium.

    2. Why Cache Hits Can Change the Effective Unit Price

    A cache hit reuses the intermediate state of an identical prefix, reducing repeated prefill computation. Common reusable prefixes include:

    • Stable system prompts
    • Fixed tool descriptions and parameter schemas
    • Long-term project conventions
    • Earlier conversation history that remains unchanged
    • Shared codebase summaries used by coding agents

    As long as the prefix token sequence remains stable, subsequent requests may be able to reuse the cached computation.

    However, caching does not automatically reduce costs. Its value depends on how much of the workload can be reused. Enterprise agents often have a high input-to-output ratio. In long-context coding tasks, for example, the number of input tokens may be dozens of times larger than the number of output tokens. Their system prompts, tool definitions, and project context are also relatively stable, making a high cache hit rate achievable.

    Infrequent developers often have the opposite profile: every prompt is different, sessions are short, and output accounts for a larger share of usage. Their cache hit rate is low, so their effective cost is closer to uncached input and output pricing.

    It is also important to distinguish between prefix caching and semantic caching.

    Prefix caching usually requires the token sequence to be exactly the same. If a dynamic timestamp, random ID, user name, or retrieval result is placed near the beginning of the prefix, it can invalidate caching for the content that follows. Semantic caching is a separate engineering technique that may reduce the number of API calls, but it does not necessarily qualify for a platform’s cached-input discount.

    In practice, engineers should first stabilize exact prefixes and then consider semantic-layer optimizations.

    3. A Normalized Cost Model

    To illustrate the mechanism, assume the following normalized prices:

    • Uncached input: 1
    • Cached input: 0.1
    • Output: 5

    Under this assumption, output costs five times as much as uncached input and 50 times as much as cached input. These values are for demonstration only and do not represent any specific provider’s pricing.

    Let r be the input-to-output ratio, meaning that every output token corresponds to r input tokens. Let h be the cache hit rate. The effective input price is:

    1. input_eff = p_hit × h + p_miss × (1 - h)

    The blended cost per token is:

    1. cost = (r × input_eff + p_out) / (r + 1)

    Example 1: Enterprise Agent

    Assume:

    • r = 20
    • h = 90%

    The effective input price is:

    1. 0.1 × 0.9 + 1 × 0.1 = 0.19

    The blended cost is:

    1. (20 × 0.19 + 5) / 21 ≈ 0.419

    In this scenario, output accounts for approximately 56.8% of the combined cost, while input accounts for approximately 43.2%. Although output has a high unit price, the large input volume and high cache hit rate substantially reduce the blended cost.

    Example 2: Infrequent Developer

    Assume:

    • r = 5
    • h = 20%

    The effective input price is:

    1. 0.1 × 0.2 + 1 × 0.8 = 0.82

    The blended cost is:

    1. (5 × 0.82 + 5) / 6 ≈ 1.517

    The ratio between the two scenarios is approximately:

    1. 1.517 / 0.419 ≈ 3.62

    In other words, under the same price list, the unit cost for a highly reusable enterprise agent may be only about one-third to one-quarter of the cost of low-frequency, one-off requests.

    A simple implementation of the model looks like this:

    1. function effective_input_price(p_hit, p_miss, hit_rate):
    2. return p_hit * hit_rate + p_miss * (1 - hit_rate)
    3. function blended_cost(input_output_ratio, hit_rate,
    4. p_hit = 0.1, p_miss = 1.0, p_out = 5.0):
    5. input_eff = effective_input_price(p_hit, p_miss, hit_rate)
    6. return (input_output_ratio * input_eff + p_out) / (input_output_ratio + 1)
    7. for scenario in [
    8. {name: "High-reuse enterprise agent", ratio: 20, hit_rate: 0.9},
    9. {name: "Low-frequency one-off request", ratio: 5, hit_rate: 0.2}
    10. ]:
    11. print(scenario.name, blended_cost(scenario.ratio, scenario.hit_rate))

    The value of this model is that it converts a published price list into a workload-based unit cost. Developers can calculate the input-to-output ratio, cache hit rate, output-length distribution, and retry rate from production logs, then use those values for estimation.

    If the billing model also includes cache-write, storage, or other charges, those should be modeled separately. The central logic remains the same: a higher cache hit rate increases the value of the cache discount, while a higher proportion of output increases the impact of output pricing.

    4. Who Benefits and Who Bears the Pressure?

    The workload profile is often more important than the price table itself.

    Highly reusable enterprise agents typically have three characteristics:

    • Fixed system prompts
    • Long contexts
    • Multiple rounds of tool calls

    They use many input tokens, relatively fewer output tokens, and achieve high cache hit rates. As a result, they are less sensitive to increases in uncached input pricing. Even when output prices rise, the impact may remain manageable because output represents a smaller share of total usage.

    Low-frequency developers and one-off tasks usually have shorter prompts, more frequent prompt changes, and a higher output share. Their cache hit rate is lower, so they are more exposed to increases in both uncached input and output pricing.

    This explains why a price list can appear to show a broad increase while the effective cost varies significantly by customer. High-reuse workloads may receive a structural discount through caching, while low-frequency requests may pay close to the uncached rate or face a larger effective premium.

    When evaluating procurement or operating costs, organizations should not compare only the price per million tokens. They should calculate costs at the level of actual business tasks, such as:

    • Cost per 1,000 code completions
    • Cost per 100 agent tasks
    • Cost per 10,000 question-and-answer requests

    Retries, failed reruns, context growth, and excessively long outputs should also be included in the calculation.

    5. Engineering Checklist: Increase Cache Hits and Control Output

    1. Stabilize the Prefix

    Place system prompts, tool schemas, and project conventions at the beginning of the context, and put dynamic content later. Keep tool descriptions in a fixed order, and avoid placing random IDs, timestamps, or user names in the prefix.

    2. Layer the Context

    Separate long-term stable context from short-term retrieval results. This reduces prefix churn and helps preserve reusable content.

    3. Control Output

    Use maximum output limits, stop conditions, structured output, and summary compression to reduce unnecessary generation. Output is an important cost lever, especially when output tokens are priced significantly higher than input tokens.

    4. Route Tasks by Complexity

    Send simple tasks to smaller models or cached results, and reserve higher-capability models for more complex work.

    5. Monitor the Right Metrics

    Track the following metrics continuously:

    • Cache hit rate
    • Input-to-output ratio
    • Cost per task
    • P95 output length
    • Retry rate

    These measurements make cost optimization data-driven rather than dependent on headline pricing.

    6. Conclusion: Real Cost Is a Function of Price and Workload

    LLM API pricing cannot be summarized simply as “more expensive” or “cheaper.” As billing structures become more detailed, the gap between published rates and real costs becomes larger.

    The combination of output pricing, uncached input pricing, and cached input pricing defines the cost function. The input-to-output ratio and cache hit rate determine where each workload falls within that function.

    For enterprises, the main optimization priorities should be prefix reuse, context layering, and output control. For individual developers, a log-based cost model is essential for making informed decisions instead of relying only on published token rates.

    The price table is only the starting point. The workload is the final explanation of unit cost.