LLM Inference Costs: How Cache Hit Rates Reshape Effective Unit Pricing

    This article explains how three-tier LLM pricing—cache-miss input, cache-hit input, and output—interacts with cache hit rate and input-to-output ratio to determine effective unit cost. It provides cost formulas, a pseudocode model, two workload examples, and engineering practices for improving prefix cache reuse and controlling output costs.

    When a new generation of large models is priced noticeably higher than the previous generation, the most discussed number is often “cost per million tokens.” For real businesses, however, the list price is only the starting point of the cost model. What ultimately determines the bill is the combination of three price tiers: cache-miss input, cache-hit input, and output. Looking only at the output price can easily lead to the conclusion that “prices have risen across the board.” But when cache hit rate, input-to-output ratio, and workload shape are included in the calculation, the effective unit price for the same model can differ several times across different users.

    The following discussion does not target any specific vendor’s billing rules. The example prices are used only to demonstrate cost calculation methods and do not represent the actual quotes of any platform.

    1. Three-Tier Pricing and Two Ratios: Cost Signals Behind List Prices

    LLM inference billing can usually be abstracted into three tiers: uncached input, cached input, and output. The reason input is split into cache-miss and cache-hit tiers is that in long-context scenarios, a large amount of prefix may be reused—for example, fixed system prompts, tool descriptions, code standards, knowledge base snippets, and conversation history. If the inference system can reuse the already-computed prefix state, the cache-hit tier is usually significantly cheaper than the cache-miss tier. The output side corresponds to token-by-token decoding, which involves memory bandwidth, scheduling, and timing overhead, so its unit price is often the highest.

    Therefore, when observing price changes, it is not enough to look only at absolute numbers. Two ratios also matter:

    1. The ratio between the cache-miss input price and the cache-hit input price. The higher this ratio is, the stronger the price incentive for cache reuse, the more “expensive” uncached input becomes, and the more businesses need to optimize prefix stability.
    2. The ratio between the output price and the cache-hit input price. The higher this ratio is, the more expensive output tokens are relative to cached input tokens, and the more valuable it becomes to control output length, reduce unnecessary generation, and use structured output and stop conditions.

    Suppose an example model has an output price of 100 RMB per million tokens, a cache-miss input price of 20 RMB per million tokens, and a cache-hit input price of 2 RMB per million tokens. The previous-generation example has an output price of 27 RMB, a cache-miss input price of 6.5 RMB, and a cache-hit input price of 1.1 RMB. The output price has increased by nearly 4x, cache-miss input by about 3x, and cache-hit input by about 2x. More importantly, the cache-miss/cache-hit ratio has expanded from about 6x to 10x, and the output/cache-hit ratio has expanded from about 25x to 50x. The signal from this structure is clear: the cache-hit tier is relatively moderate, the output tier has risen more sharply, and system design and workload shape will strongly affect the final cost.

    2. Effective Unit Price: Converting Three Prices into One Number

    A real bill should not be estimated directly from the “output price” or the “input price.” Instead, you should calculate the effective input price and the blended unit cost. Let:

    • P_in_miss be the cache-miss input price;
    • P_in_hit be the cache-hit input price;
    • P_out be the output price;
    • h be the cache hit rate;
    • N_in be the number of input tokens;
    • N_out be the number of output tokens.

    Then the effective input price is:

    P_in_eff = P_in_miss × (1 - h) + P_in_hit × h

    The blended cost per million tokens is:

    C = (N_in × P_in_eff + N_out × P_out) / (N_in + N_out)

    This formula looks simple, but it explains why “list prices have risen” does not mean “all users’ costs rise proportionally.” The higher the cache hit rate, the closer the effective input price is to the cache-hit tier. The larger the input-to-output ratio, the greater the weight of the input side. The more output there is, the greater the impact of the output price on total cost.

    The cost model can be expressed in pseudocode:

    1. function effective_cost(P_in_miss, P_in_hit, P_out, hit_rate,
    2. input_tokens, output_tokens):
    3. effective_input_price =
    4. P_in_miss * (1 - hit_rate) + P_in_hit * hit_rate
    5. total_tokens = input_tokens + output_tokens
    6. total_cost =
    7. (input_tokens * effective_input_price
    8. + output_tokens * P_out) / total_tokens
    9. return effective_input_price, total_cost

    Engineering teams should embed this model in cost dashboards and track hit rate, input-to-output ratio, and cost per task separately by business line, task type, and model version, rather than monitoring only total token counts.

    3. Two Workload Profiles: The Same Model Can Differ Several Times in Cost

    Consider two typical scenarios.

    Scenario A: an enterprise agent for long-context code tasks, with an input-to-output ratio of about 20:1 and a cache hit rate of about 90%. Using the example prices, the effective input price is approximately:

    20 × 0.1 + 2 × 0.9 = 3.8 RMB per million tokens

    The blended cost is approximately:

    (20 × 3.8 + 1 × 100) / 21 ≈ 8.4 RMB per million tokens

    Scenario B: an occasional individual developer whose prompt changes each time, with an input-to-output ratio of about 5:1 and a cache hit rate of about 20%. The effective input price is approximately:

    20 × 0.8 + 2 × 0.2 = 16.4 RMB per million tokens

    The blended cost is approximately:

    (5 × 16.4 + 1 × 100) / 6 ≈ 30.3 RMB per million tokens

    The former’s unit cost is about one-third of the latter’s. The difference does not come from the model “favoring” one user over another; it comes from workload shape. Highly reused system prompts, stable tool descriptions, fixed code standards, and long prefix caching place a large amount of enterprise agent input in the cache-hit tier. In contrast, individual usage that changes prompts each time, inserts random content, and frequently alters context places more input in the cache-miss tier.

    This shows that when discussing price, it is important to distinguish between “list price” and “effective price.” For enterprise customers with long-running workloads and highly reused prefixes, cache hits act like a structural discount. For users with low reuse, short sessions, and frequently changing prompts, usage is closer to paying the original prices for cache-miss input and output. If output accounts for a high proportion, the impact of output-tier price increases will be further amplified.

    4. Why the Cache-Hit Tier Is Relatively Moderate and Output Is More Expensive

    From a techno-economic perspective, cached input is cheaper because reusing prefixes avoids repeated prefill computation and some state construction. The output side requires token-by-token decoding, with more intensive computation and memory access, and scheduling overhead is harder to amortize through caching. As model capabilities improve, contexts become longer, and inference pipelines become more complex, cost pressure on the output side usually becomes more obvious.

    Therefore, if a new-generation model widens the “cache-miss/cache-hit” ratio and the “output/cache-hit” ratio, it is essentially using the price structure to guide users toward workload optimization: stabilizing prefixes, reducing repeated input, improving cache reuse, and controlling output length. For platforms, this helps improve inference resource utilization. For users, the optimization focus needs to shift from “sending fewer requests” to “increasing prefix reuse” and “reducing unnecessary output.”

    5. Engineering Practices: Improve Hit Rate and Control Output Cost

    First, stabilize prefixes. System prompts, tool definitions, code standards, and output format instructions should be fixed as much as possible. Avoid inserting high-frequency changing content such as timestamps, random IDs, or user nicknames into the prefix. Dynamic information can be moved later into the user message or the end of the context.

    Second, organize context in layers. Place long-term stable knowledge, rules, and templates at the front, and place temporary conversation, retrieval results, and task parameters at the back. This is better for prefix cache reuse and also makes auditing and version management easier.

    Third, template prompts. Build a template library for common tasks to reduce the need to assemble prompts from scratch each time. Template versions should be controllable to avoid frequent minor changes that invalidate the cache.

    Fourth, control output. Use explicit stop conditions, maximum output length, structured output constraints, and summary-style returns. For agent tasks, prioritize returning necessary fields and operation instructions rather than lengthy explanations.

    Fifth, monitor key metrics. At minimum, track cache hit rate, cache-miss input share, output token share, effective input price, cost per task, and cost per successful task. During procurement evaluation, require stress tests based on business profiles rather than comparing only the list price per million tokens.

    Sixth, pay attention to isolation and invalidation. Cache reuse needs to account for tenant isolation, permission boundaries, and privacy requirements. Dynamic content, version upgrades, and context reordering can all cause hit rates to decline. The cost model should be able to simulate the impact of hit-rate fluctuations on the bill.

    6. Conclusion: Decide with Effective Unit Price, Not List Price

    The fact that a new-generation model is priced higher than the previous generation does not mean that all users’ costs rise proportionally. Under three-tier pricing, cache hit rate and input-to-output ratio reshape effective unit price. High-reuse enterprise agents can achieve lower unit costs through stable prefixes and cache reuse. Low-reuse individual developers who change prompts each time are more likely to pay for cache-miss input and the high output price. The rational approach is to build a cost model, calculate effective input price and blended cost based on business workload, and then make decisions using cache hit rate, output control, and procurement tests. List price indicates direction; effective unit price determines the bill.