Optimizing LLM Input Costs: How Cache Hit Rate Determines Actual Billing

    This article explains how cache hit rate and cached-token pricing determine the effective input cost of LLM APIs. It covers the billing formula, a worked example, engineering practices that improve prefix cache hit rates, monitoring metrics, cost-accounting code, common pitfalls, and implementation recommendations.

    Optimizing LLM Input Costs: How Cache Hit Rate Determines Actual Billing

    1. Why Standard Unit Price Alone Can Mislead Cost Estimates

    LLM APIs typically bill input tokens and output tokens separately. Many providers offer cached billing for repeated input prefixes: when a request prefix has already been cached on the server, those tokens are billed at a lower unit price. As a result, the same standard unit price can lead to very different actual bills under different calling patterns.

    When evaluating cost, the standard unit price is only a starting point. Two variables determine actual spending: the ratio of the cached price to the standard price, and the cache hit rate, meaning the proportion of input tokens that hit the cache. Looking only at the standard price is equivalent to assuming a hit rate of zero, which is not realistic in most continuous conversations, batch processing, and fixed system-prompt scenarios.

    2. Basic Structure of the Billing Model

    Let the standard input price be P, the price for cached hits be C, and the hit rate be h (a decimal between 0 and 1). The effective price per million input tokens is:

    Effective price = h × C + (1 − h) × P

    This formula depends on only three parameters and applies to any service that uses tiered cache billing. It leads to two conclusions. First, the effective price decreases linearly with the hit rate; for every 10 percentage points increase in hit rate, the savings are fixed at (P − C) × 0.1. Second, the closer the cached price is to the standard price, the smaller the benefit from caching. Therefore, model selection should compare both prices at the same time.

    Example Calculation

    The following numbers are hypothetical and are used only to demonstrate the calculation method. Actual prices should be taken from the provider’s pricing page. Assume a standard price P = 10 currency units per million tokens and a cached price C = 2 currency units per million tokens:

    • Hit rate 0%: effective price = 10, equivalent to the standard price.
    • Hit rate 50%: effective price = 0.5 × 2 + 0.5 × 10 = 6, about 60% of the standard price.
    • Hit rate 90%: effective price = 0.9 × 2 + 0.1 × 10 = 2.8, about 28% of the standard price.

    As the hit rate rises from 0% to 90%, input cost drops by more than 70%. This shows that engineering optimization is often more valuable than price comparison alone.

    3. Engineering Factors That Affect Cache Hit Rate

    Caching is usually based on input prefix matching. Once the prefix changes, the content after it cannot hit the cache. Therefore, the key to improving hit rate is to keep request prefixes as stable as possible.

    3.1 Put Fixed Content First

    Place unchanged content such as system prompts, tool definitions, role settings, and long documents at the beginning of the request, and put the changing user question at the end. Reversing the order causes the prefix to differ on every request, making caching largely ineffective.

    3.2 Avoid Dynamic Information in the Prefix

    If dynamic fields such as timestamps, request IDs, random strings, or usernames appear at the beginning of the prompt, they break prefix consistency. Such information should be placed at the end or omitted when it does not affect semantics.

    3.3 Keep Conversation History Incremental

    In multi-turn conversations, each turn should append content to the full context of the previous turn rather than reorganizing historical messages. When truncating history, truncating from a different position each time causes the prefix to change repeatedly. A safer approach is to define a fixed sliding-window strategy and trim at a fixed step size where possible.

    3.4 Share Prefixes Across Batch Tasks

    When asking multiple questions about the same document, those requests should share the same document prefix and append different questions in sequence. Concurrent requests sent before the cache is established may also fail to hit. Therefore, you can send one warm-up request first, then send the remaining requests.

    4. Monitoring and Cost Accounting

    To optimize continuously, treat hit rate as a core metric. It is recommended to record the following data at the gateway or invocation layer:

    • Total input tokens per request;
    • Cached tokens and uncached tokens among them;
    • Effective unit price aggregated by day, model, and business line;
    • Periodic changes in hit rate, to detect drops caused by prompt changes or releases.

    Most providers’ response usage information includes token statistics for cached and uncached tokens. The exact field names depend on the API documentation in use. Below is a generic cost-accounting example using placeholder field names:

    1. def effective_input_price(standard_price: float, cached_price: float, hit_rate: float) -> float:
    2. """Calculate the effective price per million input tokens in the same unit as the input prices."""
    3. if not 0.0 <= hit_rate <= 1.0:
    4. raise ValueError("hit_rate must be between 0 and 1")
    5. return hit_rate * cached_price + (1.0 - hit_rate) * standard_price
    6. def hit_rate_from_usage(cached_tokens: int, total_input_tokens: int) -> float:
    7. """Calculate the hit rate from usage statistics."""
    8. if total_input_tokens <= 0:
    9. return 0.0
    10. return cached_tokens / total_input_tokens
    11. if __name__ == "__main__":
    12. # Hypothetical prices and usage, for demonstration only
    13. standard, cached = 10.0, 2.0
    14. cached_tokens, total_tokens = 900_000, 1_000_000
    15. h = hit_rate_from_usage(cached_tokens, total_tokens)
    16. price = effective_input_price(standard, cached, h)
    17. print(f"Hit rate: {h:.0%}, effective price: {price:.2f} currency units per million tokens")

    The output is a 90% hit rate and an effective price of 2.80 currency units per million tokens. After integrating this logic into a monitoring system, teams can use the same metrics to compare the cost differences between prompt versions.

    5. Common Pitfalls

    Pitfall 1: Assuming caching depends only on the model. In practice, cache effectiveness is mainly determined by request structure. With the same model, different prompt organization can lead to hit rates that differ by several times.

    Pitfall 2: Looking only at the average hit rate. An average can hide long-tail problems. Some business lines or request types may have very low hit rates and need to be observed separately by dimension.

    Pitfall 3: Ignoring cache expiration. Cache entries usually have a time to live. For sessions with no requests for a long time, the prefix may have expired. For low-frequency workloads, evaluate the relationship between hit rate and request interval.

    Pitfall 4: Pursuing hit rate at the expense of quality. Removing necessary context just to keep the prefix fixed may reduce answer quality. Cost optimization should be performed only when quality remains acceptable.

    6. Implementation Recommendations

    1. Measure the current state first. Record one week of hit rate and effective unit price without any changes to establish a baseline.
    2. Adjust prompt structure. Put fixed content first and dynamic content last, and compare hit rates before and after changes.
    3. Standardize conversation management. Unify the strategy for concatenating and truncating historical messages to avoid irregular context reorganization.
    4. Set up alerts. Notify the responsible owner when the hit rate falls below a defined threshold, preventing silent cost increases after releases.
    5. Review regularly. Re-evaluate model selection and calling strategy based on changes in business volume and unit prices.

    7. Conclusion

    Input cost is determined jointly by the standard unit price, the cached price, and the hit rate. In most continuous invocation scenarios, hit rate is a variable that can be significantly improved through engineering. Teams should include hit rate in daily metrics, use effective unit price rather than standard unit price for budgeting and model selection, and maintain a high hit rate through stable request structures.