LLM API Cache Hit Rate and Real Input Cost

    This article explains why estimating LLM API costs with standard input pricing alone can be misleading. It introduces a cache hit rate–based input cost formula, walks through example calculations and sensitivity analysis, and outlines engineering practices for improving cache hit rates and building a monitorable cost model.

    LLM API Cache Hit Rate and Real Input Cost

    A common mistake in LLM API cost estimation is to multiply the standard input price by call volume. This looks simple, but it ignores a key variable: cache hit rate. For requests with reusable prefix content, the cached portion is usually billed at a lower cache price, while only the uncached portion is billed at the standard price. When the hit rate rises from 0% to 90%, the effective input price can differ by several times. Real cost evaluation therefore needs to move from “list-price thinking” to “billing-structure thinking.”

    Why Cache Hit Rate Affects the Effective Price

    In LLM inference scenarios, a request is usually composed of system prompts, tool descriptions, few-shot examples, conversation history, user questions, and other parts. If multiple requests share the same or highly stable prefix content, the service side can reuse part of the computation through general techniques such as prefix caching, reducing repeated computation overhead. For the user, this portion may be billed at the cache price; content that misses the cache is billed at the standard input price.

    Cache hit rate can be understood as the proportion of billable input tokens that hit the cache and receive cache pricing. It is not a fixed value. It fluctuates with prompt structure, request reuse patterns, cache validity period, concurrent access, routing strategy, and other factors. As a result, APIs with the same list price can have noticeably different actual costs under different business call patterns.

    It is important to note that the specific implementation of caching, hit conditions, validity period, and billing rules should follow the actual service documentation. This article uses a set of example price parameters to explain the cost calculation logic; these do not represent any specific service’s quote or commitment.

    General Billing Model and Example Calculation

    Let the standard input price be (P_s) CNY per million tokens, the cache price be (P_c) CNY per million tokens, the cache hit rate be (H), and the input token volume be (T) in millions of tokens. The actual input cost can be expressed as:

    [
    Cost = T \times [H \times P_c + (1-H) \times P_s]
    ]

    Using example parameters: standard input price CNY 4 per million tokens, cache price CNY 0.7 per million tokens, hit rate 90%, and input volume 1 million tokens:

    • 90% cached portion: (0.9 \times 0.7 = 0.63) CNY;
    • 10% uncached portion: (0.1 \times 4 = 0.4) CNY;
    • Actual input cost: (0.63 + 0.4 = 1.03) CNY.

    Compared with the standard price of CNY 4, CNY 1.03 is about 25.75% of the standard price, close to 25%. In other words, a 90% cache hit rate makes the actual input cost far lower than the surface list price. If the hit rate is 0%, the cost is CNY 4; if the hit rate is 90%, the cost is about CNY 1.03, a difference of about 3.88 times. The higher the hit rate, the more the gap between cache price and standard price translates into an actual cost advantage.

    Cost Sensitivity to Hit Rate Fluctuations

    In real business scenarios, the hit rate is often not stable and fluctuates within a range. Still using the example parameters above:

    • Hit rate 85%: cost = (0.85 \times 0.7 + 0.15 \times 4 = 0.595 + 0.6 = 1.195) CNY;
    • Hit rate 95%: cost = (0.95 \times 0.7 + 0.05 \times 4 = 0.665 + 0.2 = 0.865) CNY.

    As the hit rate rises from 85% to 95%, the actual cost drops from about CNY 1.20 to about CNY 0.87, a decrease of about 28%. If call volume is large, this difference is amplified. Cost monitoring should therefore not only record total tokens, but also record the distribution of cached and uncached tokens, broken down by day, business line, and prompt template version.

    Below is a cost calculation example that does not depend on a specific API:

    1. def estimate_input_cost(tokens_m, hit_rate, standard_price, cache_price):
    2. hit_tokens = tokens_m * hit_rate
    3. miss_tokens = tokens_m * (1 - hit_rate)
    4. cost = hit_tokens * cache_price + miss_tokens * standard_price
    5. standard_cost = tokens_m * standard_price
    6. return {
    7. "actual_cost": round(cost, 4),
    8. "standard_cost": round(standard_cost, 4),
    9. "discount_ratio": round(cost / standard_cost, 4)
    10. }
    11. result = estimate_input_cost(
    12. tokens_m=1,
    13. hit_rate=0.9,
    14. standard_price=4.0,
    15. cache_price=0.7
    16. )
    17. print(result)
    18. # actual_cost: 1.03, standard_cost: 4.0, discount_ratio: 0.2575

    The point of this code is to bring the hit rate into the budget model. Engineering teams can periodically regression-test based on actual token distributions in logs, instead of relying only on static estimates using the standard unit price.

    Engineering Practices for Improving Cache Hit Rate

    The core principle for improving hit rate is to “keep stable content stably reusable and push changing content as late as possible.” The following practices can help:

    1. Stabilize the prompt prefix: Place relatively fixed content such as system roles, output constraints, tool descriptions, and few-shot examples at the front of the request, and keep wording, order, and format stable. Avoid inserting high-frequency changing fields such as timestamps, random IDs, or user nicknames into the prefix.

    2. Manage template versions: Prompt templates should have explicit version numbers. Even small template changes can alter the prefix and reduce the hit rate. Before release, evaluate the impact of template changes on cache reuse and keep a rollback mechanism.

    3. Place changing content later: Put user questions, real-time retrieval results, dynamic context, and other changing content after the stable prefix. This makes the shared prefix easier to reuse, and even if the dynamic portion misses the cache, it will not break the overall prefix structure.

    4. Reuse within batches and sessions: Consecutive requests in the same business scenario often share a large amount of context. Organizing sessions reasonably and reducing meaningless repeated calls can help improve reuse probability. At the same time, pay attention to cache validity periods and context length boundaries.

    5. Evaluate multimodal and long-context scenarios separately: Different modalities such as text, images, and documents may have different cacheable segments and matching methods. For long-context scenarios, analyze which content can be reused and which content changes on every call, and avoid mixing dynamic content into stable prefixes.

    6. Monitoring and alerting: Record fields such as request template version, total input tokens, cached tokens, uncached tokens, and estimated cost in logs, and build cost dashboards by business line. Trigger alerts when the hit rate drops abnormally, and investigate template changes, traffic structure changes, or cache expiration policies.

    Key Principles of Cost Governance

    First, the budget model should use “actual input cost” rather than “standard unit price.” Second, hit rate should be treated as a cost metric as important as call volume and output length. Third, optimizing hit rate must not come at the expense of answer quality; stable prefixes must be evaluated for effectiveness. Fourth, example price parameters are only used to explain the calculation logic; actual quotes and billing rules should follow the service provider’s documentation. Fifth, cost optimization should form a closed loop: monitor hit rate, analyze the causes of misses, adjust prompt structure, run regression tests, and monitor again.

    Conclusion

    Cache hit rate is a key bridge from “list price” to “real cost.” For scenarios with large amounts of stable prefix content, such as high-frequency templates, customer support Q&A, coding assistance, and document summarization, improving the hit rate often brings significant cost benefits. The example of an actual input price at about 25% of the standard price under a 90% hit rate shows that LLM API cost evaluation must consider standard price, cache price, hit rate, and business call structure together. Only by building a formula-based, monitorable, and regression-testable cost model can teams avoid being misled by surface list prices and make more accurate capacity planning and engineering optimization decisions.