How Cache Hit Rates Impact Actual Costs After LLM API Price Adjustments
After recent LLM API pricing adjustments, the cost of cache misses has increased significantly compared to cache hits. This article explores how cache hit rates affect actual API costs, explaining the mechanics behind prefix caching and providing a unified calculation model. It also offers practical strategies to improve cache hit rates, mitigate risks, and optimize your LLM application budget.
1. Changes in Example Pricing
The prices used in this article are based on example billing data from a large language model, measured in cost units per million tokens. These figures are used to demonstrate calculation methods; refer to the official billing page for actual prices.
| Tier | Old Version (Example) | New Version (Example) | Increase |
|---|---|---|---|
| Output | ~27 | 100 | ~3.7x |
| Input (Cache Miss) | ~6.5 | 20 | ~3.1x |
| Input (Cache Hit) | ~1.1 | 2 | ~1.8x |
The smallest increase is in the cache hit tier. The price gradient has also widened: the ratio of cache miss inputs to cache hit inputs rose from ~6x to 10x, and the ratio of outputs to cache hit inputs rose from ~25x to 50x. Vendors are concentrating the price hike pressure on low-reuse calls, while cache hit calls remain relatively cheaper.
Whether this signal is beneficial for users depends on the hit rate and the input/output ratio. You cannot draw a conclusion just by looking at the unit price table.
2. What Determines Cache Hits?
Disaggregated inference and prefix caching are often discussed together, but they serve different purposes.
Disaggregated inference splits model inference into two phases: the Prefill phase processes input and generates the KV Cache, while the Decode phase generates output tokens one by one. These phases are scheduled to different resource pools, improving throughput and making latency more stable. It primarily solves resource scheduling problems and does not directly cause cache hits.
Prefix caching is the direct source of cache hits. If the beginning of two requests are exactly the same, the system can reuse the already computed KV Cache and skip that part of the Prefill calculation. The two technologies are often deployed together, but they solve different problems.
The hit rate mainly depends on the following factors:
- Whether the prefix is character-by-character identical, including punctuation, spaces, and variable order.
- Cache capacity and eviction policy; caches may be evicted during peak times.
- Whether requests are routed to the node that holds the cache.
- Session continuity; whether multi-turn dialogues can reuse the same history.
In scenarios like programming agents with fixed long system prompts and stable routing, the hit rate can reach a high level. In random Q&A scenarios, it is usually much lower. Different vendors also have different statistical criteria for hit rates, so verify the methodology before quoting numbers.
3. Unified Cost Calculation
To compare the old and new versions, we uniformly use the following assumption:
- The input-to-output ratio is 20:1, meaning out of every 21 tokens, 20 are input and 1 is output.
- The comprehensive price is calculated using the following formula, in cost units per million tokens:
Comprehensive Price = (20 × Effective Input Price + 1 × Output Price) ÷ 21
Effective Input Price = Hit Price × Hit Rate + Miss Price × (1 − Hit Rate)
Based on this formula, the results for four scenarios are as follows:
| Version | Cache Hit Rate | Effective Input Price | Comprehensive Price |
|---|---|---|---|
| Old | 90% | 1.64 | ~2.85 |
| New | 90% | 3.80 | ~8.38 |
| Old | 20% | 5.42 | ~6.45 |
| New | 20% | 16.40 | ~20.38 |
Three takeaways can be drawn from these results:
First, at the same hit rate, the new version’s comprehensive price is about 2.9x (90% hit) to 3.2x (20% hit) that of the old version. Caching narrows the increase but does not lower the absolute cost.
Second, in the new version, increasing the hit rate from 20% to 90% drops the comprehensive price from ~20.4 to ~8.4, a reduction of about 59%. The hit rate is one of the biggest variables affecting cost.
Third, in the 90% hit rate scenario, output accounts for about 57% of the comprehensive cost (100 ÷ 176). Because output pricing is much higher than input pricing, controlling output length and avoiding invalid generation can also save significant costs.
It is worth noting that claims like “enterprise user costs are far lower than individual users” often stem from mixing comparisons of different hit rates and input/output ratios. When comparing different types of users, unify the assumptions first; otherwise, the results primarily reflect differences in assumptions.
4. Strategies to Improve Hit Rates
1. Fixed prefix, dynamic content at the end. Place stable content like system prompts, code guidelines, and few-shot examples at the very beginning. Put user queries and retrieved results at the end. Any change in punctuation, spaces, or variable order can break the prefix and cause a cache miss.
SYSTEM_PROMPT = """You are a senior Python backend engineer.1. Follow PEP8 guidelines.2. Functions must have type annotations."""def build_prompt(project_docs: str, chat_history: str, user_input: str) -> str:# Stable prefix: keep it exactly the same for every requestprefix = SYSTEM_PROMPT + project_docs + chat_history# Dynamic part: place at the endsuffix = f"\nLatest user question: {user_input}"return prefix + suffix
Note that cache granularity and prefix matching rules vary by vendor; refer to the official documentation for specifics.
2. Trim history instead of rewriting every turn. In multi-turn dialogues, the history grows longer. You can keep only the recent turns and compress earlier content into a summary. Since summarization itself changes the prefix, it should be updated at fixed intervals (e.g., every 10 turns) rather than rewriting every turn.
3. Maintain session affinity. Route requests from the same session to the same inference node or cache pool. Otherwise, even with identical prefixes, requests might hit nodes that do not have the cache.
4. Track hit rates by scenario. Many APIs return the number of hit tokens in their usage information. Refer to the official documentation for exact field names. Tracking hit rates by scenario allows you to determine if optimizations are effective and provides a basis for budgeting.
5. Risks and Considerations
Cache consistency. If your application implements external result caching, set expiration times. After updating knowledge bases or code repositories, proactively invalidate the corresponding cache to avoid returning outdated answers.
Privacy and data isolation. In multi-tenant shared caching, verify if the vendor isolates caches by tenant and how long the cache is stored. Do not put sensitive data in prefixes that might be reused across sessions unless the vendor explicitly guarantees isolation methods.
Budget for low hit rates. When cache misses occur, costs are calculated at the miss price, and response latency will be higher. Budgets should account for both high and low hit rate scenarios to avoid overspending if the hit rate drops.
6. Checklist
- Verify current pricing, cache tier definitions, and billing rules on the official billing page.
- Track recent input/output ratios and cache hit rates.
- Plug the data into the formula above to calculate the comprehensive cost per million tokens.
- Budget based on low hit rate scenarios and review periodically.
After a price adjustment, your actual cost depends on two main factors: how much output you generate, and how much prefix content can be reused.
