Cache Hit Rate: The Real Key to LLM API Costs
When evaluating LLM API costs, many teams rely solely on the standard input price, overlooking a critical billing variable: the cache hit rate. This article explains, from the perspectives of billing models, sensitivity analysis, engineering practices, and cost monitoring, why effective input cost depends on the share of tokens billed at the lower cached rate, and shows how to build a more accurate cost model for real workloads.
Cache Hit Rate: The Real Key to LLM API Costs
When evaluating the cost of LLM APIs, many teams simply look at the “standard input price” and use it to estimate their monthly budget. While this approach seems intuitive, it overlooks a critical variable in the billing logic: the cache hit rate. For API services that support context caching or prefix caching, input tokens are not always billed at the same price. Tokens that hit the cache are typically billed at a lower cache-read price, while only the non-cached portion is billed at the standard input price. In other words, the standard price is only “the price when nothing is cached”—it is not the effective input price under real workloads.
This article explains how to evaluate the actual cost of LLM APIs more accurately, covering four aspects: the billing model, sensitivity analysis, engineering practices, and cost monitoring. The prices used here are for demonstrating the calculation logic only; they do not represent the pricing or billing rules of any specific platform. Different services may differ in cache-write pricing, cache-read pricing, minimum cacheable length, cache lifetime, and measurement methodology. Actual evaluations should be based on the relevant API documentation and billing data.
1. Why Looking Only at the Standard Price Misleads Cost Estimates
The standard price typically corresponds to the scenario where input tokens do not hit the cache. If business requests are highly repetitive—for example, a fixed system prompt, a stable tool-calling schema, a long retrieval template, or reused conversation history across multiple turns—a large share of input tokens may hit the cache. In that case, the real cost consists of two parts: cached tokens billed at the cache price, and uncached tokens billed at the standard price.
The problem with looking only at the standard price is that it assumes all input tokens are billed at the highest rate, which tends to overestimate costs. Conversely, if a team assumes every request will achieve a high hit rate, it may underestimate the costs caused by cold starts, cache expiration, and dynamic contexts. Therefore, the core of cost evaluation is not memorizing a single price, but establishing the computational relationship among hit rate, pricing, and request structure.
2. The Calculation Model for Effective Input Price
Let the standard input price be (P{std}), the cache-read price be (P{cache}), and the cache hit rate be (h), where (0 \le h \le 1). The effective input price per million tokens can be expressed as:
[
P{eff} = h \times P{cache} + (1-h) \times P_{std}
]
The discount ratio relative to the standard price is:
[
D = \frac{P{eff}}{P{std}} = h \times r + (1-h)
]
where (r = P{cache}/P{std}) is the ratio of the cache price to the standard price. Simplifying gives:
[
D = 1 - (1-r)h
]
This formula reveals two facts. First, the higher the hit rate, the lower the effective price. Second, the lower the cache price relative to the standard price, the greater the leverage provided by the hit rate. If the cache price is close to the standard price, even a very high hit rate yields limited savings; if the cache price is significantly lower, improvements in hit rate rapidly reduce the effective cost.
3. The Discount Effect of a 90% Hit Rate with Illustrative Parameters
Suppose the standard input price of a certain type of LLM API is 10 CNY per million tokens, and the cache-read price is 17.5% of the standard price, i.e., 1.75 CNY per million tokens. Further suppose a workload has a 90% cache hit rate. The effective cost per million input tokens is then:
- 90% cached: (0.9 \times 1.75 = 1.575) CNY;
- 10% uncached: (0.1 \times 10 = 1) CNY;
- Total effective input cost: (1.575 + 1 = 2.575) CNY.
Compared to the standard price of 10 CNY, 2.575 CNY is about 25.75% of the standard price—roughly a 74% discount. This is not because the standard price changed, but because most input tokens are billed at the cache price. Budgeting purely at 10 CNY per million tokens would overestimate the cost by about 3.88x; conversely, if the actual hit rate is below 90%, the effective cost rises.
It is important to emphasize that 10 CNY, 1.75 CNY, and 90% are all illustrative parameters meant to demonstrate the framework. Real evaluations should substitute the actual prices, actual hit rates, and billing methodology of the corresponding service.
4. Hit Rate Sensitivity: How Big Is the Gap from 0% to 90%?
The cache hit rate is not a minor linear factor—it is a high-leverage variable in the cost model. Using the same illustrative prices:
| Cache Hit Rate | Effective Input Price (CNY/million tokens) | Relative to Standard Price |
|---|---|---|
| 0% | 10.000 | 100% |
| 30% | 7.525 | 75.25% |
| 50% | 5.875 | 58.75% |
| 70% | 4.225 | 42.25% |
| 90% | 2.575 | 25.75% |
| 95% | 2.1625 | 21.625% |
As shown, the difference between 0% and 90% hit rates produces roughly a 3.88x difference in effective input price. If the ratio of cache price to standard price is even lower—for example, if the cache price is 10% to 20% of the standard price—the gap between the standard and effective prices near a 90% hit rate can typically reach about 3.5x to 5x. This is exactly why the “standard price” and the “actual input cost” can diverge so easily.
Looking further, when the hit rate rises from 0% to 50%, the cost is nearly halved; from 50% to 90%, it continues to drop substantially. For workloads with a high proportion of input tokens, long contexts, and obvious template reuse, optimizing the hit rate is often more valuable than simply comparing standard prices.
5. Engineering Factors That Affect the Cache Hit Rate
The cache hit rate is not determined solely by the API provider—it is closely tied to request structure, context organization, and traffic patterns. The following factors deserve attention in engineering practice.
First, stable prefixes. The system prompt, role definitions, output format constraints, tool-calling schemas, and similar content should remain as stable as possible and be placed at the beginning of the request. Frequently changing dynamic fields placed inside the prefix can prevent cache reuse.
Second, avoid meaningless timestamps and random IDs. Some requests insert the current time, random numbers, or request IDs at the start of the system prompt or template, which breaks prefix stability. If the business genuinely needs this information, consider placing it toward the end of the context or passing it through structured fields, so the cacheable portion is not affected.
Third, templated retrieval and context assembly. In retrieval-augmented generation (RAG) scenarios, the retrieval template, ordering rules, and truncation strategy should remain stable. If the concatenation order, delimiters, and metadata formats differ on every request, cache reuse will decline. It is recommended to organize fixed instructions and dynamic retrieval results in separate layers.
Fourth, session reuse and context management. In multi-turn conversations, a higher hit rate usually results when historical context can be stably reused; however, appending history indefinitely leads to token bloat. Context windows, summarization strategies, and expiration policies should be designed around the business to balance hit rate against total token volume.
Fifth, pay attention to cache lifetime and cold starts. Caches typically have constraints such as expiration, minimum cacheable granularity, or write costs. Low-frequency requests, bursty traffic, and long-interval sessions tend to have lower hit rates. Cost evaluation should distinguish hot requests from cold requests rather than relying only on averages.
Sixth, distinguish measurement methodologies. Is the hit rate measured per token or per request? Are cache writes billed separately? Is the cache-read price tiered? All of these affect the final bill. Engineering monitoring should record input tokens, cached tokens, uncached tokens, and output tokens simultaneously, and avoid extrapolating token-level costs directly from request-level hit rates.
6. A Practical Checklist for Including Cache Hit Rate in Cost Evaluation
To upgrade from “looking at the unit price” to “looking at effective cost,” it is recommended to establish the following checklist in budget evaluation and production monitoring:
- Clarify billing methodology: standard input price, cache-read price, cache-write price, output price, minimum billing unit, and tiered pricing rules.
- Collect real distributions: measure input token lengths, the proportion of cached tokens, request frequency, and session intervals by business scenario.
- Estimate by scenario: customer service, knowledge-base Q&A, code assistance, and document summarization differ in repetitiveness and context structure, and should be estimated separately.
- Perform sensitivity analysis: compute effective prices and monthly costs at hit rates of at least 0%, 50%, 70%, and 90%.
- Set up cost monitoring: include cache hit rate, effective input price, average cost per request, and budget consumption in routine monitoring.
- Recalibrate regularly: after changes to business templates, model versions, or context strategies, re-evaluate hit rates and billing.
7. Pseudocode Example: Pre-Request Estimation and Post-Request Monitoring
The following platform-agnostic pseudocode demonstrates effective input cost estimation. All prices are in “CNY per million tokens” and are for logical illustration only.
def effective_input_price(hit_rate, standard_price_per_m, cache_price_per_m):if not 0 <= hit_rate <= 1:raise ValueError("hit_rate must be between 0 and 1")return hit_rate * cache_price_per_m + (1 - hit_rate) * standard_price_per_mdef discount_ratio(hit_rate, standard_price_per_m, cache_price_per_m):effective = effective_input_price(hit_rate, standard_price_per_m, cache_price_per_m)return effective / standard_price_per_m# Illustrative parameters; do not represent any specific platform's pricingstd_price = 10.0cache_price = 1.75hit_rate = 0.90eff = effective_input_price(hit_rate, std_price, cache_price)ratio = discount_ratio(hit_rate, std_price, cache_price)print(eff) # 2.575print(ratio) # 0.2575
Post-request monitoring can compute the actual input cost as follows:
def actual_input_cost(input_tokens, cache_hit_tokens,standard_price_per_m, cache_price_per_m):if cache_hit_tokens > input_tokens:raise ValueError("cache_hit_tokens cannot exceed input_tokens")miss_tokens = input_tokens - cache_hit_tokensreturn (cache_hit_tokens * cache_price_per_m / 1_000_000+ miss_tokens * standard_price_per_m / 1_000_000)def token_hit_rate(input_tokens, cache_hit_tokens):if input_tokens == 0:return 0.0return cache_hit_tokens / input_tokens
If the service charges cache-write fees, cache-storage fees, or output fees, these should be added separately to the total cost—the input-side formula alone cannot replace the complete billing model. The monitoring dashboard should display the token-level hit rate, effective input price, total cost, and per-business-unit cost simultaneously, so it is possible to judge whether an optimization truly delivers benefits.
8. Conclusion
LLM API cost evaluation should not stop at the standard price. The standard price is the price when nothing is cached, whereas the effective input price in a real business depends on the cache hit rate, the ratio between the cache price and the standard price, and whether the request structure enables stable reuse. In scenarios where the cache price is significantly lower than the standard price and the hit rate is high, the actual input cost may be only 20–30% of the standard price. Conversely, workloads with frequent cold starts, rapidly changing contexts, or fast cache expiration will see costs much closer to the standard price.
A more robust approach is: build a cost model with formulas, run sensitivity analyses using real business distributions, improve the stability of cacheable prefixes through engineering practices, and continuously calibrate hit rates against billing through monitoring. Only by incorporating the cache hit rate into cost evaluation can teams avoid the budgeting errors caused by “looking only at the standard price” and establish a sustainable cost-control mechanism for operating LLM applications at scale.
