LLM API Cache Hit Rate and Real Cost Calculation Methods
This article presents a general method for estimating LLM API costs using cache hit rate, non-cached input share, output share, and unit prices. It explains why comparing cache-read pricing alone can be misleading, provides a weighted cost formula and sensitivity analysis example, and discusses package utilization, peak and off-peak constraints, retries, context growth, and other hidden costs. Readers can apply the model to compare providers and plans under their own traffic mix.
In LLM inference service selection and capacity planning, many teams first look at the “cache-hit price” or the “price per million tokens” and then use that to judge overall cost. In real production environments, however, the final bill is usually determined by multiple variables: cache hit rate, non-cached input share, output share, peak and off-peak periods, request retries, context growth, package quota utilization, and more. Comparing a single unit price can easily lead to one-sided conclusions.
This article does not target any specific product. Instead, it provides a general cost calculation method. Readers can plug publicly available billing items from a provider into the model to obtain an explainable and verifiable estimate.
Why You Cannot Look Only at Cache Unit Price
LLM inference billing usually does not include only “cache reads.” A more common billing structure includes:
- Cache-hit read price: the cost incurred when cached context or prefixes are reused;
- Non-cached input price: the cost of input tokens that do not hit the cache;
- Output price: the cost of model-generated content;
- Cache write or storage fees: some services may charge for cache creation or retention time;
- Peak and off-peak differences: unit prices may differ between peak and off-peak periods;
- Package versus pay-as-you-go differences: packages may have monthly quotas, time restrictions, concurrency limits, or overage rules.
Therefore, real cost should be expressed as a weighted form. Let total token volume be (T), cache hit share be (h), non-cached input share be (u), and output share be (o), with (h+u+o=1). Let the corresponding unit prices be (p_c), (p_i), and (p_o). The pay-as-you-go cost is:
[
C = T \times (h \cdot p_c + u \cdot p_i + o \cdot p_o)
]
This formula shows that a low cache unit price affects only the first term. If the output share is not low and the output price is significantly higher than the cache price, total cost may still be dominated by the output term.
Cache Hit Rate Is a Core Sensitivity Variable
When the cache hit rate is high, cost is mainly determined by the cache-read price. When the hit rate falls, the non-cached input term rises quickly. Assume a service provider has the following illustrative prices, used only to demonstrate the method and not representing any actual product:
- Cache read: CNY 0.1 per million tokens;
- Non-cached input: CNY 1 per million tokens;
- Output: CNY 4 per million tokens.
Take 100 million tokens, or 100 one-million-token units. If the cache hit rate is 98%, non-cached input is 1.5%, and output is 0.5%, then:
[
C = 100 \times (0.98 \times 0.1 + 0.015 \times 1 + 0.005 \times 4) = 13.3 \text{ CNY}
]
If the cache hit rate drops to 95%, non-cached input rises to 4.5%, and output remains 0.5%, then:
[
C = 100 \times (0.95 \times 0.1 + 0.045 \times 1 + 0.005 \times 4) = 16 \text{ CNY}
]
As shown, when the hit rate falls from 98% to 95%, total cost rises by about 20%. If different providers have different ratios between non-cached input price and output price, the cost ranking may even reverse. Therefore, evaluation must collect the hit rate, input-output mix, and each unit price together rather than comparing cache prices alone.
Using Code for Sensitivity Analysis
Below is an example that does not depend on third-party libraries and can calculate the cost per 100 million tokens under different hit rates. In actual use, replace the illustrative prices with the provider’s publicly available prices.
def cost_per_100m(hit_ratio, miss_input_ratio, output_ratio,cache_price, input_price, output_price):total_million = 100.0 # 100 million tokens = 100 million-token unitscost = total_million * (hit_ratio * cache_price +miss_input_ratio * input_price +output_ratio * output_price)return costcache_price = 0.1input_price = 1.0output_price = 4.0for hit in [0.98, 0.95, 0.90, 0.80]:miss = 1.0 - hit - 0.005cost = cost_per_100m(hit, miss, 0.005,cache_price, input_price, output_price)print(f"Hit rate={hit:.0%}, cost per 100M tokens={cost:.2f} CNY")
The value of this code is that it quickly answers two questions. First, is the impact of hit-rate changes on cost significant? Second, within what hit-rate range does a given billing structure have an advantage? It is recommended to include the output ratio in the scan as well, because output prices are usually higher and fluctuations in output length can amplify bill differences.
Package Costs Should Be Converted Based on Usable Capacity
Package prices cannot simply be divided by pay-as-you-go unit prices. A more reasonable approach is to calculate the package’s effective pay-as-you-go equivalent cost:
[
\text{Package equivalent cost} = \frac{\text{monthly package fee}}{\text{actually consumable token volume} \times \text{weighted unit price}}
]
If a package has peak-period restrictions, such as rate limits or price limits only during fixed weekday periods, the usable capacity during peak and off-peak periods must be further separated. Whether a package is cost-effective depends on three conditions:
- Whether the quota can be used up consistently;
- Whether business traffic falls within periods that can be used efficiently;
- Whether rate limits, concurrency limits, and overage rules cause queuing or failed retries.
If team traffic fluctuates heavily, a package may create a situation where “low-price quota is not fully used, while peak periods are restricted,” making the real unit cost higher than pay-as-you-go. If traffic is stable and can be scheduled to off-peak periods, the converted unit price of a package is often more favorable, but it must be verified against actual utilization.
Real Cost Should Also Include Hidden Items
In addition to the three terms in the formula, production environments must also consider:
- Retry cost: failed requests and timeout retries can be billed repeatedly;
- Context growth: multi-turn conversations continuously append history, so input tokens keep growing;
- Cache invalidation: prefix changes, inserted tool results, timestamps, and similar factors can reduce hit rates;
- Output length control: summaries, structured output, and maximum generation length limits can significantly reduce cost;
- Monitoring and logging: request logs, audit records, and metric storage are also part of total cost of ownership;
- Concurrency rate limits: business waiting caused by queuing is not directly billed, but it affects throughput and experience.
Practical Recommendations
It is advisable to build a cost dashboard that includes at least: cache hit rate, non-cached input share, output share, token volume per 10,000 requests, peak and off-peak request distribution, package utilization, and retry rate. Recalculate the weighted unit price every week based on the real distribution, and perform sensitivity analysis on key variables. During selection, do not ask only “How much does the cache cost?” Instead, ask: “Under my hit rate and input-output structure, what is the total cost per 100 million tokens? Can the package be fully used? Do the peak and off-peak rules match the business?”
Only when the billing formula, traffic distribution, and package utilization are all included in the evaluation does cost comparison have engineering meaning.
