How to Accurately Estimate LLM Inference Costs Based on Cache Hit Rate
Sticker prices like "per million tokens" rarely reflect what you actually pay for large model inference. With context caching, input tokens are split into cache-hit and cache-miss portions, and input and output prices usually differ. This article walks through how to reconstruct your real token composition, build a general cost formula, identify the cache hit rate threshold where pricing conclusions reverse, evaluate subscription plans fairly, and apply engineering practices to reduce actual costs.
When deploying large language model applications, the “per million tokens” price on a rate card often does not directly represent your true cost. For models with context caching capabilities, input tokens may be split into cache-hit and cache-miss portions; meanwhile, input and output token prices usually differ, and subscription plans may also be affected by peak hours, call quotas, and rate limits.
Therefore, when comparing different models or service plans, you should not look only at the cached input price, nor simply compare a single token price across services. A more reliable approach is to first reconstruct your application’s token composition, then calculate costs based on cache hit rate, input/output ratio, and actual usage patterns.
1. The Three Types of Tokens in an LLM Call
A typical model request can be broken down into three billing categories:
Cache-hit input tokens
These tokens have already appeared in previous requests. The server can reuse existing computation results, so the price is typically lower than for regular input tokens.Cache-miss input tokens
These are context tokens the model must process fresh—for example, new questions, dynamic parameters, user input, and historical information that cannot be reused.Output tokens
These are the model’s generated responses, code, structured data, or tool call parameters. Output tokens are often priced higher than input tokens, so in long-form text generation, code generation, and multi-turn reasoning scenarios, the output share cannot be ignored.
Suppose the total token count in a billing period is (T), the cache hit rate is (r), the cache-miss input share is (i), and the output share is (o). These must satisfy:
[
r+i+o=1
]
If the cache hit rate is 98%, the cache-miss input share is 1.5%, and the output share is 0.5%, then for every 100 million tokens:
- Cache-hit tokens: 98 million;
- Cache-miss input tokens: 1.5 million;
- Output tokens: 0.5 million.
Each of these three portions must be multiplied by its corresponding price; only then does the result reflect actual spending.
2. A General Cost Formula
Assume:
- Cached input price is (P_c), in currency per million tokens;
- Regular input price is (P_i), in currency per million tokens;
- Output price is (P_o), in currency per million tokens;
- Total usage is (T), in tokens;
- Cache hit rate is (r);
- Regular input share is (i);
- Output share is (o).
The total cost is then:
[
C=\frac{T}{1,000,000}\times(rP_c+iP_i+oP_o)
]
For a volume of 100 million tokens, the formula simplifies to:
[
C=100\times(rP_c+iP_i+oP_o)
]
The value of this formula is that it converts “advertised prices” into “actual business prices.” As long as the three prices and the token composition are known, any model service can be compared on the same basis.
Note that some services use different billing conventions—for example, counting input and output separately, tracking cache hits separately, imposing minimum charges per request, or applying tiered pricing for very long contexts. Always defer to the service’s official billing documentation before integrating.
3. When Cache Hit Rate Is High, Lower Cached Input Prices Win
In long-context Q&A, code repository analysis, and enterprise knowledge base retrieval, systems typically send the same system prompts, rules, project background, and conversation history repeatedly. Cache hit rates in these scenarios can be high.
Take a 98% hit rate as an example, with the token composition being:
- 98% cache-hit input;
- 1.5% regular input;
- 0.5% output.
Using the pricing from the original example, one plan costs roughly 12.57 (in the reference currency) per 100 million tokens, while another costs about 9.40. This shows that in high cache-hit scenarios, the cached input price significantly influences the final result. Even if regular input and output prices are not fully competitive, as long as cached tokens make up the vast majority of usage, differences in the cached price get amplified.
Typical characteristics of such workloads include:
- Long, fixed system prompts;
- The same project specifications attached to every request;
- Follow-up questions about the same document;
- Multi-turn code edits based on the same context;
- RAG applications where fixed templates and shared knowledge content make up a large share of usage.
However, a high hit rate is not automatic. Applications that frequently reorder prompt content, insert random dynamic elements, or rebuild the context on every request may rarely hit the cache. When evaluating, use statistics from production traffic rather than theoretical hit rates.
4. When Cache Hit Rate Drops, the Conclusion Can Flip
If the cache hit rate falls from 98% to 95%, it seems like only a 3 percentage point drop—but those tokens shift from low-priced cached input to higher-priced regular input, and cumulative costs can rise noticeably.
Based on the original example, at a 95% hit rate:
- One plan costs about 14.92 per 100 million tokens;
- The other costs about 18.25 per 100 million tokens.
At this point, the former becomes the more cost-effective option.
This shows that comparing model pricing is not simply “whoever has the lower cache price is cheaper”—it is a threshold problem tied to the cache hit rate. You can further calculate the critical hit rate: the point at which the total costs of the two plans are equal is a key reference for model selection.
Given two plans with regular input and output prices (P{i1},P{o1}) and (P{i2},P{o2}), and cached input prices (P{c1},P{c2}), solve the following equation for the critical cache rate:
[
rP{c1}+(1-r-o)P{i1}+oP{o1}
=
rP{c2}+(1-r-o)P{i2}+oP{o2}
]
The resulting (r) is the cost crossover point. When your actual cache hit rate is above this value, plan one is likely more cost-effective; below it, plan two wins.
5. Don’t Ignore the Cost of Output Tokens
Many teams focus only on input tokens when budgeting, because input usually accounts for the majority of total volume. But in the following scenarios, output cost can become the dominant variable:
- Generating long code automatically;
- Generating test cases and technical documentation;
- Multi-step reasoning;
- Batch generation of structured data;
- Returning many candidate results in a single request;
- Continuous code edits and explanations in AI coding workflows.
For example, with the same 100 million total tokens, if the output share rises from 0.5% to 10%, the cost structure changes dramatically. A plan with a higher output price can quickly lose its advantage.
Therefore, maintain at least three sets of business metrics:
- Average input tokens;
- Average output tokens;
- Cache-hit input tokens.
Don’t just record the total token count. Two workloads with identical totals can differ several times in cost because of different input/output ratios.
6. Monthly Subscription Plans Are Not the Same as Free Quotas
Beyond pay-as-you-go billing, many model services offer monthly subscription plans for coding or API usage. In the original example, the plan tiers were priced at 118, 538, and 1078 (in the reference currency), with weekly token quotas based on off-peak usage.
The approximate weekly token quotas are:
- Entry tier: 317 million tokens per week;
- Professional tier: 1.9 billion tokens per week;
- Premium tier: 4.433 billion tokens per week.
Converted at pay-as-you-go prices, these quotas correspond to an equivalent value of roughly 160, 955, and 2229 respectively. In other words, the plan prices work out to roughly 50% to 70% of the equivalent pay-as-you-go cost.
But this does not mean a plan suits every user. To judge whether a plan is worthwhile, consider the following factors:
1. Can you consistently use the full quota?
If your actual usage is only 20% of the plan’s quota, your overall spend may still exceed pay-as-you-go pricing even if the plan’s unit price looks low. The core value of a subscription is not “low unit price” but “stable, sufficient usage.”
2. Are token quotas time-limited?
Some plans allocate quotas weekly, daily, or per billing cycle, and unused portions may not roll over. For teams with volatile usage, wasted quotas erode the plan’s value.
3. Do peak hours affect availability?
In the example, peak hours are 14:00–18:00 on weekdays—4 hours per day. This window is relatively limited, but if your team mainly works during office hours, you still need to confirm whether peak periods involve throttling, reduced quotas, or slower responses.
4. Is the plan’s quota tied to specific models?
Some plans only apply to specific models, features, or clients. When converting plan quotas into monetary equivalents, make sure the covered scope matches your actual usage—don’t treat non-transferable quotas as cash value.
7. Build a Cost Model for Your Organization
Before launch, teams can set up a simple monthly cost model:
[
C{month}=N\times T{avg}\times P_{effective}
]
where:
- (N) is the number of monthly requests;
- (T_{avg}) is the average token count per request;
- (P_{effective}) is the blended effective price per token.
For better accuracy, segment users into categories such as:
- Casual Q&A users;
- High-frequency coding users;
- Long-context analysis users;
- Automated batch jobs;
- Admin or test accounts.
Calculate request volume, input/output ratio, and cache hit rate for each segment, then aggregate into an overall budget. This is far more reliable than a generic “average tokens per user per day” estimate.
It’s also wise to model three scenarios:
- Conservative: low cache hit rate, long outputs;
- Baseline: based on current production data;
- Optimistic: high cache hit rate, constrained outputs.
If monthly costs vary widely across these scenarios, your system is sensitive to prompt structure, caching strategy, or output length—and engineering optimization should be prioritized.
8. Engineering Practices That Reduce Real Costs
Cost optimization shouldn’t rely solely on switching models or plans—you can also work on the call chain itself:
Put stable content first
Place consistent system rules, project specifications, and shared background at the front of the context to reduce cache invalidation.Put dynamic content last
Place user questions, timestamps, and real-time data at the end, so small changes don’t invalidate the entire context.Constrain unnecessary output
Reasonably limit response length, format, and maximum output tokens to prevent the model from generating irrelevant explanations.Tier tasks by difficulty
Use low-cost models for simple classification, summarization, and format conversion; reserve high-capability models for complex reasoning and code changes.Log real usage
At the gateway level, record request tokens, cache-hit tokens, input tokens, output tokens, response times, and error retry counts.Reduce duplicate retries
Network timeouts, parameter errors, and duplicate client submissions can all trigger extra billing—add idempotency controls at the call layer.
Conclusion
The key to understanding true LLM costs is not comparing any single price in isolation, but analyzing the complete token structure. When cache hit rates are high, cached input prices become the dominant factor; once hit rates fall, regular input and output prices quickly matter more. For monthly plans, you must weigh actual usage, quota validity windows, and peak-hour restrictions together.
The most reliable approach is: first gather production traffic statistics, then build a model based on cache hit rate, input/output ratio, and monthly request volume, and finally compare pay-as-you-go and subscription options in the same cost table. Only then can you avoid being misled by a single price metric and establish a solid basis for model selection, budgeting, and system optimization.
