LLM API Pricing Explained: Cache Hit Rate and Its Cost Leverage
LLM API pricing is built from three tiers — uncached input, cache-hit input, and output — so comparing headline unit prices across model generations can easily lead to the wrong conclusion. This article uses a worked example to show how a new generation's output price is up nearly 4x while its cache-hit price is up less than 2x, and why the ratio between tiers has become the real signal. It then applies the formula for effective blended input price to show how a 90% cache hit rate drops the effective input cost from 20 to 3.8 CNY per 1M tokens. Two realistic scenarios (an enterprise agent versus an occasional individual developer) are then compared, revealing a roughly 3.6x difference in blended cost for the same model. Finally, the article lays out four practical optimization levers: raising cache hit rate, compressing output tokens, optimizing the input/output ratio, and building cost observability.
Introduction: Unit Prices Went Up Several Times — So Why Doesn’t the Bill Look That Bad?
Every few months a model vendor releases a new generation, and with it comes a price sheet that looks absurdly expensive at first glance. The usual reaction in the community is to compare the unit price of the old and the new generation directly and conclude that prices have gone up three or five times. But if you actually pay per call based on that reading, the bill usually differs noticeably from the intuition.
The reason is that modern LLM API pricing is not a single dimension. It is a composite structure built from input, output, and cache-hit tiers. Looking at just one of those list prices is a reliable way to reach the wrong conclusion.
Below, a mature previous-generation model is compared against a “new-generation model” as illustrative samples to break down this pricing logic systematically and derive a practical cost estimation method. (The prices used here are illustrative numbers meant to explain the methodology; they do not represent any specific product.)
1. The Three-Tier Billing Structure: Input, Output, and Cache
The inference cost of a large language model naturally splits into three kinds of tokens:
- Uncached input: the part of the prompt that must be fully processed on every request. Highest unit price.
- Cache-hit input: the prompt prefix that repeats across requests and can be reused. Very low unit price.
- Output: the tokens generated by the model. Usually the most expensive of the three tiers.
Using the illustrative samples in this article, the comparison looks like this:
| Tier | Previous generation (CNY / 1M tokens) | New generation (CNY / 1M tokens) | Increase |
|---|---|---|---|
| Output | 27 | 100 | ≈3.7× |
| Uncached input | 6.5 | 20 | ≈3.1× |
| Cache-hit input | 1.1 | 2 | ≈1.8× |
One detail is easy to overlook: the cache-hit tier rose the most gently (under 2x), while the output tier rose the most sharply (close to 4x). This reveals the vendor’s pricing intent — to encourage callers to reuse the cache as much as possible, while concentrating cost pressure on the side that “actually does the work,” namely output.
2. Shifts in the Ratio Between Tiers Are the Real Signal
What explains the situation better than absolute unit prices is the ratio relationship between tiers:
- Previous generation: uncached input / cache-hit input ≈ 6x; output / cache-hit input ≈ 25x.
- New generation: uncached input / cache-hit input widens to 10x; output / cache-hit input widens to 50x.
The ratios widening sharply means this pricing scheme has promoted “whether the cache is hit” from a secondary factor to the core variable that determines cost. For the same model and the same business, raising the cache hit rate from 20% to 90% can produce a difference of several times on the actual bill. Understanding this is the prerequisite for every cost optimization that follows.
3. The Leverage Effect of Cache Hit Rate
To calculate real cost, you cannot use list prices directly. You first have to compute the weighted effective unit price.
The formula for the effective input unit price is straightforward:
Effective input price = cache-hit price × hit rate + uncached price × (1 - hit rate)
In scenarios like coding agents — long context, with a highly stable prompt prefix — if the inference architecture supports prefix reuse (industry-common designs such as prefill/decode disaggregation or Prefix Cache), achieving a cache hit rate above 90% is feasible. In that case:
Effective input price = 2 × 0.9 + 20 × 0.1 = 3.8 CNY / 1M tokens
Note that the list price is 20, but after weighting it is only 3.8. This is the leverage of caching — it compresses the high price of “uncached” cost down into a very small slice.
4. Real Cost Comparison Across Two Typical Scenarios
The difference only truly shows up when input and output are combined according to the business’s actual ratio. The formula for the blended unit price is:
Blended price = (input token share × effective input price + output token share × output price) / total tokens
Scenario A: Enterprise-grade long-context agent (input:output ratio ≈ 20:1, 90% cache hit rate)
- Effective input price = 2 × 0.9 + 20 × 0.1 = 3.8 CNY
- Output price = 100 CNY
- Blended = (20 × 3.8 + 1 × 100) / 21 = 176 / 21 ≈ 8.4 CNY / 1M tokens
Scenario B: Occasional individual developer (input:output ratio ≈ 5:1, 20% cache hit rate)
- Effective input price = 2 × 0.2 + 20 × 0.8 = 16.4 CNY
- Output price = 100 CNY
- Blended = (5 × 16.4 + 1 × 100) / 6 = 182 / 6 ≈ 30.3 CNY / 1M tokens
Same model, blended cost differs by roughly 3.6x. The difference does not come from the price sheet — it comes from how you use the model: a high-frequency, long-context, high-reuse enterprise workload is far cheaper than a low-frequency individual workload that changes its prompt every time. This explains why “the unit price went up” and “my bill got more expensive” are not necessarily linked.
5. Four Levers for Cost Optimization
- Raise the cache hit rate: Put stable system prompts, tool definitions, and long-document prefixes at the very front of the request and keep them unchanged, so this portion hits the cache. Session isolation and fixed prompt templates can also significantly improve reuse.
- Compress output tokens: Output has the highest unit price, so optimize it first. Constraining generation length, using structured output, and avoiding verbose restatement directly cut the most expensive tier.
- Optimize the input/output ratio: Input-heavy workloads at 20:1 are extremely sensitive to caching and are worth investing engineering effort in prefix reuse, while output-heavy workloads need tighter control over generation volume.
- Build cost observability: Using general-purpose capabilities such as logging, monitoring, and alerting, instrument token consumption and cache-hit behavior by business line and API, and alert on anomalous calls promptly. On platforms such as Baidu AI Cloud, you can combine general-purpose components like object storage, log analysis, and monitoring/alerting to build a cost dashboard, making usage transparent and giving every optimization decision a data basis.
6. Conclusion
Faced with a model price change, the right approach is not to stare at a single tier’s list price in shock, but to build an estimation framework of “three-tier billing + weighted unit price + business ratio.” Cache hit rate is an underestimated cost lever: it determines the effective price on the input side, and through the input/output ratio it is indirectly amplified into the overall bill. Whoever can make their calling pattern closer to the “high-hit, long-reuse” enterprise scenario can still achieve a lower real inference cost even when nominal unit prices rise. Pricing is the vendor’s business, but cost structure can be reshaped by engineering — and that is the most important takeaway of this article.
