LLM API Cost Breakdown: Cache Hit Rate and Effective Input Pricing

1. The Overlooked Variable: List Price Is Not What You Pay

When teams estimate the cost of calling a large language model API, the instinct is to open the official pricing page, find the “input price” and “output price” columns, and multiply by projected token usage. This is a reasonable starting point—but it misses a critical fact: list prices are a ceiling, not your average cost per token.

The variable that truly determines your bill is cache hit rate. A cache hit occurs when the context in a new request closely overlaps with content the server has already processed. The server can reuse previously computed intermediate results rather than recalculating everything from scratch. Because the compute cost drops substantially, most cloud providers offer deeply discounted rates for cached tokens.

In practical terms, two teams consuming 1M input tokens each can see a 3× to 5× difference in their bills, depending on whether their cache hit rate is near 0% or near 90%. Relying on list prices alone means budgeting for the worst case and misrepresenting your typical spend.

2. How Caching Works Under the Hood

To understand why cache hit rate matters so much, it helps to review the basics of LLM inference.

When a request arrives, the model performs a forward pass on each token in the input sequence, producing a Key-Value Cache (KV Cache). The KV Cache stores the key and value matrices from every attention layer and is the core data structure that avoids redundant computation during autoregressive generation. In multi-turn conversations, long-document QA, code completion, and similar workloads, consecutive requests often share a large prefix—system prompts, chat history, document bodies, and so on. Recomputing all of that every time would waste enormous compute.

The caching mechanism solves this: the server retains previously computed KV Cache entries for a window of time, matched by prefix. When a new request hits the same prefix, the server skips the cached portion and only computes the new tokens. This cuts both latency and cost, so providers pass a portion of the savings back to users as a discount.

From an engineering standpoint, cache hit rate depends on several factors:

• Prefix stability: Whether system prompts, role definitions, and fixed templates remain identical across calls.
• Request pattern: Multi-turn conversations naturally hit more often than isolated single-shot requests.
• Cache TTL: How long the server retains KV Cache entries.
• Content overlap: Whether the task itself involves large amounts of shared context.

3. Calculating the Real Input Price

Let’s walk through a concrete example. Suppose a model has a standard input price of 4 CNY per M tokens, a cached-token price of 0.7 CNY per M tokens, and the actual cache hit rate is around 90%.

For 1M input tokens, the breakdown looks like this:

Component	Share	Unit Price (CNY / M tokens)	Cost (CNY)
Cached tokens	90%	0.7	0.63
Non-cached tokens	10%	4.0	0.40
Total	100%	—	1.03

The effective input cost is 1.03 CNY per M tokens—roughly a 75% discount off the list price.

This is not an outlier. Publicly available real-time statistics show that under high-concurrency workloads, mainstream LLM APIs typically sustain cache hit rates between 85% and 95%. Using 90% as a median estimate is well-grounded.

Expressed as a formula:

Effective Input Price = Hit Rate × Cache Price + (1 − Hit Rate) × List Price

Plugging in the numbers:

Effective Input Price = 0.9 × 0.7 + 0.1 × 4.0 = 0.63 + 0.40 = 1.03 CNY / M tokens

The discount ratio:

Discount Ratio = 1.03 / 4.0 ≈ 0.2575 ≈ 25.75% (about 2.5× off)

4. Cost Comparison Across Hit Rates

The table below shows how the effective input price shifts at different hit rates:

Cache Hit Rate	Effective Input Price (CNY / M tokens)	Discount vs. List Price
0%	4.00	None
50%	2.35	~41% off
70%	1.69	~58% off
85%	1.20	~70% off
90%	1.03	~75% off
95%	0.87	~78% off

From 0% to 90%, the per-token price drops about 74%. Even moving from 50% to 90% nearly halves the cost. This explains a common observation: two teams using the same model and consuming similar token volumes can end up with bills that differ by several times—and the gap often lives in the cache hit rate.

5. Engineering Practices to Boost Hit Rate

Once you understand the mechanism, the practical question is: how do you push the hit rate closer to its theoretical ceiling?

1. Keep prefixes stable

Place system prompts, role definitions, and fixed instructions at the very start of every request, and make sure they are byte-identical across calls. Even a trivial change—an extra space or a different punctuation mark—can break prefix matching and invalidate the entire cache.

2. Structure multi-turn conversations carefully

Multi-turn chats are naturally cache-friendly: the message history stays as a fixed prefix while the new question is appended. Avoid reordering or rewriting past messages between turns, as that destroys prefix continuity.

3. Share context across batch jobs

For document QA, data extraction, and other batch workloads, if multiple requests target the same long document, position that document content at the prefix so subsequent requests can reuse the same KV Cache.

4. Measure, don’t assume

Do not guess the hit rate—measure it. Most API responses include the count of cached tokens, which lets you compute the actual hit rate and feed it into your cost-monitoring dashboard. If the rate stays low, audit how you construct your requests.

5. Separate cold-start from steady state

The first request in any new session cannot hit the cache; that is a normal cold-start cost. When estimating total spend, distinguish between the cold-start phase and the steady-state phase rather than extrapolating from the first-request price.

6. A Better Approach to Cost Estimation

Putting it all together, a more accurate cost-estimation workflow looks like this:

1. Confirm both rates: Pull the list price and the cached-token price from the official documentation.
2. Estimate hit rate: Use historical data from your own workload or benchmarks from similar tasks to define a range.
3. Compute the weighted price: Apply the hit-rate-weighted average to get the effective input price.
4. Add output cost separately: Output tokens typically do not qualify for cache discounts and must be calculated on their own.
5. Apply a cold-start adjustment: For workloads with a high proportion of first requests, bump the estimate upward accordingly.
6. Calibrate continuously: After launch, back-calculate the hit rate from real invoices and refine your model.

Here is a Python-style pseudocode for the core logic:

def effective_input_price(
    standard_price: float,
    cache_price: float,
    hit_rate: float
) -> float:
    """
    Compute the effective per-token input price.
    :param standard_price: List input price (CNY / M tokens)
    :param cache_price: Cached-token price (CNY / M tokens)
    :param hit_rate: Cache hit rate, between 0 and 1
    :return: Effective input price
    """
    return hit_rate * cache_price + (1 - hit_rate) * standard_price
def discount_ratio(standard_price: float, effective_price: float) -> float:
    """Compute the discount ratio relative to the list price."""
    return effective_price / standard_price
# Example
std = 4.0
cache = 0.7
hit = 0.90
eff = effective_input_price(std, cache, hit)
print(f"Effective input price: {eff:.2f} CNY / M tokens")
print(f"Discount ratio: {discount_ratio(std, eff):.2%}")  # ~25.75%

7. Conclusion

Optimizing LLM API costs goes far beyond “pick the cheapest model.” Cache hit rate is an observable, optimizable, and highly impactful variable that deserves serious attention in architecture design and budget planning.

Three takeaways:

• List prices are ceilings, not averages. Effective input cost is driven by cache hit rate; in high-hit scenarios it can drop to roughly 25% of the list price.
• Hit rates can be actively improved. Stable prefixes, well-structured multi-turn conversations, and shared-context batch jobs all push the needle upward.
• Cost estimation must be data-driven. Build hit-rate monitoring, calibrate with real invoices, and avoid budget surprises.

Factoring cache hit rate into your cost model not only sharpens your forecasts—it also incentivizes better request design, creating a virtuous cycle of lower cost and higher efficiency.