LLM API Real Cost: Cache Hit Rate Is the Variable That Actually Matters

1. The Overlooked Billing Variable: Standard Pricing Is Not Your Actual Cost

When evaluating large language model APIs, most teams start by comparing price sheets side by side, looking at the input and output prices per million tokens. That seems reasonable, but it ignores a variable that has an outsized impact on the final bill: cache hit rate.

Major LLM platforms commonly apply differentiated pricing to input tokens. The portion that hits the context cache is billed at a discounted “cache hit” rate, while the portion that misses is billed at the standard rate. Cache hit pricing is often only 10% to 40% of the standard input price. This means that for the same model and the same token volume, the real input cost can differ by 3x to 5x depending on the shape of your traffic.

In other words, the standard price on the pricing page is the upper bound of your cost, not the cost itself. Your actual cost is determined jointly by your request traffic structure and your cache hit rate. Making budget and model selection decisions based only on standard pricing often leads to significantly overestimated costs and can even push teams toward the wrong model choice.

2. How Context Caching Works

To understand why cache hit rate matters so much, it helps to understand the caching mechanism itself:

1. KV cache reuse: During Transformer inference, every token at every layer produces Key/Value intermediate results. If two requests share the same beginning, or prefix, the later request can reuse the already-computed KV results and skip redundant prefill computation. Providers pass part of those saved compute costs back to users in the form of a “cache hit” price that is far lower than the standard input price.
2. Exact prefix matching: Most implementations require the beginning of a request to match a previous request token by token. If a single token in the prefix differs, the cache for everything after it becomes invalid. Cache matching is based on exact literal equality, not semantic similarity.
3. Time to live: Caches typically expire at the minute level. A cache hit refreshes the expiration timer. Entries that are not accessed before the timeout are evicted, and the next request must write the cache again.
4. Minimum thresholds and granularity: Some platforms enforce a minimum cacheable token count and determine caching at the granularity of prefix segments.

This leads to a general formula for actual input cost:

Actual input cost = cache hit tokens × cache hit price + cache miss tokens × standard price (some platforms also charge a cache write fee or storage fee)

Major LLM platforms, such as Baidu AI Cloud, generally support context caching and differentiate between cache hit and cache miss pricing in their billing. However, the discount levels, write and storage fees, and cache expiration windows vary by provider. Evaluation should always be based on the official billing documentation of the platform you are using.

3. Calculating the Real Cost: Simple Arithmetic

Let’s walk through a calculation with illustrative parameters. Assume a model has a standard input price of 4 yuan per million tokens and a cache hit price of 0.7 yuan per million tokens. The exact values will depend on the platform you choose.

Suppose your traffic has a cache hit rate of 90%, and you process 1 million input tokens:

• Cache hit portion: 900,000 tokens × 0.7 yuan per million = 0.63 yuan
• Cache miss portion: 100,000 tokens × 4 yuan per million = 0.40 yuan
• Total: 1.03 yuan

Compared with the standard price of 4 yuan, the actual input cost is about 25.8% of the list price, roughly a 75% discount.

A sensitivity analysis makes the point even clearer:

Cache hit rate	Actual input cost for 1M tokens	Relative to standard price
0%	4.00 yuan	100%
50%	2.35 yuan	58.8%
80%	1.36 yuan	34.0%
90%	1.03 yuan	25.8%
95%	0.87 yuan	21.6%

The gap is striking: moving from a 0% hit rate to 90% cuts input cost by nearly 4x. And that final improvement from 90% to 95% still saves about another 15%. Cache hit rate is a genuine cost lever. In workloads with heavy prefix reuse, such as long contexts and multi-turn conversations, engineering teams commonly observe hit rates in the 80% to 95% range. This calculation is not a theoretical exercise; it closely reflects the billing profile of real production traffic.

Here is a small utility function that captures the same logic for reuse in cost models:

def actual_input_cost(total_tokens, hit_rate, price_std, price_hit):
    hit = total_tokens * hit_rate
    miss = total_tokens * (1 - hit_rate)
    return hit * price_hit + miss * price_std
# Illustrative parameters:
# standard price = 4 yuan/M, cache hit price = 0.7 yuan/M, hit rate = 90%
cost = actual_input_cost(1_000_000, 0.9, 4, 0.7)
print(cost)        # 1.03 yuan
print(cost / 4)    # 0.2575, about 25.8% of the standard price

4. Which Scenarios Benefit Most from Caching

High-benefit scenarios:

1. Long, stable system prompts: Role definitions, business rules, output format specifications, and other fixed content appear in every request and naturally form a cacheable prefix.
2. Multi-turn conversations: Conversation history is inherently append-only. The prefix of turn N is almost identical to the prefix of turn N-1, so the hit rate rises as the conversation progresses.
3. Agent and tool-calling loops: Multi-step reasoning for the same task shares extensive system prompts, tool definitions, and intermediate context across requests, resulting in high reuse.
4. RAG retrieval-augmented generation: Stable domain knowledge and document chunks can be organized into the prefix. Even when retrieval results change, placing the variable content at the end preserves the prefix cache.
5. Batch document analysis and code repository Q&A: The same long document or codebase serves as a prefix reused by hundreds or thousands of questions.

Low-benefit scenarios:

• One-off short requests or long-tail requests with completely different prefixes;
• Very low-frequency tasks where the interval between calls exceeds the cache expiration window;
• Implementations that place timestamps, random IDs, or other dynamic fields at the very beginning of the prompt.

5. Engineering Practices for Improving Cache Hit Rate

Cache hit rate is not a matter of luck. It is the result of a series of engineering decisions:

1. Design prompts with a stable prefix and a dynamic suffix. Put system prompts, tool definitions, knowledge base documents, few-shot examples, and other stable content at the front. Put user questions, timestamps, retrieval results, and other variable content at the end. The more stable and longer the prefix, the more cache you can reuse.

def build_prompt(system_rules, knowledge, few_shots, user_query):
    # Stable prefix: byte-for-byte identical across requests,
    # which is the prerequisite for cache hits
    stable_prefix = [system_rules, *knowledge, *few_shots]
    # Dynamic suffix: request-specific content goes at the end
    return "\n\n".join([*stable_prefix, user_query])

1. Eliminate prefix jitter. Do not insert timestamps, request IDs, or randomly ordered fields at the beginning of the prompt. Keep list content in deterministic order. Avoid rewrites that are semantically equivalent but literally different. Caching relies on exact token-prefix matching, and a single character difference can invalidate everything that follows.

2. Maintain multi-turn conversations in an append-only fashion. Keep the order and content of the message history unchanged, and only append to the end for each new turn. Avoid rewriting, compressing, or reordering history between turns, since that invalidates the entire cached prefix.

3. Pay attention to concurrency and routing. Under high concurrency, requests from the same conversation may be distributed across different cache partitions, diluting the hit rate. If your platform or gateway supports session affinity routing, enable it to preserve the temporal and spatial locality of requests.

4. Align call patterns with cache TTL. For high-frequency interactions such as customer support, agents, and IDE autocomplete, the cache stays warm almost continuously. For low-frequency tasks, evaluate whether the interval between calls exceeds the expiration window, and consider prewarming or task orchestration to improve reuse.

5. Make cache hit rate an observable metric. Track “cache hit tokens / total input tokens” in billing and monitoring systems, and break it down by business scenario and prompt version. Before rolling out a new prompt, run regression tests with replayed production traffic to ensure that a seemingly harmless prompt change does not double your cost.

6. Building a Complete Cost Evaluation Framework

1. Replace the standard unit price with an effective unit price: Effective input unit price = (cache hit tokens × cache hit price + cache miss tokens × standard price + write fees + storage fees) / total input tokens. Some platforms charge additional fees for the initial cache write. The higher your hit rate and the more thoroughly you reuse cached content, the more those overhead costs are amortized.

2. Do not ignore output costs: Output tokens generally cannot be cached or reused, and their unit price is usually higher than input. Cache optimization mainly compresses input cost. The overall bill reduction depends on your workload’s input-to-output ratio, so model them separately when evaluating.

3. Benchmark with real traffic: Cache hit rate is a function of traffic, not an attribute of the model. Replay production request logs to measure the actual hit rate for each scenario, then plug those numbers into the formula to get a credible cost forecast.

4. Normalize your comparison across providers: When comparing platforms, use a consistent “effective unit price” that includes cache hit pricing, write fees, storage fees, expiration windows, and minimum cache thresholds, rather than comparing only the standard prices listed on pricing pages.

7. Summary

Three key takeaways:

First, standard pricing is a cost ceiling, not the cost itself. Any cost estimate that ignores cache hit rate will be distorted.

Second, cache hit rate is the biggest cost lever. A 90% hit rate can compress input cost to roughly a quarter of the standard price.

Third, cache hit rate is an engineerable metric. Through prompt structure design, append-only conversation maintenance, session affinity routing, and continuous monitoring, teams can actively manage it. Bringing cache hit rate into your cost model and vendor selection process is the key step from “comparing unit prices” to “managing costs.”