Cache Hit Rate: The Hidden Variable That Determines LLM API Costs

1. Why Standard Pricing Alone Is the Biggest Cost Evaluation Mistake

When teams evaluate LLM APIs, the most common first step is to copy each vendor’s listed prices into a spreadsheet: “X per million input tokens, Y per million output tokens,” then pick the cheapest. This approach is roughly acceptable for short-context, single-turn calls. But once you move into multi-turn conversations, long-document Q&A, codebase analysis, or RAG retrieval-augmented scenarios—workloads with “large input, small output”—the conclusion can be badly distorted.

The reason is that it ignores a core variable in modern LLM billing: cache hit rate. Most major LLM providers support context caching (also called prompt caching). When the input prefix of a request matches a recent request, the matched portion is billed at a “cache price” far below the standard rate. Depending on the engineering implementation, hit rates can range from near 0% to over 90%, producing a 3-5x difference in effective input price. In other words, a service with a 50% higher list price but a much better hit rate can actually end up cheaper.

2. What Context Caching Actually Saves

2.1 The Two Stages of Inference

Transformer inference has two phases:

• Prefill: The model reads all input tokens at once and computes Key/Value matrices for every attention layer, caching them (the KV Cache). This phase is compute-intensive and expensive.
• Decode: Output tokens are generated one at a time, with each step reading the KV Cache for attention computation.

The key insight: the KV Cache depends only on the input prefix itself. If two requests share the same first 200,000 tokens (for example, an identical system prompt plus an identical knowledge document), the second request does not need to recompute that prefix—it can reuse it directly. What caching saves is exactly this redundant prefill computation. The billing discount is essentially a partial rebate of the saved compute cost.

2.2 Three Conditions for a Cache Hit

• Strict prefix match: Matching starts from the very first token, position by position. Any difference invalidates everything after it.
• Within the validity window: Cache entries have a TTL (time to live). If not refreshed before expiry, they are invalidated.
• Not evicted: Cache space is limited, and entries that go unused for a long time may be evicted.

3. The Blended Billing Model: One Formula for Real Input Cost

With caching in play, input token billing splits into two segments: the hit portion is billed at the cache unit price (commonly 10-30% of the standard price in the industry), and the miss portion is billed at the standard unit price. This gives the general formula:

actual input cost = input volume × hit rate × cache price + input volume × (1 − hit rate) × standard price

A hypothetical example: suppose a model’s standard input price is ¥4 per million tokens and its cache hit price is ¥0.7 per million tokens (about 17.5% of standard, a common range). Processing 1M input tokens:

Hit rate	Hit portion	Miss portion	Total	Relative to standard
0%	¥0	¥4.00	¥4.00	100%
50%	¥0.35	¥2.00	¥2.35	~59%
80%	¥0.56	¥0.80	¥1.36	~34%
90%	¥0.63	¥0.40	¥1.03	~25%
95%	¥0.665	¥0.20	¥0.865	~22%

Three conclusions emerge immediately:

1. At a 90% hit rate, the actual input cost for 1M tokens is about ¥1.03—roughly 25% of the standard price.
2. Going from 0% to 90% hit rate produces nearly a 4x cost difference—larger than any list-price gap between vendors.
3. The marginal benefit of hit rate is always positive: going from 80% to 90% still saves about another 24%.

Two boundary conditions matter. First, caching applies only to input tokens; output tokens are still billed in full at the output price, so total cost = blended input cost + output volume × output price. Second, some billing rules charge a small fee for cache writes or offer different TTL tiers—these should be included in any calculation.

4. Three Steps to Estimate Your Real Hit Rate

4.1 Structural Estimation: Dissect Your Prompt

Split a request’s input into a “stable prefix” and a “volatile suffix”:

• Stable parts: system prompt, few-shot examples, tool (function calling) definitions, attached reference documents—typically unchanged across requests.
• Volatile parts: the current user question, the most recent turns of conversation, real-time retrieved snippets.

The stable portion’s share of total input is the natural ceiling on your hit rate. For example, a Q&A application with 100,000 input tokens, 90,000 of which come from a fixed knowledge document, has a theoretical ceiling of 90%.

4.2 Measurement: Read the Usage Statistics Fields

Mainstream API responses typically include usage details. A typical shape:

{
  "usage": {
    "prompt_tokens": 120000,
    "cached_tokens": 102000,
    "completion_tokens": 640
  }
}

cached_tokens / prompt_tokens is the per-request hit rate. In production, feed this field into your logging and monitoring stack and aggregate by both endpoint and prompt version. Look at distributions, not single data points.

4.3 Replay Load Testing for Calibration

In a staging environment, replay real historical traffic and measure the hit-rate distribution across different time windows and load levels. Avoid extrapolating an annual budget from an occasional single-request value.

5. Five Engineering Practices to Raise Hit Rate

5.1 Stabilize the Prefix: Put the Unchanging First, the Changing Last

This is the highest-leverage practice. Recommended prompt ordering:

[System prompt (fixed version)]         ← strongly stable
[Tool definitions / few-shot examples]  ← strongly stable
[Attached knowledge documents (fixed version)] ← strongly stable
[Conversation history (append-only, no rewrites)] ← weakly stable
[Current question + real-time retrieved snippets] ← volatile, placed last

Common anti-patterns:

• Prepending timestamps, request IDs, or auto-incrementing sequence numbers to the prompt header—a single differing token invalidates the entire prefix that follows.
• Rewriting, reordering, or inline-summarizing history on every turn.
• Tweaking system prompt wording frequently without version management.

5.2 Organize Conversation History as Append-Only

In multi-turn conversations, history is naturally “append-only”—an ideal cache prefix. If summarization is necessary, concatenate the summary after the stable region rather than replacing the prefix itself.

5.3 Version-Control System Prompts and Documents

Roll out changes gradually by version rather than switching all at once. This smooths cost fluctuations and makes it easier to attribute hit-rate changes.

5.4 Watch TTL and Call Cadence

Low-frequency scheduled jobs (such as an hourly batch analysis) may hit an expired cache every time. Evaluate the net benefit of sending a lightweight keep-alive request within the TTL window: trading a tiny cost for cache-priced large inputs is usually worthwhile.

5.5 Two-Tier Context Organization for RAG

Retrieved snippets differ on every call, so they are inherently hard to cache. Use a frequently referenced fixed document set as the stable prefix, append the real-time retrieved snippets afterward, and consider pre-attaching retrieval results for high-frequency queries when necessary.

6. Model Selection and Budgeting: A Reusable Checklist

1. Collect list prices: standard input price, cache hit price, output price, and any cache write/storage fees.
2. Estimate hit rate: start with a structural ceiling estimate from your prompt layout, then calibrate by measurement.
3. Measure the input/output ratio: weight by your application’s real token structure to compute per-request blended cost.
4. Include add-ons: TTL tiers and how rate limits and concurrency quotas indirectly affect hit rate.
5. Run sensitivity analysis: the budget range when hit rate varies by ±10% and input volume by ±20%.
6. Assess engineering cost: the development effort for prefix restructuring and monitoring integration, and the payback period.

A general-purpose estimation function:

def estimate_cost(input_tokens_m: float, hit_rate: float,
                  cache_price: float, base_price: float,
                  output_tokens_m: float, output_price: float) -> float:
    """Units: millions of tokens; price units just need to be consistent"""
    cached = input_tokens_m * hit_rate * cache_price
    fresh  = input_tokens_m * (1 - hit_rate) * base_price
    output = output_tokens_m * output_price
    return cached + fresh + output

Major LLM platforms, including Baidu AI Cloud, generally provide context caching and usage statistics capabilities within their model services. Specific price ratios and TTL rules should be confirmed against each platform’s official documentation. Only by plugging different platforms’ parameters into the same formula can you get genuinely comparable conclusions.

7. Conclusion

Cache hit rate is the variable “hidden beneath the list price” in LLM API costs: it is determined by the quality of your prompt engineering, and it in turn determines the actual billing structure. The right approach is to push from both sides—at selection time, evaluate total cost using the blended billing formula plus measured hit rate; at engineering time, raise the hit rate with stable prefixes, append-only conversations, and version management. When the effective input price is only 20-30% of list price, many scenarios that previously looked “too expensive” come back into feasible territory.