LLM API Cost Evaluation: How Cache Hit Rate Affects Input Pricing

1. A Widely Overlooked Billing Variable

When evaluating LLM API costs, many developers and enterprise procurement decision-makers instinctively turn to the provider’s published pricing table as the primary basis for budgeting and comparing options. The approach is intuitive, but it contains a critical blind spot: standard pricing describes only the base price per token and does not reflect the complexity of billing logic in real-world call scenarios.

In practice, mainstream LLM APIs do not simply bill input usage as the standard unit price multiplied by the total token count. Billing systems split input tokens into two tiers depending on whether the request hits the server-side context cache: a cache-hit portion and a cache-miss portion. Cache-hit tokens are billed at a significantly discounted price, while cache-miss tokens are charged at the standard rate. The cache hit rate—the proportion of a request’s input tokens that fall into the cache-hit tier—therefore becomes the core variable that determines actual input cost.

The difference between a 0% and a 90% cache hit rate often corresponds to a 3–5x gap in effective input pricing. A gap of this magnitude can change the cost-effectiveness assessment of a technical solution and even influence model selection and architecture design decisions. Unfortunately, many cost estimation models still ignore this variable, which is why budgets are so frequently overstated.

2. How Context Caching Works

To understand how cache hit rate affects cost, it helps to first clarify how context caching in LLM APIs operates.

When a client sends a request to an LLM service, the request usually carries a block of context—system prompt, conversation history, document fragments, or tool definitions. After receiving the request, the server processes and analyzes the input context. If it detects that the context prefix of the current request is highly consistent with recent requests—for example, when large numbers of calls from the same application share the same system prompt and tool configuration—this repeated portion can be read directly from the cache instead of being recomputed through the full model forward pass.

The key point: whether the cache hits depends primarily on how much the context prefixes overlap across requests. If every request from an application carries an identical system prompt (say, a 2,000-token instruction template), then in high-concurrency scenarios a large proportion of those tokens will be counted as cache hits. Conversely, if each request carries different, non-reusable long-form content, the cache hit rate will be low.

From a billing perspective, cache-hit tokens skip the cost of repeated computation, so their unit price is typically only a fraction of the standard rate. Using a common pricing structure as an example: the standard input price might be around RMB 4 per million tokens, while the cache-hit input price can be as low as RMB 0.7 per million tokens—roughly 17.5% of the standard rate. This price gap is the root cause of the cache hit rate’s impact on actual cost.

3. A Cost Calculation at a 90% Cache Hit Rate

To show the effect concretely, let’s run a standardized 1M-token cost calculation using typical pricing for a multimodal LLM API. Assume the following:

• Standard input price: RMB 4 / 1M tokens
• Cache-hit input price: RMB 0.7 / 1M tokens

Suppose that during steady operation, server-side statistics show the application’s cache hit rate stabilized at around 90%. For a total input of 1M tokens, actual billing splits into two parts:

Part 1: The cache-hit portion (90%)

Number of cache-hit tokens:
1,000,000 × 90% = 900,000 tokens

Billed at the cache-hit price:
900,000 / 1,000,000 × 0.7 = RMB 0.63

Part 2: The cache-miss portion (10%)

Number of cache-miss tokens:
1,000,000 × 10% = 100,000 tokens

Billed at the standard input price:
100,000 / 1,000,000 × 4 = RMB 0.4

Total actual input cost:
0.63 + 0.4 = RMB 1.03 / 1M tokens

Compared with the standard price of RMB 4 / 1M tokens:

1.03 ÷ 4 = 0.2575 ≈ 25.75% of the standard price

In other words, at a 90% cache hit rate, the actual input cost is only about one quarter of the standard price. If the hit rate drops to 50%, the actual cost becomes:

50% × 0.7 + 50% × 4 = 0.35 + 2 = RMB 2.35 / 1M tokens

At that level, the effective price is about 59% of the standard rate. Compared with the RMB 1.03 at a 90% hit rate, the difference is more than double. These numbers make the point clearly: cache hit rate is the key lever on actual input cost, and its impact far exceeds what most people intuitively expect.

4. Engineering Factors That Determine Cache Hit Rate

Given how strongly cache hit rate affects cost, which engineering factors determine whether an application’s hit rate is high or low? Based on common LLM API usage patterns, the main factors are:

4.1 System Prompt Stability

The system prompt is the most fixed part of an application’s context. It typically contains role definitions, output format requirements, safety constraints, and task instructions. If the system prompt remains exactly the same across a large number of requests, those tokens can almost entirely hit the cache. Conversely, if the system prompt is dynamically generated—varying by user type, scenario, or A/B testing strategy—the cache hit rate drops significantly.

4.2 Context Prefix Reuse

Server-side caches usually match on the context prefix: only when two requests begin with exactly the same content can the following content reuse cached results. Placing the most stable content (system prompt, tool definitions, shared knowledge base sections) at the very front of the request and the volatile content (the latest user input) at the end is therefore an effective strategy for raising the hit rate. If each user message is placed before the system prompt, the cache mechanism can barely function at all.

4.3 Request Volume and Timing

Caches are time-sensitive. When many requests arrive within a short time window and share the same context prefix, the probability that they hit one another’s cache rises significantly. If requests are spread across long intervals, or concurrency is low, the cache may already have expired or been evicted, and the hit rate falls.

4.4 History Management in Multi-Turn Conversations

In multi-turn conversations, history is a major component of the context. If each turn resends the full conversation history starting from the first message, the earlier messages—provided their content stays identical—have a continuous chance of hitting the cache. But as the number of turns grows, the total history keeps expanding, and the share of volatile content shifts as well. A more refined approach is to trim or summarize the history, keeping only the parts most relevant to the current question, to control the volume of tokens that miss the cache.

5. Practical Recommendations for Cost Evaluation

For developers and enterprise users, incorporating cache hit rate into the cost evaluation framework is a key step toward more accurate budgets. Here are some actionable recommendations:

First, do not treat standard pricing as the sole basis for cost comparison. During the selection stage, obtain the target API’s cache billing rules, including the discount ratio for cache-hit tokens, the cache validity window, and how the hit rate is measured. If the official documentation does not disclose these details clearly, confirm with the provider’s technical support or estimate them through small-scale testing.

Second, estimate the effective unit price with a weighted formula. Build a simple estimation model:

Effective input unit price = cache_hit_rate × cache_hit_price + (1 - cache_hit_rate) × standard_price

When an exact cache hit rate is unavailable, use a conservative range based on application type. For applications with a highly stable system prompt and high call volume, the hit rate may fall between 80%–95%. For applications whose context differs substantially on every call, the hit rate may be below 30%.

Third, optimize the context structure in code. Put stable instructions, tool definitions, and shared knowledge base content at the very front of the request context, and put user input and dynamic data at the end. Avoid dynamically generating system prompts in high-frequency scenarios. For multi-turn conversations, evaluate whether history compression is needed to reduce the number of cache-miss tokens.

Fourth, keep monitoring actual billing data. Where conditions allow, export the actual input token totals and actual charges from bills or usage reports, work backward to the blended unit price, and compare it with the standard price. This after-the-fact validation helps calibrate the cost model and can also surface abnormal drops in cache hit rate.

6. Conclusion

Evaluating LLM API costs is far more involved than simply reading the standard pricing table. As the core variable in the billing logic, cache hit rate can move the actual input price by a factor of several. In a typical 90% hit-rate scenario, the actual input cost of 1M tokens may be only about one quarter of the standard price—a gap large enough to influence model selection, architecture design, and budget planning.

Understanding how the caching mechanism works, mastering cost estimation based on cache hit rate, and actively optimizing context structure to raise the hit rate are core capabilities for every LLM application developer and technical decision-maker. Truly professional teams do not stop at pricing comparison tables; they dig into the details of the billing logic and make better-informed, data-driven decisions.