Understanding KV Cache: The Core Mechanism Behind LLM Inference Cost Reduction
This article explains how KV Cache works and why it has become the technical foundation of the recent wave of LLM inference cost reductions. Starting from the repeated computation in autoregressive generation, it shows how caching Key and Value vectors turns quadratic attention cost into linear cost, how to precisely estimate the GPU memory KV Cache consumes, and the common techniques used to reduce that memory footprint and raise cache hit rates, including Grouped-Query Attention (GQA)/Multi-Query Attention (MQA), paged memory management, and prefix caching. It closes with practical engineering advice on prefix consistency, request routing, context-length control, cache lifetime, and observability for lowering inference costs in long-context and high-concurrency scenarios.
1. Inference Cost: The New Battlefield for LLMs
Over the past two years, the price war around large models has intensified, with the unit price of some inference services permanently cut to a fraction of what it once was. Many people’s first reaction is that “vendors are burning cash on subsidies,” but from an engineering perspective, what actually enables these price cuts is a series of system-level optimizations on the inference side, and the single most important piece is caching.
When the cache hit rate approaches 100%, the vast majority of tokens no longer need to be recomputed, and the real compute cost per token can approach zero, leaving plenty of room for prices to drop. For this reason, understanding KV Cache and the prefix caching mechanism is essentially understanding the technical foundation behind this wave of cost reduction.
2. Starting from Autoregressive Generation: Where Repeated Computation Comes From
When a large language model generates text, it is fundamentally an autoregressive process: it produces one token at a time, appends it to the end of the sequence, and then generates the next.
- Step 1: Input “today the weather” → the model produces “is”
- Step 2: Input “today the weather is” → the model produces “nice”
- Step 3: Input “today the weather is nice” → the model produces the next token…
The problem is that every step recomputes attention over the entire sequence. In the attention mechanism, each token produces three vectors:
- Q (Query)
- K (Key)
- V (Value)
The key issue is this: in Step 2, the K and V of the first three tokens were already computed in Step 1, and the results are identical, yet they get recomputed all over again. The longer the sequence, the more severe this repeated computation becomes, and the total compute grows quadratically with sequence length. This is precisely why long-context inference is expensive and slow.
3. KV Cache: Storing What Has Already Been Computed
The core idea of KV Cache is quite simple: every time a new token is computed, its Key and Value are cached in GPU memory. On the next inference step, the K and V of historical tokens are read directly from the cache without recomputation; only the single newly generated token needs a full computation of Q, K, and V, after which its Q is used to compute attention scores against all cached K values, followed by a weighted sum over the cached V values.
As a result, the compute for each step drops from “processing the whole sequence” to “processing only the newest token,” and the complexity falls from quadratic to linear. This brings two direct benefits:
- Inference latency drops significantly — especially for long-text generation, where the improvement grows more pronounced toward the end.
- Repeated compute is saved — the same context no longer needs to be recomputed repeatedly, and compute utilization improves markedly.
4. How Much GPU Memory Does KV Cache Use? You Can Calculate It Exactly
The GPU memory footprint of KV Cache can be estimated precisely using the formula:
KV Cache size = 2 × num_layers × num_heads × head_dim × sequence_length × bytes_per_value
Here the factor 2 accounts for the two cached copies: Key and Value.
Example: Suppose a model has 32 layers, 32 heads, a head dimension of 128, and a sequence length of 4096, using FP16 (2 bytes):
Per-token, per-layer KV size = 2 × 32 × 128 × 2 = 16384 bytes = 16 KBAll layers + all tokens = 16 KB × 32 layers × 4096 tokens ≈ 2 GBWith batch size = 10 ≈ 20 GB
The conclusion is clear: for long-text or large-batch scenarios, the GPU memory consumed by KV Cache can exceed the model parameters themselves and become a direct throughput bottleneck. This is also why inference costs are so sensitive to “context length” and “concurrency.”
5. General Techniques for Reducing Memory and Improving Hit Rates
A number of general optimization techniques have emerged around KV Cache:
Grouped-Query Attention (GQA) / Multi-Query Attention (MQA)
Multiple Query heads share the same group of Key/Value heads, reducing the number of heads that need to be cached at the model-architecture level. KV Cache usage drops proportionally while maintaining quality close to the original.Paged memory management
Borrowing the paging idea from operating-system virtual memory, KV Cache is split into fixed-size blocks that are allocated and reclaimed on demand, significantly reducing memory fragmentation and allowing limited GPU memory to hold more concurrent requests.Prefix caching / context caching
When many requests share the same prefix (system prompts, few-shot examples, document bodies in long-document Q&A), the KV of the prefix portion only needs to be computed once and cached for reuse. Subsequent requests hit the cache directly, avoiding repeated computation.
6. Why Hit Rate Determines Cost: An Almost-Free Experience
Once you understand the mechanism, a common phenomenon becomes easy to explain: why users with high cache hit rates feel that it is “essentially free.”
The reason is that the billing logic is tied to the computation logic. The portion that hits the cache requires almost no recomputation, and you are charged only for the cache read and the few newly generated tokens. So when a large number of requests share a stable long prefix, the KV of that prefix is computed once and reused repeatedly, the cache hit rate becomes very high, the vast majority of compute is skipped, and the unit cost drops substantially.
From both an engineering and a business standpoint, the prerequisite for a “permanent price cut” is a genuine reduction in unit cost, and caching is one of the core levers that brings cost down. It is not marketing spin, but a measurable, reproducible system benefit.
7. Key Factors Affecting Hit Rate and Engineering Recommendations
To make caching actually deliver, engineers should pay attention to the following:
- Prefix consistency: Keep system prompts and instruction templates as fixed as possible, and avoid inserting volatile content at the beginning such as timestamps, random IDs, or session variables; otherwise prefix matching breaks and the cache is invalidated frequently.
- Request routing strategy: Route requests that share the same prefix to the same cache context or instance to improve local hit rate and avoid scattering the cache across multiple nodes.
- Context-length management: Trim ineffective content and avoid unnecessarily lengthening the sequence; since KV Cache grows linearly with length, a restrained context is direct cost reduction.
- Cache lifetime: Define the retention duration and invalidation policy explicitly, balancing hit rate against GPU memory usage.
- Observability: Use general capabilities such as monitoring, alerting, and logging services to continuously track key metrics like hit rate, GPU memory usage, and inference latency, so optimization is grounded in data.
8. Conclusion
The reduction of LLM inference costs is not simply a commercial concession, but the combined result of a series of system technologies: KV Cache, prefix caching, paged memory management, and attention-structure optimization. Among these, KV Cache is the foundation, prefix caching is the amplifier, hit rate is the metric, and cost is the outcome.
For developers and enterprises, the practical levers are to fix prefixes, route requests sensibly, control context length, and continuously measure cache performance with an observability system. Cloud providers such as Baidu AI Cloud typically offer model inference, caching, monitoring, alerting, and logging services as general cloud capabilities, which enterprises can combine flexibly according to their own workloads, reducing inference costs in long-context and high-concurrency scenarios while maintaining generation quality.
