LLM Inference: Why We Need KV Cache Instead of Q Cache

1. Autoregressive Generation: Compute and Discard

Autoregressive language models typically process the prompt once during inference and then generate subsequent content token by token. Each newly generated token is appended to the end of the existing sequence to serve as part of the input for the next step. Without any caching, generating a single token requires feeding the entire historical sequence back into the model to recompute the Query, Key, and Value for all tokens before executing Attention. Since the representations of historical tokens do not change when new tokens appear—guaranteed by the causal mask—this repeated computation is pure waste.

The essence of KV Cache is to save “what has already been computed multiple times,” allowing new calculations to focus solely on the incremental parts.

2. The Mathematical Structure of the Attention Layer

Let the input of a self-attention layer be the hidden state matrix X, where the number of rows equals the current sequence length and the number of columns equals hidden_size. Three learnable matrices (W_Q), (W_K), and (W_V) project X into three matrices:

[
Q = X W_Q,\quad K = X W_K,\quad V = X W_V
]

The Attention output is:

[
O = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V
]

Where (d_k) is the dimension of K. Causal attention requires that each position can only attend to itself and previous positions, meaning the (QK^T) matrix is a lower triangular matrix with the upper right corner masked out.

Observing this formula reveals the structural characteristic that “historical K/V are reused multiple times”: the V of the i-th token will be used by all subsequent positions in the calculation; the K of the i-th token will also be used by all subsequent positions to calculate attention scores. However, the Q of the i-th token is used only once when computing the output for the i-th position, and is never used again by the outputs of other positions.

This provides the theoretical foundation for KV Cache: K and V are “memories” reused across steps, while Q is a “query” used only within a single step.

3. The Two-Phase Division of Prefill and Decode

Practical inference engines typically divide the generation process into two phases.

The Prefill phase receives the user’s input prompt. Since all tokens within the prompt are available simultaneously, this phase can compute Q, K, and V for each token in parallel and obtain the Attention output for all positions at once. Ultimately, the output of the last position is mapped to the probability distribution of the first new token. Meanwhile, the K and V matrices generated by this layer are written to the cache.

The Decode phase is entirely different. It receives only the latest generated token at a time. Assuming the input vector of the current newest token is (x_{\text{new}}):

[
q = x{\text{new}} W_Q,\quad
k = x{\text{new}} WK,\quad
v = x{\text{new}} W_V
]

Next, we simply dot-product q with the historical K (concatenated with the new k) to get attention scores, and then perform a weighted sum with the historical V (concatenated with the new v) to get the current token’s Attention output.

The pseudocode looks like this:

def decode_step(x_new, W_Q, W_K, W_V, K_cache, V_cache):
    q = x_new @ W_Q
    k = x_new @ W_K
    v = x_new @ W_V
    K = concat(K_cache, k, dim=seq)
    V = concat(V_cache, v, dim=seq)
    scores = q @ K.T / sqrt(d_k)
    attn = softmax(scores, mask=causal_mask)
    out = attn @ V
    return out, K, V

Note that historical Q is not used here. No matter how long K_cache and V_cache are, the q for the current step is just the new query for the current token itself. The q of historical tokens completed its mission during the prefill phase and will not appear again in subsequent calculations. Therefore, caching Q not only fails to speed up inference, but also consumes extra memory and increases read/write overhead.

4. Why There is No Q Cache

To rigorously explain “why there is no Q Cache,” we can look at the order of matrix multiplication. The actual Attention output that needs to be calculated during the decode phase is:

[
o{\text{new}} = \text{softmax}\left(\frac{q \cdot (K{\text{cache}} \oplus k)^T}{\sqrt{dk}}\right) (V{\text{cache}} \oplus v)
]

In this equation, q is the newly calculated vector for the current step, while K_cache and V_cache are the cached data. Among all the variables involved in the calculation, Q_cache is completely absent. This is because the attention scores of future tokens are always the dot product of “new Q and historical K,” rather than “historical Q and anything else.” In causal attention, historical positions do not attend to future positions, so historical Q does not need to be retrieved at the current position.

To put it more simply: Attention is “using a query to look up a key-value table.” Once the keys and values in the table are written, they persist long-term for all subsequent queries to retrieve. The query itself only represents “what I want to search for right now,” and is useless once the search is done. Saving all historical queries is like saving the search terms entered by every user during every search. It might be useful for analyzing user behavior, but it provides no help for the retrieval process of the “current new query.”

Of course, this does not mean Q is unimportant. The prefill phase must compute Q for every token in the prompt, because each position in that prompt needs to generate its own Attention output to yield the correct first new token. The decode phase must also compute Q for the newest token to obtain the new query vector. It’s just that Q’s lifecycle is extremely short: from computation to usage, it is completed within a single step.

5. The Costs of KV Cache and Optimization Directions

The primary cost of KV Cache is memory. Because the K/V of every token needs to be cached by layer and by attention head, the total memory is approximately:

[
\text{memory} \approx 2 \times \text{batch} \times \text{seq_len} \times \text{num_layers} \times \text{num_heads} \times \text{head_dim} \times \text{bytes}
]

As the sequence continues to grow, especially when processing long documents or multi-turn conversations, KV Cache becomes the primary consumer of GPU memory. To maintain inference speed under lower memory constraints, the industry has developed various derivative solutions. For example, reducing the number of K/V heads (such as Multi-Query Attention and Grouped-Query Attention) allows multiple query heads to share a single set of K/V projections, thereby reducing the cache size exponentially. Another approach is based on sliding windows or sparse attention, which only caches the K/V of recent tokens, discarding distant history. These solutions represent further trade-offs regarding “what and how much to cache,” but they do not overturn the fundamental conclusion that “K/V need caching while Q does not.”

Another easily overlooked detail is that each layer requires an independent KV Cache. Because the input of the next layer is the output of the previous one, the historical information across different layers is mutually independent. If only the last layer is cached, the intermediate layers still need to be recalculated, failing to eliminate the main repetitive computation. Practical inference frameworks typically organize the cache structure by layer and synchronously update the K/V of all layers after each decode step.

6. Summary

KV Cache is a natural byproduct of the autoregressive Attention computation structure, not merely a clever trick. The original input is just a sequence of tokens, but the Attention formula itself dictates that K and V will be reused by all future tokens, while Q is an “use-once-and-discard” query vector. Therefore:

• Caching K/V avoids repeatedly calculating the key-value projections and Attention aggregation for historical tokens.
• Caching Q lacks any subsequent read path and only wastes memory and bandwidth.

Understanding this not only helps you answer “why there is KV Cache but no Q Cache,” but also enables you to make more rational caching and memory access optimization decisions when designing long-context inference, quantized inference, or distributed inference.