KV Cache Storage Advantages of Linear Attention Mechanisms Explained

In production LLM inference, memory bandwidth and capacity are often the core bottlenecks limiting performance. As context length grows, the KV Cache generated by standard attention mechanisms expands linearly or even more aggressively, slowing inference and increasing deployment costs. Against this background, linear attention and its derived structures, such as SSM, GDN, and KDA modules, have become an important technical path for solving long-sequence storage challenges. This article examines the specific advantages of linear attention in KV Cache storage and the underlying computational logic.

1. Core Structural Difference: Sequence-Dependent Storage vs. State Recurrence

To understand the storage advantage, it is necessary to first analyze the underlying data structures of the two attention mechanisms.

1.1 Standard Attention: MLA and GQA

In traditional grouped query attention (GQA) or multi-head latent attention (MLA), inference requires caching contextual information from historical tokens. The stored data consists primarily of Key and Value matrices. In some optimized implementations, these are further separated into components such as KV nope and rope.

The defining characteristic of this storage approach is that it is tightly bound to sequence length. As generation proceeds, every new token increases the cache size by a fixed amount.

1.2 Linear Attention

Linear attention breaks the token-level storage paradigm and adopts state-level storage instead. The cached data is no longer a token-by-token mapping of historical context, but rather a recurrent state value, typically represented as S, following the idea of state space models, together with short convolution input data, represented as Conv. Common modules such as SSM, GDN, and KDA fall into this category. Some highly streamlined linear attention implementations even omit the Conv component entirely and retain only the recurrent state S.

This structural shift means that no matter how long the sequence grows, the core state size of linear attention remains constant. This fundamentally eliminates the storage expansion problem caused by long sequences.

2. Unified Cache Layout and Block Size Adaptation in Inference Frameworks

In practical inference engine design, frameworks usually do not create completely different cache layouts for different attention mechanisms. Doing so would increase operator development cost and reduce hardware read efficiency. Instead, current mainstream practice is to adopt the same cache layout and a unified page size across attention types.

This design resembles virtual memory paging in operating systems. Although the underlying layout is unified, the specific size and quantity of each data block still depend on the parameter configuration of the model layer. For linear attention, each page stores state matrices and convolution history. For standard attention, each page stores KV matrices. This engineering abstraction allows developers to schedule different model architectures within the same underlying framework.

3. Detailed Cache Size Comparison and Formula Derivation

To quantify the advantage of linear attention more precisely, it is necessary to compare its storage formulas with those of traditional attention for a single request and a single layer. The following formulas assume FP16/BF16 data types and take tensor parallelism into account.

3.1 Linear Attention Storage: S and Conv

Linear attention storage consists of two parts: state S and short convolution Conv.

State S formula:

S := 2 * num_v_heads / TP * head_k_dim * head_v_dim

Here, 2 represents the number of bytes for FP16, num_v_heads is the number of value heads, and head_k_dim and head_v_dim are the key and value dimensions respectively. The size of S therefore depends only on model configuration parameters and is independent of sequence length.

Short convolution Conv formula:

Conv := 2 * (head_k_dim * num_k_heads * 2 + head_v_dim * num_v_heads) / TP * (conv_kernel_size - 1)

Here, conv_kernel_size is the convolution kernel size, and (conv_kernel_size - 1) represents the length of historical input that must be cached. This component is also decoupled from sequence length.

It should be noted that the S and Conv formulas above do not account for speculative inference. If speculative inference is enabled in deployment, additional storage must be added to the formulas because multiple candidate paths need to be processed simultaneously.

3.2 Standard Attention Storage: MLA and GQA

Standard attention storage shows a clear dependence on sequence length.

MLA formula:

MLA := 2 * seq_len * (kv_lora_rank + qk_rope_head_dim)

GQA formula:

GQA := 2 * 2 * seq_len * num_key_value_heads * head_dim / TP

In these formulas, seq_len is the key variable determining storage size. As generation progresses, seq_len increases continuously, causing storage requirements to grow accordingly.

4. Deriving the Crossover Sequence Length

By comparing the two sets of formulas, an interesting engineering observation emerges: at short sequence lengths, linear attention may not use less memory than MLA or GQA because it needs to store fixed-size state matrices and convolution kernel history. However, once the sequence length crosses a certain threshold, the constant-size advantage of linear attention becomes evident and standard attention storage is quickly surpassed.

The following examples use two common model architecture configurations, assuming TP=1.

Case 1: A model combining GDN and GQA

Assume the GDN parameters are: linear_num_key_heads=16, linear_num_value_heads=32, linear_value_head_dim=128, linear_key_head_dim=128, linear_conv_kernel_dim=4. The GQA parameters are: num_key_value_heads=2, head_dim=256.

By substituting these parameters into the formulas and plotting cache growth against sequence length, the crossover sequence length for this model is approximately 536 tokens. This means that when the context exceeds 536 tokens, the storage advantage of the GDN module begins to appear and the gap continues to widen.

Case 2: A model combining KDA and MLA

Assume the KDA parameters are: head_dim=128, num_heads=32, short_conv_kernel_size=4. The MLA parameters are: qk_rope_head_dim=64, kv_lora_rank=512.

Following the same calculation, the crossover sequence length for this architecture is also approximately 536 tokens.

These two cases reveal a key conclusion: the storage advantage of linear attention is not absolute in all scenarios. It becomes overwhelming only in long-sequence inference. In short-conversation scenarios, its fixed state overhead may even become a burden. However, for tasks such as long document parsing and large-scale code generation, linear attention can reduce memory usage from O(n) to O(1), making unbounded-context inference more feasible.

5. Summary and Deployment Recommendations

For developers optimizing LLM inference, understanding the underlying storage logic of different attention mechanisms is essential for architecture selection. If deploying services centered on long-text processing in cloud container platforms or elastic compute instances, prioritizing models built on linear attention architectures can significantly reduce KV Cache memory pressure and support higher concurrent throughput.

However, in short-context, high-frequency Q&A scenarios, traditional GQA or MLA mechanisms may have a more compact cache layout because the crossover point is not reached. In practice, developers should evaluate the typical sequence length distribution of their workloads, memory quotas, and whether speculative inference is enabled to select the most appropriate model architecture and inference configuration for the best balance of performance and cost.