Million-Token Context Memory: Sparse Attention and Inference Optimization Explained
This article explains why million-token context windows hit KV cache memory limits before compute limits, estimates KV cache size with a worked example, and reviews architectural and engineering techniques for making long contexts practical: hierarchical compression, compressed sparse attention (CSA), heavily compressed attention (HCA), normalization placement, partial RoPE, attention sinks, phased sequence-length training, and the scarcity of long-dependency training data.
Million-Token Context Memory: Sparse Attention and Inference Optimization Explained
Extending a model’s context window from 128K to 1M tokens or beyond usually runs into a memory bottleneck before a compute bottleneck. Standard attention must store Key and Value vectors for every input token, and this cache is called the KV cache. The longer the sequence, the larger the cache, and it grows linearly with sequence length.
For a single request in half precision, where each value takes 2 bytes, the KV cache size can be estimated as:
KV cache bytes = sequence length × layers × KV heads × (key dimension + value dimension) × 2
Consider a set of hypothetical parameters. The sequence length is 1 million tokens, the model has 60 layers, uses 8 KV heads, key and value dimensions are each 128, and the batch size is 1. The result is:
1,000,000 × 60 × 8 × (128 + 128) × 2 ≈ 2.46 × 10^11 bytes, or about 246 GB (about 229 GiB).
These numbers are only an example; actual values depend on the specific model. With traditional multi-head attention, the number of KV heads equals the number of attention heads. If there are 64 heads, the cache expands to about 2 TB, far beyond the memory capacity of a single inference GPU. Grouped-query attention (GQA) reduces this number by lowering the number of KV heads, but at million-token lengths the pressure remains severe.
Supporting million-token contexts therefore requires changing how the KV cache is stored and computed. Tuning hyperparameters alone has limited effect. The techniques below all center on this problem.
Early Mitigation Approaches
The industry tried several approaches in the early days of long-context work. The first was sliding-window attention, which keeps only the Key and Value pairs for the most recent tokens; the cost is that earlier information cannot be accessed directly. The second was a hybrid of global tokens and local windows, where a small number of key tokens can see the entire sequence while other tokens see only nearby content. The third was state space models, which replace a growing cache with a fixed-size hidden state.
Each method involves trade-offs. Sliding windows lose long-distance dependencies. State space models are not stable enough on tasks that require precise retrieval, such as code understanding and multi-step reasoning. As a result, hierarchically compressed attention became a route worth attention, and some long-context models have adopted similar ideas.
Hierarchical compression separates computation by precision level: recent content remains at full precision, while historical content is compressed at a high ratio to retain an overall semantic summary. The model maintains two attention paths at the same time, and the outputs of the two paths are merged through gating to produce the final result.
This design matches the intuition of reading a long document. When reading the current paragraph, the model needs sentence-level understanding, but for content tens of thousands of tokens earlier, it often only needs to remember the main idea.
Compressed Sparse Attention (CSA)
Compressed Sparse Attention (CSA) is the main mechanism in this type of architecture. It processes historical content in two steps.
The first step is history compression. The model divides historical KV vectors into fixed-size windows and aggregates each block into one vector using lightweight convolution or pooling. The number of compressed vectors is much smaller, so both computation and memory usage decrease. Compression reduces the precision of each block, but these vectors still participate in attention computation.
The second step is proximal exact computation. The most recent tokens—for example, several thousand—keep uncompressed KV so that details in the immediate context are not lost. This part is ordinary window attention.
After the two steps are combined, each query token reads two types of information at the same time: nearby details and distant summaries.
Heavily Compressed Attention (HCA)
Compression introduces a problem: important information may be spread across different compressed blocks, and coarse-grained aggregation can mix it together. Heavily Compressed Attention (HCA) is a variant that fills this gap, mainly handling “old but important” content in extremely long sequences.
HCA uses a more aggressive compression ratio than CSA, so each compressed vector covers a larger token range. It also includes a routing mechanism: query tokens use relevance scores to select a small number of the most relevant blocks from the compressed vectors for computation. This is close to the idea of vector retrieval in retrieval-augmented generation (RAG), but it happens inside the attention layer and does not require an external vector database.
The specific scoring method and number of selected blocks depend on the model implementation and should be checked against the corresponding technical report or open-source code.
Engineering Detail 1: Where Normalization Is Placed
Compression changes the distribution of activations. If the original normalization placement is reused directly, training may become unstable. Some implementations add a normalization layer at both the input and output of the compression path to correct the distribution. This change is simple, but its benefit depends on the specific implementation and data. This article does not provide a universal convergence-speed number; readers who want to cite the effect should consult the ablation experiments in the corresponding paper.
Engineering Detail 2: Partial Rotary Positional Encoding (Partial RoPE)
Rotary positional encoding (RoPE) injects position information into attention queries and keys through rotation matrices. Partial RoPE applies rotation only to part of the dimensions in each attention head, while the remaining dimensions carry no position information. For example, in a 128-dimensional head, only the first 64 dimensions are rotated.
Dimensions without position information can carry pure semantic matching, which may help with long-distance retrieval. Compressed tokens are also handled differently from ordinary tokens in terms of position. A compressed vector represents a range, so its position signal is usually provided in a simplified form or handled by a global token; the exact approach varies by implementation.
Engineering Detail 3: Attention Sinks
The 2023 StreamingLLM paper observed that the first few tokens in a sequence absorb a large amount of attention weight regardless of their semantic content. These tokens are called attention sinks. In long sequences, attention distributions can become unstable without a stable anchor. In engineering practice, the first few tokens are often preserved as global tokens to stabilize the attention distribution.
Training: Extending Sequence Length in Stages
Training directly at million-token lengths is expensive and unstable. A common approach is to proceed in stages. First, complete the main pretraining with short sequences from 4K to 32K to learn language ability and knowledge representation. Then switch to a medium length of about 128K for long-text adaptation training. Finally, fine-tune at 1M length. This progressive approach is usually more stable than starting with the full sequence length from the beginning.
Training Data: The Scarcity of Long-Dependency Samples
Long-context capability depends on data. A model learns to use distant information only if it has seen cross-segment dependencies during training. Such samples are essential. The central requirement is that training data expose the model to dependencies spanning segments; otherwise, long-context capability will not be learned reliably.
