Building Million-Token Context LLMs: Attention Redesign, Data Strategy, and Infrastructure Optimization

    This article examines the engineering path to million-token context LLMs, covering KV cache constraints, layered attention designs that combine compression and sparsity, Partial RoPE and normalization details, staged long-context training data, and infrastructure requirements such as paged KV management and custom kernels. It also outlines four practical questions for evaluating whether a long-context system is production-ready.

    1. Why Million-Token Context Is So Hard: KV Cache Is the First Barrier

    Autoregressive inference in Transformers has an unavoidable cost: to preserve causality during decoding, the Key and Value vectors of every historical token must be cached. This cache is the KV Cache, and its size directly determines how long a conversation the model can “remember.”

    Under Batch=1 and BF16/FP16 precision for the KV Cache, memory usage can be estimated with the following formula:

    1. KVCache = L_ctx × N_layers × N_kv_heads × (d_K + d_V) × 2 bytes

    Plugging different architectures into this formula makes the differences immediately clear. Using a 1,000,000-token context and decimal GB as a rough estimate:

    Architecture Typical Configuration Estimated at 1M Context
    Standard MHA (no GQA) Full KV heads Hundreds of GB; not deployable at all
    GQA / MQA KV heads compressed by several times Still in the range of tens of GB to over a hundred GB
    Multi-layer low-rank compressed KV Further compression of KV dimensions Significantly lower, but long-sequence attention computation remains expensive

    These numbers explain two things. First, simply reducing KV heads through methods such as GQA is still not enough for million-token contexts. Second, the O(N²) complexity of attention computation itself also becomes a bottleneck for extremely long sequences. As a result, mainstream solutions must address both storage and computation, and they typically advance architecture, data, and infrastructure in a coordinated way.

    2. Attention Mechanism Redesign: A Layered Design of Compression and Sparsity

    Recent long-context LLMs show a common trend in attention structure: abandoning a single global attention mechanism and moving toward layered, heterogeneous attention mixtures. A typical approach assigns tokens to different importance levels and processes them with different attention mechanisms.

    2.1 Heavily Compressed Attention: Local High Fidelity

    For tokens close to the current position, models usually preserve higher information fidelity. One common method applies light compression to KV within a nearby window so that attention can still “see” local details clearly. This type of mechanism can be understood as a short-range, high-precision channel: tokens inside the window follow a low-compression path, preserving modeling quality for syntax, local coreference, and adjacent logic.

    2.2 Compressed Sparse Attention: Low-Cost Global Coverage

    For distant tokens, retaining the full KV is both wasteful in memory and adds unnecessary computation. Industry solutions generally use a two-level approach:

    • Compression: merge the KV of a segment of distant tokens into fewer compressed vectors, usually through pooling or a learnable compression projection. Compression ratios can reach dozens of times or higher.
    • Sparsity: perform selective attention over the compressed representation, allowing only truly relevant distant blocks to participate in computation rather than taking dense dot products over all historical tokens.

    Compressed Sparse Attention is a representative design of this kind: coarse-grained compression handles “global scanning,” while fine-grained windows handle “local close reading.” Compared with earlier approaches that relied only on sliding windows, it fills the gap in long-distance dependencies. Compared with linearization schemes for full attention, it preserves retrieval-style precision.

    2.3 Sliding Window Attention and Attention Sink

    Sliding Window Attention limits each token to attending only to neighbors within a fixed window, turning the memory and computation cost of attention from a function of sequence length into a constant-level function. But pure sliding windows have a well-known flaw: information outside the window is permanently lost.

    Common remedies in practice include:

    • Attention Sink: research has found that a small number of “sink” tokens exist in attention distributions, often near the beginning of the sequence, and the model pours a large amount of attention weight onto them. Explicitly retaining the KV of these sink tokens can stabilize long-sequence generation quality at very low cost and avoid sharp performance degradation after early information slides out of the window.
    • Layered windows: apply fine-grained attention to recent windows and periodic sampling or compressed summaries to older history, forming a “pyramid” memory structure.

    With these mechanisms combined, the attention module of a typical long-context model often consists of three parts: fine-grained local attention, long-range compressed sparse attention, and sink token retention. Each plays a distinct role, together reducing KV Cache and computation to a deployable level.

    3. Easily Overlooked Details: RoPE and Norm Placement

    Beyond the attention backbone, two “small changes” at the architecture level have a major impact on long-context extrapolation.

    3.1 Partial RoPE

    RoPE (Rotary Position Embedding) is the current mainstream positional encoding scheme, but full RoPE is not always friendly to extrapolation over extremely long sequences. Partial RoPE applies rotary positional encoding only to a subset of channels in the head dimension, while the remaining channels carry no positional information. Its effects are:

    • Channels carrying position encode relative distance and serve local dependencies.
    • Channels without position degrade into “semantic content” channels and become less sensitive to absolute position.

    This decoupling makes it smoother for the model to extend context beyond the training length and makes the representation of compressed tokens—which do not inherently have a precise position concept—more stable.

    3.2 RMSNorm at Specific Locations

    The placement of normalization layers is also a sensitive point in long-context scenarios. Adding RMSNorm at the input and output of newly introduced modules such as attention outputs, compression projections, and sparse selection can stabilize numerical distributions along compression and sparsity paths and avoid scale drift between branches with different compression ratios. Such changes may look minor, but in long-sequence training they often determine whether training converges stably.

    4. Data and Training Paradigm: Capabilities Are “Fed” Into the Model

    Architecture only provides possibility. The actual long-context capability requires coordination with data and training procedures. Public industry practice usually follows a multi-stage, progressive pattern:

    1. Pretraining stage: focus on short and medium documents to build basic language and knowledge capabilities.
    2. Long-text extension stage: gradually extend sequence length—for example, stepwise from 32K to 128K and then to 1M—while adjusting hyperparameters such as the RoPE base frequency and mixing long documents, multi-document aggregation, and repository-level code context into the data.
    3. Post-training stage: construct long-context instruction data, including ultra-long document QA, cross-paragraph information retrieval (“needle-in-a-haystack” tasks), and long-range multi-turn conversations, so that the model learns to use rather than ignore long-distance information.

    A key insight here is that long-context capability is not the same as being able to process long input; it means being able to precisely locate and use information within long input. Therefore, the core of data construction is the proportion of “retrieval-reasoning” tasks, not simply piling up long-text tokens.

    5. Infrastructure: The Prerequisite for Architecture to Land

    No matter how good an attention design is, if the infrastructure cannot keep up, it remains in papers. Long-context models place concentrated demands on inference systems in three areas:

    • Paged KV management: following the PagedAttention idea popularized by systems such as vLLM, manage KV Cache in paged blocks to eliminate pre-allocation waste and greatly improve memory utilization and concurrency capacity for long sequences.
    • Heterogeneous storage and offloading: store compressed long-range KV at lower precision or offload it to host memory, retrieving it on demand only when sparse selection hits, further extending the context limit a single GPU can serve.
    • Operator optimization for hybrid attention: compression projections, sparse gather, and sink token handling introduce irregular memory access. Dedicated fused kernels are needed to eliminate fragmented computation overhead; otherwise, the compute saved by sparse designs will be consumed by scheduling overhead.

    Notably, architecture and infrastructure constrain each other. The final values of hyperparameters such as compression ratio, block size, and sparsity rate often need to be determined by working backward from memory bandwidth and kernel efficiency.

    6. Conclusion

    Taken together, recent public technical reports and open-source implementations point to a clear main line for achieving million-token context: center on KV Cache; use layered attention reconstruction based on “compression + sparsity + local fine-grained attention” to reduce storage and computation costs; use Partial RoPE and normalization details to ensure stable length extrapolation; use staged long-text training to turn architectural potential into real capability; and finally rely on paged KV management, heterogeneous storage, and custom operators to bring all of this into a deployable inference system.

    For engineering teams, the value of this methodology goes beyond reproducing a specific model. When building or selecting a long-context service, a solution can be evaluated through four questions: How does KV memory grow with context length? Is long-distance retrieval accuracy validated at the task level? Is the data mix for multi-stage training disclosed? Does the inference system support paged KV and efficient execution of sparse paths? The answers to these four questions largely determine whether million-token context capability is a “paper specification” or “production-ready.” As the open-source ecosystem continues to open up, the reproducibility of this technical path is improving rapidly and deserves close tracking by every developer deploying LLMs.