Million-Token Context: Efficient Computation and Low-Cost KV Caching

    Supporting million-token contexts requires more than increasing the input limit. This article explains the main decoding and KV cache bottlenecks, estimates cache requirements, reviews sparse, compressed, hybrid, and tiered attention strategies, and outlines training and evaluation practices for long-context systems.

    1. The Real Challenge

    Long-context capability is often simplified to the question of how many tokens a model can accept. For an inference system, however, input length is only the starting point. Every time the model generates a new token, it must read the intermediate representations of all previous tokens. As a result, the cost of each decoding step increases with the context length. If the architecture remains unchanged, expanding the context window from a few thousand tokens to one million causes both computation and memory usage to grow substantially.

    A long-context design should answer three questions:

    1. How much computation is required for each new token during decoding?
    2. How much memory does the historical information occupy, and can it fit on the target hardware?
    3. Can the model reliably use this information across very long distances?

    2. The Two Core Bottlenecks

    2.1 Decoding Computation and Memory Access

    In standard self-attention, the nth token must compute its relationship with all preceding tokens. During training, an entire sequence can be processed in parallel. During inference, however, tokens are generated one at a time, and each step must scan the keys and values of the full history.

    If the context length is L, the attention computation for each layer and decoding step grows linearly with L. At the million-token scale, both the number of multiply-add operations and the amount of data that must be read become substantial, even when the model size remains unchanged.

    A common limitation is memory bandwidth. Each decoding step must read a large amount of KV data from GPU memory, so the compute units may spend much of their time waiting for data. Optimization therefore involves more than reducing the number of multiplications. It must also reduce the amount of data transferred at every step.

    2.2 KV Cache Capacity

    To avoid recomputing historical representations, inference frameworks cache the key and value tensors for every layer. The cache size can be estimated with the following formula:

    1. KV cache size in bytes =
    2. 2 × number of layers × number of KV heads × head dimension
    3. × bytes per element × sequence length × number of concurrent requests

    Assume an illustrative model with 60 layers, 8 KV heads, a head dimension of 128, and 16-bit floating-point storage. For a single request with a context length of one million tokens:

    1. 2 × 60 × 8 × 128 × 2 × 1,000,000
    2. ≈ 2.46 × 10^11 bytes
    3. ≈ 229 GiB

    This is only the cache for one request. As concurrency increases, KV cache usage can quickly exceed the memory capacity of a single GPU. The following code provides a quick estimate for different configurations:

    1. def kv_cache_bytes(layers, kv_heads, head_dim, bytes_per_elem, seq_len, batch=1):
    2. # The factor of 2 accounts for one copy of the key and one of the value
    3. return 2 * layers * kv_heads * head_dim * bytes_per_elem * seq_len * batch
    4. GiB = 1024 ** 3
    5. size = kv_cache_bytes(layers=60, kv_heads=8, head_dim=128,
    6. bytes_per_elem=2, seq_len=1_000_000, batch=1)
    7. print(f"{size / GiB:.1f} GiB") # Approximately 228.9 GiB

    This estimate shows that cache capacity can become a bottleneck before model parameters do. When planning a deployment, calculate the cache budget first, then determine the supported concurrency and maximum context length.

    3. Common Optimization Strategies

    3.1 Sparse Attention

    Sparse attention is based on a simple idea: each query does not need to attend to the entire history. It can access only the tokens relevant to the current computation. Common approaches include sliding windows, block selection, and score-based top-k selection.

    The benefit is direct. If the number of participating tokens is reduced from L to k, both per-step computation and memory traffic decrease accordingly. The trade-off is that the selection mechanism may miss important information. In practice, systems often retain fixed prefix tokens and a recent window, then evaluate recall on the selected tokens to verify that critical information is not discarded.

    3.2 Compressing Historical Representations

    Another approach is to reduce the number of entries that must be stored. Several neighboring tokens can be merged into a compressed block, or their representations can be reduced through low-rank projection or quantization. Compression reduces both cache size and memory traffic.

    The main challenge is information loss. Coarser compression produces a smaller cache but makes distant details less precise. Engineering designs commonly use multiple levels of granularity: recent context is retained at a finer resolution, while distant history is stored more coarsely.

    3.3 Hybrid Attention

    Hybrid attention combines multiple mechanisms within the same model. One common design keeps fine-grained local attention in some layers or branches to handle local syntax and short-range dependencies, while other layers or branches use compressed or sparse global attention to aggregate long-range information.

    The two paths serve complementary purposes, allowing the system to cover a longer context at an acceptable cost. When designing a hybrid architecture, engineers need to determine the proportion of each mechanism, how the layers are arranged, and how information flows between the two paths. These choices should be validated through ablation studies rather than selected purely based on intuition.

    3.4 Tiered Caching and Offloading

    When the cache remains too large, infrequently accessed historical pages can be moved to host memory or a lower storage tier and brought back to GPU memory when needed.

    This approach depends on assumptions about access locality and introduces additional scheduling latency. Online services also need dedicated strategies for cache allocation, reclamation, and fragmentation management.

    4. Training Stability Matters Too

    Long-context support is not only an inference problem. During training, the model must be exposed to sufficiently long sequences. A common approach is to extend the sequence length in stages: first complete most training with shorter sequences, then gradually increase the length while adjusting the scaling parameters of the positional encoding.

    With very long sequences, attention scores cover a wider numerical range and gradients can fluctuate more significantly. Common techniques include normalizing or clipping attention logits, monitoring the norms of outputs at each layer, applying gradient clipping, and mixing a certain proportion of short samples into the training data. These measures help prevent the model from losing its short-range capabilities.

    Training data quality is equally important. Long documents should contain genuine long-distance dependencies. If samples are created merely by concatenating short texts, the model may learn superficial positional signals instead of developing the ability to reason across paragraphs.

    5. How to Evaluate Long-Context Capability

    “Supports one million tokens” describes only the input limit, not how useful the capability is in practice. Evaluation should cover at least the following dimensions:

    1. Time to first token: Longer inputs increase prefill time, so this phase should be measured separately.
    2. Per-token generation latency: This reflects decoding computation and memory traffic and should be tested at different context lengths.
    3. Peak GPU memory and concurrency limits: These measurements determine how many requests can be served simultaneously on the target hardware.
    4. Long-range retrieval accuracy: Insert key facts into very long documents and test whether the model can retrieve them accurately.
    5. Long-document reasoning quality: Tasks such as cross-section question answering and multi-document summarization test whether information is genuinely integrated.

    A simple retrieval test framework can be structured as follows:

    1. def build_needle_case(filler_tokens, needle, position_ratio):
    2. # Insert the key information at the specified relative position