Reducing Million-Token Context Inference Costs Through Cross-Layer Index Reuse

    As context windows reach one million tokens, attention-related computation and memory bandwidth can dominate inference costs even when model size remains unchanged. This article explains how sparse attention and latent KV compression work together, why per-layer indexing becomes a bottleneck, and how cross-layer index reuse can reduce redundant top-k selection. It also covers complexity, cache and positional-encoding alignment, implementation strategies, quality trade-offs, and an evaluation checklist for production deployment.

    1. Why Long-Context Inference Is Expensive

    With model size remaining stable, rapidly expanding context windows are becoming a primary driver of inference cost. Recent foundation-model architectures commonly build on two established ideas: multi-head latent attention and top-k-based sparse attention. Cross-layer index sharing recombines these ideas to reduce the cost of million-token inference without changing the basic principle of sparse attention.

    1.1 Cost Is Shifting from Parameter Count to Context Length

    In a conventional Transformer, every generated token attends to the key-value cache for the entire history. The computation therefore grows with context length. In long-context scenarios, the model may have the same number of parameters, while the number of tokens processed per request grows by tens or even hundreds of times. As a result, attention—not parameter count—becomes the dominant cost center.

    1.2 The Double Pressure of Dense Attention

    • Compute pressure: The attention score matrix has size (n \times n), resulting in (O(n^2)) complexity. The model must consider every historical token, even though only a small subset usually makes a meaningful contribution.
    • Memory bandwidth pressure: During token-by-token decoding, attention is often limited by GPU memory bandwidth. Every generated token requires reading the entire KV cache, even when most of the cached values have little relevance.

    1.3 An Illustrative Scale Estimate

    Consider a hypothetical setup. Suppose each layer stores a 512-dimensional KV representation per token using 8-bit quantization, and the model has 60 layers. The KV cache for one token is approximately:

    [
    512 \times 60 = 30\text{ KB}
    ]

    Context length Total KV cache KV data read per generated token
    32K tokens Approximately 0.94 GB Approximately 0.94 GB
    1M tokens Approximately 30 GB Approximately 30 GB

    At a one-million-token context length, even if the entire cache fits in GPU memory, a single request must read approximately 30 GB from memory for every generated token. With an effective bandwidth of 3 TB/s, the ideal lower bound for this operation is about 10 ms, corresponding to a maximum of roughly 100 tokens per second for a single decoding stream. This is only a theoretical limit; real performance is lower.

    1.4 Two Optimization Paths and One Key Assumption

    Most optimization efforts follow two directions:

    • Reduce the KV size per token: Examples include multi-query or grouped-query attention, latent KV-cache compression, and KV quantization.
    • Reduce the KV range accessed for each token: This is the goal of sparse attention.

    Both approaches rely on compression or sparsity based on importance estimation. Their shared assumption is that the actual contribution of many tokens to the current query is close to zero. The rest of this article builds on that assumption.

    2. Core Components: Sparse Attention and Latent KV Compression

    2.1 Sparse Attention: Introducing Top-k Selection

    The central observation behind sparse attention is that attention distributions are often highly concentrated. A small number of tokens carry most of the relevant information, so each query does not need to attend to the entire history.

    A typical implementation has two stages:

    1. Lightweight indexer: A low-dimensional representation is used to calculate relevance scores between the query and candidate tokens, usually aggregated into blocks. The highest-scoring (k) positions are selected.
    2. Sparse attention: Standard attention is computed only over the selected positions.

    If each block contains (b) tokens and (k) blocks are selected, the attention workload for one query is reduced from (n) positions to (k \times b) positions. At million-token context lengths, (k \times b) is often only a small fraction of (n), significantly reducing both attention computation and KV reads.

    2.2 Latent KV Cache Compression: Making Each Read Smaller

    Sparse attention reduces how much data must be read. Multi-head latent attention (MLA) reduces how large each data unit is.

    Its basic approach is:

    • Low-rank compression: Project each token’s key and value representations into a lower-dimensional latent vector. The cache stores the latent vector along with a small position-aware key representation.
    • Reconstruction during attention: During attention computation, the original per-head keys and values are reconstructed through up-projection matrices. Through matrix associativity, the up-projection can be fused with the query and output-side weights, avoiding repeated reconstruction for every token.

    The benefit is not limited to lower memory consumption. The amount of data transferred during every KV read also decreases. When latent KV compression and sparse attention are combined, each token has a smaller KV unit and fewer units need to be read, producing a multiplicative reduction in memory pressure.

    2.3 Positional Encoding and Long-Context Extrapolation

    • The latent representation should be decoupled from positional encoding: Latent compression packs content into a low-dimensional space. If positional information is stored in the same representation, the two signals can interfere. A small position-aware key is therefore retained separately.
    • Long-context extrapolation requires dedicated design: Attention scores depend on positional encoding, so positions beyond the training length require a dedicated extrapolation strategy to keep scoring stable. This is particularly important for the indexer. If it selects the wrong candidates, all subsequent layers can only compute attention within an incorrect candidate set.

    3. The Key Bottleneck: Repeated Indexing at Every Layer

    Sparse attention is often described as reducing (O(n^2)) attention to (O(nk)). That statement applies only to the main attention operation. The complete cost must also include the indexer:

    • Per-layer computation: In a standard implementation, the indexer runs independently at every layer and scores all query tokens.
    • Full candidate scoring: The indexer calculates relevance between each query token and all candidate blocks. This operation itself remains approximately quadratic with respect to context length.
    • Small-kernel overhead: Scoring and top-k selection involve many relatively small operations. With limited parallelism, kernel launch overhead and inefficient memory access can become significant.

    3.1 Indexing Cost Scales Linearly with Depth

    Let (L) be the number of layers and (C_{\text{idx}}) the indexing cost for one token. The indexer alone incurs:

    [
    L \times C_{\text{idx}}
    ]

    The deeper the model, the more often this fixed cost is paid.

    3.2 The Indexer Cannot Be Treated as Negligible

    For long sequences, indexer cost can be comparable to the main attention cost. Assume each block contains (b) tokens. The indexer touches approximately (n/b) blocks for each query, while sparse attention touches only (k) blocks. Even if the indexer uses a lower scoring dimension than the attention operation, the two workloads can become similar once (k) reaches several dozen.

    Per-layer indexing also has a quality-related role. Different layers and attention heads naturally focus on different parts of the context: shallow layers often emphasize local syntax, while deeper layers may rely more heavily on long-range information. A unified sparse pattern can force some heads to operate on a suboptimal candidate set. Whether cross-layer sharing is appropriate must therefore be evaluated against the target workload.

    3.3 Attention Patterns Are Often Correlated Across Layers

    Despite this, per-layer indexing contains substantial redundancy. Adjacent layers derive their queries from the same evolving residual stream, so their attention distributions are often similar. A small number of retrieval-oriented heads may repeatedly focus on the same positions. Engineering observations also show considerable overlap between the tokens selected by different layers.

    If several layers want to inspect largely the same tokens, recomputing the selection from scratch at every layer may simply repeat the same work (L) times.

    4. Core Mechanism: Cross-Layer Index Sharing

    4.1 Design

    Let (g) be a fixed interval between index refreshes, typically set to 4. The model is divided into groups. The first layer in each group runs the complete indexer, while the following (g-1) layers reuse the selected indices and skip scoring and top-k selection.

    The attention computation in every layer remains sparse: each query still attends only to the selected tokens.

    4.2 Illustrative Pseudocode

    1. GROUP_SIZE = 4 # The grouping interval can be tuned between 2 and 8
    2. def forward_with_index_sharing(hidden_states, layers):
    3. shared_index = None # Block-level top-k indices for the current group
    4. for layer_id, layer in enumerate(layers):
    5. # 1) Group-leading layer: run the full lightweight indexer
    6. # and select top-k blocks for all query tokens.
    7. if layer_id % GROUP_SIZE == 0:
    8. # q_idx/k_idx: low-dimensional query/key projections
    9. # Returns indices with shape [num_heads, seq_len, top_k]
    10. shared_index = layer.indexer(
    11. hidden_states,
    12. shared_index=None
    13. )
    14. else:
    15. # 2) Later layers in the group reuse the index from
    16. # the group-leading layer.
    17. shared_index = layer.indexer(
    18. hidden_states,
    19. shared_index=shared_index
    20. )
    21. # 3) Every layer still performs sparse attention:
    22. # gather indexed blocks from the compressed KV cache,
    23. # run sparse attention, then apply the residual connection and FFN.
    24. hidden_states = layer(
    25. hidden_states,
    26. block_index=shared_index
    27. )
    28. return hidden_states

    4.3 Progressive Sharing by Depth

    A more conservative approach gradually reduces the index refresh frequency with depth:

    1. Compute independent indices for each shallow layer.
    2. Share an index every two layers in the middle section.
    3. Share an index every four layers in deeper sections, or compute it only at selected boundary layers in the most aggressive configuration.

    The scheduling policy can be generated from layer configuration:

    1. def build_index_plan(num_layers, deep_group=4, mid_group=2):
    2. """index_plan[i] specifies the index strategy for layer i:
    3. 'own' means compute an independent index,
    4. 'recompute' means refresh the shared index,
    5. 'reuse' means reuse the previous layer's index.
    6. """
    7. index_plan = []
    8. for layer_id in range(num_layers):
    9. if layer_id < 3 or layer_id == num_layers - 1:
    10. index_plan.append("own")
    11. else:
    12. group = (
    13. deep_group
    14. if layer_id >= num_layers * 2 // 3
    15. else mid_group
    16. )
    17. index_plan.append(
    18. "recompute" if layer_id % group == 0 else "reuse"
    19. )
    20. return index_plan

    This design preserves fine-grained context awareness in shallow layers while reducing deep-layer indexing overhead to approximately one quarter of the original cost.

    4.4 Complexity Analysis

    Let (L) denote the number of layers, (n) the context length, and (C{\text{idx}}) and (C{\text{attn}}) the per-layer indexer and sparse-attention costs.

    Strategy Total indexer cost Total sparse-attention cost
    Independent indexing at every layer ((g=1)) (L \times C_{\text{idx}}) (L \times C_{\text{attn}})
    Shared index every four layers ((g=4)) ((L/4) \times C_{\text{idx}}) (L \times C_{\text{attn}})
    Extreme sharing, indexing only once (1 \times C_{\text{idx}}) (L \times C_{\text{attn}})

    The indexing cost decreases approximately in proportion to (1/g), while the main attention cost remains unchanged. This leads to several conclusions:

    • The benefit grows with context length: As the context becomes longer, (C{\text{idx}}) becomes larger relative to (C{\text{attn}}), making the removal of redundant indexing more valuable.
    • Short requests may still need independent indexing: For short contexts, the indexer is already inexpensive. Sharing provides limited savings but can introduce approximation error.

    A depth-aware sharing policy follows directly from these observations.

    5. System Co-Design: Aligning Indices, KV Caches, and Position Encoding

    5.1 Align Block Indices with Paged Cache Granularity

    The indexer usually returns block-level positions, while production inference systems commonly use paged KV caches. These caches divide storage into fixed-size pages that can be allocated and evicted on demand and can support prefix sharing.

    The block size and page format must be aligned. An index returned by the indexer should map directly to a physical cache page, avoiding additional lookup and reordering overhead.

    5.2 Incremental Decoding and Causal Visibility

    Cross-layer sharing also applies to incremental decoding:

    • Update the index at every decoding step: Each newly generated token must participate in index scoring so that later steps can access the latest content.
    • Preserve causal constraints: When an index is reused, a query must still see only its own position and earlier tokens. During training with parallel computation or chunked prefill, indices beyond the query position must be masked.
    • Coordinate with context parallelism: When a long sequence is distributed across multiple devices, top-k selection must be reduced over the global candidate set. Independent local selection on each device can produce locally optimal but globally suboptimal results.

    5.3 Prefill and Decoding Have Different Workloads

    During prefill, both the indexer and main attention are typically compute-intensive and benefit from large-scale parallelism. During decoding, the number of queries per step is small, and both operations become more memory-bound. Latency then depends heavily on the amount of KV data read per step.

    The two phases should therefore be tuned separately and may use different sharing intervals.

    5.4 Quality Costs of Approximation

    Cross-layer sharing is an approximation. Layers that reuse an index can calculate attention only within the candidate set selected by the group-leading layer. It therefore introduces a trade-off between efficiency and accuracy.

    • Sensitive workloads: Long-document retrieval and multi-hop reasoning across distant sections require highly accurate position selection, so errors are more likely to be amplified.
    • Less sensitive workloads: Long-conversation continuation and long-code understanding may be less sensitive to individual positions and often experience smaller quality changes.

    The sharing interval must be selected through workload-specific evaluation rather than applied universally.

    6. Engineering Considerations

    1. Keep training and inference behavior aligned: If inference uses cross-layer index sharing, training should use the same or a similar indexing pattern whenever possible. This reduces behavioral drift caused by training with precise per-layer indexing but deploying approximate reuse. Existing models can introduce the mechanism during post-training alignment.

    2. Optimize small-scale parallelism: The indexer consists of many small operations. In production serving, operator fusion, request batching, and appropriate parallel partitioning can reduce kernel-launch and scheduling overhead. In some implementations, the indexer and attention can share or fuse input projections.

    3. Configure separate sharing intervals for prefill and decoding: Prefill may tolerate more aggressive reuse, while decoding may benefit from shorter sharing intervals to keep output quality stable.

    4. Build observability into the serving system: Track time to first token, per-token generation latency, and the indexer’s share of total inference time. When deploying inference services on a cloud platform, container orchestration can support elastic scaling, while logging, monitoring, and alerting can provide dashboards and thresholds for capacity planning and cost analysis.

    5. Provide a fallback switch: Make index sharing configurable by model and request context length. If quality degradation appears, the service should be able to return to independent indexing quickly, ideally together with canary or blue-green deployment controls.

    7. Validation Checklist

    Validation area Key question Recommended method
    Quality baseline Do long-context task metrics regress after index sharing? Use a fixed evaluation set to compare independent indexing with multiple sharing intervals.
    Performance benefit How much does indexer time decrease, and what is the end-to-end improvement? Measure prefill and decoding separately.
    Sensitivity analysis Which layers and heads are least tolerant of index reuse? Replace indexing incrementally by layer to identify sensitive components.
    Workload segmentation Should long and short requests use different policies? Report latency and quality by context-length range.
    Stability Are indices consistent with paged caches and context parallelism? Validate physical-page mapping and cross-device top-k reduction results.

    8. Conclusion

    The value of cross-layer index sharing lies in recognizing a form of redundant computation in long-context inference. The attention operation itself cannot simply be removed, but the process of deciding which tokens to inspect does not necessarily need to start from scratch at every layer.

    Together with latent KV-cache compression, block-level sparse selection, and paged cache management, cross-layer sharing forms part of a broader optimization strategy for million-token inference. It is not free: the sharing interval, workload sensitivity, and systems implementation all determine the final benefit.

    For teams planning million-token context capabilities, asking whether indexing must be computed independently at every layer can open more cost-saving opportunities than simply adding compute capacity.