Cross-Layer Index Reuse for Sparse Attention in Large Language Models
As context windows expand toward millions of tokens, sparse attention must reduce not only the cost of attention computation but also the repeated overhead of token scoring and top-k selection. This article explains how cross-layer index reuse groups transformer layers so that a full indexer runs only once per group while subsequent layers reuse the selected token set. It covers the mechanism, cost model, accuracy trade-offs, causal masking requirements, cache and kernel implementation considerations, and evaluation metrics for long-context inference.
As large language model context windows continue to grow, inference over million-token sequences is moving from an experimental capability toward an engineering challenge. The main difficulties are not limited to the increasing size of the attention matrix. Key-value cache reads, memory bandwidth, indexer computation, and scheduling overhead also accumulate.
Multi-head latent attention (MLA) compresses the key-value cache through low-rank latent representations, while sparse attention reduces the main attention workload by selecting only a small number of important tokens. Combining these techniques has become a common approach for extremely long-context architectures.
However, when sparse attention runs independently at every layer, each layer must rescore tokens and perform a new top-k selection. As the number of layers and the sequence length increase, this indexer cost can become significant. Cross-layer index reuse addresses this issue by grouping layers and reusing selected token indices within each group instead of running a complete indexer at every layer. The goal is to reduce long-context inference cost while preserving the basic semantics of sparse attention.
1. Background: What Sparse Attention Saves—and What It Does Not
Sparse attention is based on a simple assumption: for a given query, not every historical token is equally important. Instead of computing attention between each query and the entire history, a lightweight indexer first scores candidate tokens, selects the top-k most relevant ones, and then performs attention only over those tokens.
This changes the main attention workload from a cost that grows quadratically with sequence length to a cost that is approximately linear, or otherwise lower-order, with respect to the number of selected tokens. For million-token inputs, this reduction is critical for keeping computation manageable.
Sparse attention does not eliminate every source of overhead, however. The indexer still needs to score candidate tokens, and top-k selection may involve sorting, threshold comparisons, or approximate selection operations. If every layer independently runs a complete indexer, several types of redundant work appear:
- Token representations in adjacent layers are often strongly correlated, so rescoring them independently can duplicate similar computation.
- Top-k selection over extremely long sequences may become a scheduling bottleneck, particularly in batched and highly concurrent workloads.
- The selected indices must be converted into masks, index lists, or bitmaps for the attention kernel. Rebuilding these structures at every layer adds metadata-management and memory-access overhead.
Cross-layer index reuse is based on the observation that sparse patterns across nearby layers may be reusable. It changes the design from “select tokens at every layer” to “select tokens once per group and share the indices within that group.”
2. Core Mechanism of Cross-Layer Index Reuse
The main idea can be summarized as grouped indexing, intra-group reuse, and sparse execution.
A typical design runs the complete indexer once every several layers. The other layers do not recompute top-k selection; instead, they reuse the token index set generated by the most recent complete indexer. For example, four consecutive layers can form a group. The first layer runs the complete indexer and produces a selected-token set, while the remaining three layers reuse that set to construct their sparse attention masks.
“Once every four layers” is only an example design pattern, not a fixed rule. The group size must be selected based on model depth, task characteristics, sequence length, and accuracy requirements.
In the data flow, the complete-index layer computes relevance scores between the query and candidate keys, selects the top-k tokens, and stores the resulting index set in a cross-layer shared structure. Reuse layers read that index set, construct the sparse attention pattern for their current queries, and then execute attention.
The key distinction is that the mechanism reuses which tokens are selected. It does not reuse attention outputs, nor does it skip attention computation. Reuse layers must still read the relevant key-value cache entries and perform the weighted attention aggregation. Therefore, this method primarily reduces repeated indexer computation and top-k selection overhead; it does not replace the main sparse attention operation.
Illustrative pseudocode:
# N is an illustrative group size and must be determined experimentallyN = 4group_index = {}for layer_id, layer in enumerate(layers):group_id = layer_id // Nif layer_id % N == 0:# Complete index layer: score tokens and select the top-k entriesscores = indexer_score(layer.query, kv_cache)selected_ids = topk(scores, k)mask = build_sparse_mask(selected_ids, seq_len)group_index[group_id] = maskelse:# Reuse layer: reuse the selected token indices for the groupmask = group_index[group_id]output = sparse_attention(layer.query, kv_cache, mask)
This pseudocode highlights three engineering requirements:
- The indexer runs once per group rather than once per layer.
- Reuse layers must be able to read the shared sparse mask or index list.
- The sparse attention kernel still performs the actual attention computation according to the mask.
In practice, selected indices may be represented as index lists, bitmaps, block-level masks, or hybrid structures. The representation should balance metadata memory usage against the read efficiency of the attention kernel.
3. Why This Can Reduce Million-Token Inference Cost
The benefit of cross-layer index reuse comes from changing the cost structure. Let:
- I be the cost of a complete indexer,
- A be the cost of the sparse attention computation,
- R be the cost of reading reused indices and constructing masks.
With independent indexing at every layer, the total cost is approximately:
[
L \times (I + A)
]
where L is the number of layers.
With grouped reuse, if one complete indexer runs for every N layers, the cost is approximately:
[
(L/N) \times I + L \times A + L \times R
]
Because R is typically smaller than I, and N is greater than one, the total indexer cost is amortized across the group. The longer the sequence, the greater the number of layers, and the larger the batch, the more important this amortization can become.
This optimization does not change the basic sparse-attention semantics of focusing on only a subset of tokens, and it does not eliminate key-value cache reads. For million-token contexts, cache capacity and memory bandwidth remain major bottlenecks.
The value of cross-layer index reuse is to reduce repeated indexer computation, decrease the frequency of top-k selection, lower the rate at which index metadata is generated, and potentially improve scheduling stability for long sequences. It is complementary to MLA-based cache compression, paged cache management, and block-sparse computation, so these techniques can be combined.
4. Accuracy and Stability: Reuse Is Not Always Free
Cross-layer index reuse assumes that sparse patterns remain sufficiently similar across layers within the same group. If some layers are highly sensitive to token selection, reusing indices from an earlier layer may omit tokens that are important to the current layer. This can weaken long-range information retrieval.
Several issues should therefore be considered.
Group size
The group size requires a trade-off. Larger groups reduce the number of indexer executions and can provide greater savings, but sparse patterns may diverge more across layers. Smaller groups reduce the accuracy risk but also reduce the performance benefit. In practice, the group size should be selected through offline evaluation and online monitoring rather than fixed in advance.
Index stability
Index stability can be measured by comparing top-k set overlap across layers in the same group, measuring recall for critical tokens, and tracking long-context retrieval performance. A low overlap rate indicates that the reuse assumption may not hold. The system may then need smaller groups or compensation mechanisms such as a local window or an additional candidate set for reuse layers.
Causal masking and positional constraints
Sparse indices must be selected only from legal historical tokens. When a reuse layer constructs its mask, it must apply the current query position and causal constraints to prevent access to future or out-of-range tokens.
In batched inference, different sequences may have different lengths. Index sets must therefore remain isolated across sequence boundaries.
Top-k boundaries and tied scores
Tied or nearly tied scores can cause token selection to fluctuate between layers. Reuse may amplify this bias. Stable sorting, block-level candidate selection, or a small deterministic perturbation may improve consistency, but the behavior must remain compatible between training and inference.
5. Engineering Considerations
Cross-layer index reuse requires coordination between the index cache, mask construction, and attention kernel.
The index cache should be compact. Storing a complete dense mask for every layer is usually unnecessary. Block-level indices or compressed bitmaps can reduce metadata size. When reuse layers read the index set, the layout should be aligned as closely as possible with the key-value cache layout to minimize random access.
For paged or block-managed caches, it is often preferable to organize selected indices at the block level. This allows the sparse attention kernel to load key-value data by block rather than issuing a large number of scattered accesses.
Parallelism and pipelining also require attention. The complete-index layer and reuse layers have a data dependency: reuse layers must wait for the group’s index result. A scheduler can allow complete-index layers to finish earlier, or pipeline the indexer with the attention computation to hide part of the latency.
During autoregressive decoding, the selected index set may need to be updated continuously. A rolling update strategy can be combined with a sliding window or periodic recomputation. This helps prevent the historical index set from becoming stale and drifting away from the current relevance pattern.
Monitoring should include:
- The number of reuse operations per group
- The size of each selected index set
- Sparse-pattern overlap between reuse layers and complete-index layers
- Long-context task accuracy
- Memory-bandwidth utilization
Evaluation should cover different long-context workloads, including retrieval, question answering, summarization, and code understanding. Perplexity alone is not sufficient to assess the quality of this optimization. Cross-layer index reuse is an approximate optimization, so task-level metrics must confirm that its cost savings do not come at the expense of useful long-range information.
6. Conclusion
Cross-layer index reuse provides a practical optimization path for sparse attention in extremely long-context inference. By running a complete indexer once per layer group and allowing subsequent layers to reuse the selected token indices, it reduces repeated scoring and top-k selection while preserving the main computation semantics of sparse attention.
The technique complements MLA, sparse attention, paged caching, and block-level computation, making it a useful design pattern for reducing indexer overhead in million-token inference. Successful deployment depends on measuring index stability, tuning the group size, compressing masks or index metadata, and evaluating end-to-end task performance. The objective is to reduce inference cost without compromising the model’s ability to use distant information.
