Sparse Attention Index Reuse Across Layers for Million-Token Contexts

    This article analyzes the architecture of cross-layer index reuse for sparse attention in million-token contexts. It explains why the indexer becomes a bottleneck, how a layer-group-based reuse mechanism reduces redundant indexing calls, the feasibility and boundaries of reusing top-k token selections across adjacent layers, and key implementation considerations such as cache structure, masking, and batch handling. The goal is to reduce indexing overhead while preserving the benefits of sparse attention, with a clear evaluation checklist for deployment.

    Why the Indexer Becomes a Bottleneck at Million-Token Scale

    Long-context large language model inference is largely dominated by attention computation. Standard full attention requires every query token to interact with a large number of historical tokens in the sequence. When the context extends to a million tokens, the cost of reading KV caches, computing attention scores, and moving memory bandwidth increases dramatically. Sparse attention is a common optimization: instead of having each query attend to all historical tokens, an indexer (or scorer) selects the top-k most relevant tokens from the sequence, and attention is computed only on those selected tokens.

    The effectiveness of this approach depends heavily on the quality of selection. If the indexer reliably identifies important tokens, the model can maintain good long-context ability with lower computation. However, the indexer itself is not free. It typically requires scoring and ranking candidate tokens, and it is often executed at every layer, for every attention head, or for every query position. Deeper layers and longer contexts make the indexer’s invocation count and scoring overhead increasingly significant.

    In a million-token setting, running a full indexer independently at each layer causes obvious redundant computation: adjacent layers process the same context, and although token representations evolve layer by layer, the distribution of important tokens tends to be highly correlated. If every layer re-runs the complete selection from scratch, a large amount of redundant work is produced. The idea of cross-layer index reuse starts here: instead of running the full indexer at every layer, layers are grouped, and the subsequent layers in a group reuse the token index set selected by the group’s first layer. This preserves the core principle of sparse attention while reducing the frequency of indexer invocations.

    Architecture Overview: Latent Attention Compression, Sparse Selection, and Cross-Layer Reuse

    From an architectural perspective, this type of long-context sparse attention solution typically combines three complementary components:

    1. Latent attention compression: compresses the KV representations using low-rank or latent representations, reducing the cost of caching and reading long sequences. It focuses on “how to store and read historical information more efficiently.”
    2. Dynamic sparse attention: uses an indexer to select top-k historical tokens for each query and computes attention only over those tokens. It focuses on “how to reduce the number of tokens participating in attention.”
    3. Cross-layer index reuse controller: partitions adjacent layers into layer groups, runs the full indexer in the first layer of each group, and reuses the selected token indices in later layers of the same group. It focuses on “how to reduce the redundant computation of the indexer itself.”

    Together, the overall flow can be summarized as follows: after the input passes through representation layers, the latent attention module maintains compressed KV representations. In the first layer of a group, the indexer scores candidate tokens and outputs the top-k indices. The sparse attention module reads the corresponding KV entries and computes attention based on these indices. In later layers of the same group, the full indexer is not run again; instead, the cached top-k indices from the group’s first layer are reused, and each layer uses its own query, key, and value representations to compute attention.

    Note that reusing indices is not the same as reusing attention weights or KV values. The later layers reuse only the set of selected token positions. Attention scores are still computed using the current layer’s query and key. This preserves inter-layer representation differences to a large extent while cutting down the repetitive scoring cost of the indexer.

    Cross-Layer Index Reuse Mechanism: One Full Index Run per Four Layers

    The core design of cross-layer index reuse can be abstracted as a layer-group-index cache. Suppose the model has L layers. We group every G layers into an index-reuse group. The first layer of the group (the group head) runs a full indexer to obtain a top-k token index set. The remaining G−1 layers in the same group directly read that index set and do not re-run the full indexer.

    When G=4, a full indexer is run only once every four layers, and the next three layers reuse the indices selected by the group head. The number of indexer invocations drops from “once per layer” to “once per group.” The approximate number of invocations is:

    1. N_index = ceil(L / G)

    Under the simplifying assumption that the indexer cost is homogeneous, the reduction ratio in indexer calls is approximately:

    1. savings ≈ 1 - ceil(L / G) / L ≈ 1 - 1 / G

    With G=4, the number of indexer calls is reduced to about one-quarter of the original, i.e., about 75% of indexer calls are eliminated. However, this does not mean that the total inference cost is fixed at a 75% reduction. The overall benefit depends on the proportion of indexer time in total inference cost. If the indexer is a significant contributor at million-token contexts, the gain is more visible; if KV reading or attention computation still dominates, the overall benefit must be evaluated against the concrete implementation.

    This mechanism preserves the core idea of sparse attention: attention is still computed only over top-k tokens, and the computation reduction from sparsity remains. The only change is that index selection shifts from “independent per layer” to “shared per group.” The group size G is a critical tuning parameter: smaller G offers better layer adaptability but more indexer runs; larger G lowers indexer cost but widens the reuse span, potentially losing some layer-specific selection ability. Therefore, G should be configurable and chosen based on model structure, task type, and accuracy evaluation.

    Why Reuse Is Feasible: Limits and Conditions

    The intuition behind cross-layer index reuse is that adjacent layers process the same input, and token representations evolve continuously across layers. The set of important tokens tends to have high overlap. For retrieval, long-document QA, code context, and similar tasks, the positions of key information often remain relatively stable across adjacent layers, so the top-k indices chosen by the group head remain valuable for later layers.

    But reuse has boundaries. Different layers may focus on different levels of abstraction: shallow layers may pay more attention to local syntax or positional proximity, while deeper layers may focus on semantic relations or long-distance evidence. If some layers need a token set that is clearly different from what the group head selected, directly reusing the indices can cause recall loss. To mitigate this risk, common engineering strategies include:

    • Adapting the group-internal layers during training or fine-tuning to work with shared indices, reducing train/inference mismatch;
    • Running ablations on group sizes (G=2, G=4, G=8, etc.) to compare accuracy and cost;
    • Monitoring the overlap rate of top-k indices across layers; if overlap drops too low, shorten the reuse span;
    • Running offline evaluations on key tasks, including long-context retrieval, multi-hop QA, code completion, and multilingual scenarios;
    • Keeping a configurable refresh policy that allows certain layers within a group to re-index when accuracy is critical.

    Implementation Considerations: Caching, Masking, and Batching

    Several engineering details are important when implementing cross-layer index reuse.

    1. Index cache structure. The cache typically stores the top-k token ids (or an equivalent mask) for each request, each layer group, and each attention head. The cache size scales with top-k, number of heads, and number of groups, and is usually much smaller than the KV cache. It must be managed per request lifetime. In continuous batching scenarios, the cache must be released when a request finishes, and must not be shared across sequences.

    2. Causal masking and position alignment. Reused indices must align strictly with causal masks, positional encodings, and block boundaries. The index set may only contain tokens visible to the current query. Variable-length sequences and padding regions must not be included in the index selection. If chunked prefill is used, ensure that the token range covered by the group-head indices is identical to that of the later layers.

    3. Prefill vs. decode stages. During prefill, sequences are long and indexer cost is prominent, making cross-layer reuse more beneficial. During autoregressive decoding, the query changes every step, and whether to refresh indices at every step depends on the accuracy policy. A common practice is: within a layer group, cache the latest full index result and reuse it over a certain span; for accuracy-sensitive tasks, shorten the reuse window or add refresh points.

    4. Batching and variable-length sequences. Sequences in a batch may have different lengths. The index cache must be maintained per sample, and padding must not participate in top-k selection. In continuous batching, new requests joining and old requests exiting must correctly initialize and release the layer-group cache.

    5. Coordination with KV cache. Sparse attention reads only the KV entries of selected tokens. Index reuse reduces the indexer’s scoring and ranking overhead, but it does not necessarily reduce the number of selected tokens’ KV reads. Therefore, performance analysis should separately measure indexer time, KV read time, and attention computation time, so that the indexer savings are not mistaken for a linear overall gain.

    The following simplified pseudo-code illustrates the basic logic of layer-group index reuse:

    1. G = 4 # one index-reuse group per four layers
    2. index_cache = {}
    3. for layer_id in range(num_layers):
    4. group_start = layer_id - (layer_id % G)
    5. q = project_q(hidden[layer_id])
    6. k = project_k(hidden[layer_id])
    7. v = project_v(hidden[layer_id])
    8. if layer_id % G == 0:
    9. # group head: run full indexer and select top-k tokens
    10. topk_indices = full_indexer(
    11. hidden[layer_id],
    12. seq_len=seq_len,
    13. top_k=k_tokens,
    14. causal_mask=causal_mask
    15. )
    16. index_cache[group_start] = topk_indices
    17. else:
    18. # later layers in the group: reuse the group head's indices
    19. topk_indices = index_cache[group_start]
    20. # later layers still compute attention with their own q/k/v
    21. attn_out = sparse_attention(
    22. q, k, v,
    23. indices=topk_indices,
    24. mask=causal_mask
    25. )
    26. hidden[layer_id + 1] = update(hidden[layer_id], attn_out)

    This pseudo-code highlights two boundaries: first, the group head establishes the index cache; second, later layers reuse the indices but still compute attention using their own queries, keys, and values.

    Benefits, Risks, and Evaluation Checklist

    The main benefit of cross-layer index reuse is the reduced frequency of indexer calls, which is especially valuable when the indexer accounts for a significant portion of inference cost in million-token contexts. It also reduces repeated scoring over a full sequence or large candidate set, lowering some compute and bandwidth pressure. The index cache itself is typically small, but it adds implementation complexity, including layer-group partitioning, cache lifecycle management, mask alignment, and batch isolation.

    The main risk is accuracy: the larger the reuse span, the more likely later layers will lack layer-specific selection ability. We recommend a progressive rollout:

    • First, compute the overlap of top-k indices across layers offline to judge feasibility.
    • Start with a small group size and gradually increase the reuse span.
    • Evaluate both short-context and long-context tasks separately, avoiding reliance on average metrics alone.
    • Monitor the indexer time ratio, KV read time, top-k hit rate, and task metrics.
    • In a general cloud environment, combine container platforms, logging services, monitoring and alerting, and other common capabilities for canary deployment and performance observation, ensuring that issues are traceable and rollback is possible.

    Overall, the value of cross-layer index reuse lies not in changing the goal of sparse attention, but in replacing “repeated selection at every layer” with “shared selection at the group level.” It reuses the top-k token indices at the granularity of layer groups, preserving the selectivity of sparse attention while significantly reducing the number of indexer invocations. This provides a clear, configurable, and assessable optimization path for inference at million-token scale.