Cross-Layer Index Reuse for Long-Context Sparse Attention
Long-context inference is often limited not only by attention computation and KV-cache bandwidth, but also by the repeated cost of deciding which historical tokens to read. This article explains how latent KV compression, sparse token selection, and cross-layer index reuse work together. It covers the design trade-offs, implementation invariants, evaluation methods, and engineering practices for reducing repeated index computation without abandoning per-layer attention aggregation.
Long contexts turn attention from a question of whether it can be computed into a question of whether it can be computed affordably. As the sequence grows, every layer must process the same history: it must identify the relevant tokens and then read their keys and values for weighted aggregation. If every layer scans the full history and independently builds its own sparse index, both computation and memory traffic grow with the number of layers and the sequence length.
A more practical direction is not to abandon sparsity, but to separate “which tokens to select” from “how to use them” across layers. The selection can be reused, while aggregation remains layer-specific.
Why Long Sequences Hit the Indexing Bottleneck First
In standard full attention, the current query must compare against the entire available history. Both runtime and memory requirements grow roughly quadratically with sequence length, making million-token contexts difficult to serve within interactive latency budgets.
Sparse attention reduces full scoring to two stages. First, a lightweight indexer scores historical tokens and selects the top candidates. The main attention operation then aggregates only over those positions. As a result, the primary computation and key-value reads scale with the number of selected tokens rather than the total context length.
The cost is concentrated in the indexer itself. Although it is cheaper than complete multi-head attention, it still needs to scan many positions in a very long sequence. A model may also contain many layers. If every layer independently performs a full indexing pass, repeated work can consume a significant share of latency and energy.
More importantly, adjacent layers see the same history that has already been written to the cache. Their queries change, but the positions with high information density often do not completely reorder at every layer. Turning this stability into a reusable intermediate result is the motivation behind cross-layer index reuse.
Combining Latent Compression with Sparse Selection
Multi-head latent attention first compresses keys and values into lower-dimensional latent vectors. During inference, the required per-head keys and values are reconstructed from those latent vectors, significantly reducing the resident historical cache. This addresses the question of what to store and how much storage it requires.
Sparse attention addresses a different question: which positions should a layer actually read?
The two techniques are therefore complementary:
- Latent caching reduces storage and bandwidth per token.
- Sparse selection reduces the number of tokens accessed.
A common combination stores the history in compressed key-value form. The current layer then decompresses or reads only the positions selected by the indexer before performing multi-head aggregation. This separates the memory wall from the compute wall in long-context inference.
There is an important constraint. The latent representation must reliably preserve the geometric relationships needed by attention, and the indexer’s scoring space must be compatible with the key space used by final attention. Otherwise, inexpensive selection may simply select the wrong positions, causing perplexity or task accuracy to degrade before latency improves.
What Cross-Layer Index Reuse Does
Cross-layer index reuse does not change the basic contract of sparse attention. Each layer still aggregates over a limited set of historical tokens rather than falling back to full attention. What changes is the lifetime of the index.
The full indexer does not need to run at every layer. Instead, it can run at a fixed interval—for example, once every several layers—to score the full history and produce the top position indices. The following layers reuse those indices and recompute only the current layer’s query and attention weights over the selected positions.
This can be understood as a two-level pipeline:
- Selection level: relatively expensive and updated less frequently. It produces position indices rather than attention weights.
- Aggregation level: executed at every layer, but it touches only the selected positions. Queries, head-specific projections, and feed-forward computation still update independently at each layer.
Indices are much smaller than a complete attention matrix, so the additional cross-layer transfer is limited. The main savings come from avoiding repeated scans of the long history and repeated generation of candidate scores.
The per-layer attention weights are still determined by the current layer’s query. In other words, the system reuses where to look, not how to attend. When the query distribution has not changed substantially since the last indexing layer, the reused positions can still cover most of the important information. If the task causes a significant attention shift, the next full indexing pass refreshes the candidate set.
The refresh interval is the main capacity-and-cost control. A stride of 1 is equivalent to recomputing the index at every layer, offering the highest accuracy potential at the highest cost. Increasing the stride reduces the number of indexing passes approximately in proportion to the stride, but stale indices may miss positions that have recently become important.
In practice, a small fixed stride is often used as a starting point and then adjusted based on sequence length and task type. It is usually better than forcing every layer to share one selection result. The approach described here—sharing one full index across a fixed number of layers—is simply one setting of this control. The underlying principle remains sparse token selection; the saved cost comes from avoiding repeated indexer execution.
How It Differs from Full-Sequence Caching and Local Windows
A full key-value cache preserves the entire history for reading, but attention can still be expensive. A sliding window restricts attention to nearby tokens and is simple to implement, but it can lose long-distance references and cross-section evidence.
Sparse indexing allows distant but important tokens to be selected. Cross-layer reuse then prevents the system from paying for that long-distance selection again at every layer.
This design is particularly suitable for workloads such as:
- Document question answering and multi-turn conversations, where evidence may be distributed sparsely across a long document.
- Code and log analysis, where only a small number of symbol definitions or error traces are relevant.
- Retrieval-augmented generation, where retrieved passages may be long but each generation step depends on only a small subset.
The approach is less suitable when every layer must perform fine-grained reordering over the entire context, such as some position-by-position comparison tasks. In those cases, increasing the reuse stride may provide limited benefit or may not be worthwhile.
Invariants to Preserve in the Inference Pipeline
Before deploying the mechanism in a service, use observable invariants rather than relying on the assumption that it “should be faster.”
- Index validity: Reused positions must fall within the currently generated or cached range. Variable-length batches, prefix-cache hits, and mid-sequence truncation must not produce stale or out-of-bounds indices.
- Causal consistency: The indexer may see only the history allowed at the current step. Cross-layer propagation must not expose future token positions to an earlier layer.
- Head and batch alignment: In multi-head attention and packed multi-request batches, the index tensor layout must match the paging layout of the key-value cache. Shared indices must not cause histories from different requests to be mixed.
- Numerical-path consistency: Selected positions must use the same normalization and attention implementation at every layer as they do during training or calibration. Correct indices are not enough if the aggregation kernel follows a different numerical path.
The layer-level data dependency can be represented by the following pseudocode. It describes control flow only and is not tied to a specific framework interface:
if layer_id % stride == 0:index = topk(indexer(query, compressed_history), k)save(index)else:index = load_shared_index()output = attention(query, history_at(index))
Here, stride and k are architectural design parameters. The former controls index refresh frequency, while the latter controls the number of tokens read in detail by each layer. Both should be calibrated offline to find an acceptable quality-latency point rather than changed arbitrarily in production without reevaluation.
Evaluating Quality and Cost Together
On the cost side, three curves are usually sufficient:
- The indexer’s share of end-to-end latency.
- Key-value read bandwidth.
- The latency slope as sequence length increases.
When cross-layer reuse is working, indexer time should decrease approximately with the reuse stride. The main attention cost should depend more closely on the number of selected tokens multiplied by the number of layers, rather than on the full context length multiplied by the number of layers.
As sequences become longer, index scanning typically accounts for a larger share of total runtime, so the relative benefit of reuse tends to increase. On short sequences, however, launch overhead and kernel switching may outweigh the savings. Cross-layer reuse does not need to be enabled indiscriminately.
Quality evaluation should not rely only on average loss. Long-distance dependencies should be checked directly:
- Can the evidence supporting an answer still be selected?
- Do multi-hop questions lose a critical sentence within the reuse interval?
- Does the latter part of generation show repetition or forgotten information?
If degradation is concentrated in particular layers, first consider shortening the stride or forcing an index refresh at those layers. Increasing the global number of selected tokens would give back much of the saved read cost.
Training and inference must also be distinguished. If the model was trained with an independent index at every layer but shares indices during inference, the result is an approximate inference strategy. The same evaluation set should compare per-layer recomputation with stride-based reuse. If training already used the same sharing stride, inference should follow it consistently to avoid behavior drift between training and serving.
Without a controlled comparison, a reduction in cost should not be described as unchanged accuracy.
Engineering Considerations for Deployment
First, bind the index lifecycle to the same prefix as the cache lifecycle. When a prefix-cache hit occurs, historical indices may be reused, but newly appended tokens must be included in the next full indexing pass. Otherwise, new evidence can never enter the candidate set.
Second, when requests in a batch have very different sequence lengths, what is shared is the algorithm—not one identical index set for every request. Maintain indices per request and organize key-value data by page. This allows the computation graph to be reused without cross-request contamination.
Third, observability must separate indexer-kernel time from aggregation-kernel time. Otherwise, a change in total latency will not reveal whether the stride or the selected-token count should be adjusted.
Fourth, retain a fallback path that recomputes the index for the current layer. This is essential for determining whether a failure comes from stale indices or an incorrect aggregation implementation. A fast path without a comparable reference path is difficult to debug.
For resource scheduling, long-context workloads are often better isolated from ordinary short requests. Long-context jobs consume bandwidth and cache capacity, while short requests are more sensitive to concurrency and tail latency.
Cold historical segments can be stored in object storage, while near-memory caches retain latent key-value data and the most recent index. Logs and monitoring should record the stride, selected-token count, indexer time ratio, and sampled task quality. These signals make it possible to trace whether a later fallback was caused by the reuse strategy.
Conclusion
Latent compression makes the history smaller. Sparse selection narrows the portion that must be read. Cross-layer index reuse ensures that this narrowing operation does not have to be paid for again at every layer.
Together, these techniques shift the main cost of long-context inference away from scanning the full sequence and toward per-layer aggregation over a small number of positions, while retaining the ability to refresh the candidate set at a configurable stride.
The key to using the approach effectively is not adding another isolated optimization term. It is preserving index causality and bounds, comparing quality before and after reuse on the same evaluation set, and treating the stride as a calibratable architectural parameter rather than an opaque switch.
