Cross-Layer Index Reuse in Sparse Attention for Long-Context LLM Inference

    This article explains cross-layer index reuse, a way to cut the cost of the indexer in top-k sparse attention for long-context large language model inference. It covers the inference bottlenecks behind the technique, a baseline that combines multi-head latent attention with top-k sparse attention, the core idea of sharing candidate token indices across layers, a pseudocode sketch of the execution flow, the expected benefits and risks, and engineering considerations for production deployment.

    1. Background: The Bottleneck in Long-Context Inference Is Not Only Attention Computation

    As large language models handle longer contexts, inference cost is usually split into two parts. The first is the attention computation itself. The second is the memory and bandwidth pressure from the key-value (KV) cache. Attention cost grows quickly with context length, while KV cache pressure keeps increasing in batched inference, long conversations, and long-document scenarios. To control costs, common approaches include compressing KV representations, using paged cache management, introducing sparse attention, and restricting attention to a small set of candidate tokens.

    The core idea of sparse attention is simple. Instead of having each query token attend to all historical tokens, a lightweight indexer first scores the historical tokens for relevance and selects the top-k candidates. Attention is then computed only over those candidates. This reduces the work from full comparison to sparse comparison while keeping access to key information.

    However, the indexer is not free. If every layer recomputes query-key relevance scores and performs sorting or approximate selection, the repeated overhead grows with network depth and context length. For inference over million-token contexts, the indexer can shift from a lightweight auxiliary module to a significant share of latency and compute cost. Cross-layer index reuse is an optimization aimed at this tension.

    2. Baseline Architecture: Combining Multi-Head Latent Attention with Top-k Sparse Attention

    A typical long-context architecture uses both multi-head latent attention and top-k sparse attention. Multi-head latent attention compresses key-value information through low-rank or latent representations, which reduces cache size and memory access pressure. Sparse attention uses an indexer to select candidate tokens, which narrows the attention range. Together, they keep memory use and compute cost manageable over longer contexts.

    In this combination, the indexer answers one question: for the current query at the current layer, which historical tokens should it attend to? The indexer usually scores the relevance of query and key representations and keeps the highest-scoring candidates. Subsequent attention runs only over those candidates.

    The indexer’s output is a candidate set, not the final attention weights. Each layer still computes its own attention distribution from its own queries, keys, and values.

    Sparse attention quality depends heavily on the indexer’s recall. If the indexer misses critical evidence, no amount of precision in the later attention step can recover it. For this reason, the indexer should not be oversimplified, and it should not be shared across layers without care. The key to cross-layer index reuse is balancing the reduction of repeated computation against the quality of the candidate set.

    3. Core Idea of Cross-Layer Index Reuse: Share Candidates, Not Attention Outputs

    The basic approach is to stop running a full indexer at every layer. Instead, a full indexer runs every few layers, selects the top-k candidate token indices, and the following layers reuse those indices. A typical design runs the full indexer every four layers. The intermediate layers use the candidate set selected by the most recent indexer run.

    Three boundaries need emphasis.

    First, the technique reuses candidate token indices, not attention weights. Each layer still computes attention with its own queries, keys, and values. Only the range of historical tokens taking part in the computation is limited to the set that the shared indices point to.

    Second, reuse does not change the basic principle of sparse attention. The model still computes attention over a small number of candidate tokens. The only change is that the indexer’s schedule moves from once per layer to once every several layers.

    Third, the reuse interval is a design parameter. Every four layers is a typical compromise. Real systems can adjust the interval based on differences in attention patterns across layers, the task type, the context length, and quality requirements. A longer interval reduces indexer calls and increases the potential savings, but it also raises the risk that the candidate set drifts away from what the current layer actually needs.

    The intuition is that adjacent layers often have related attention patterns. Shallow layers may focus on local syntax and nearby tokens. Middle layers may focus on entities and paragraph structure. Deeper layers may focus on task-relevant evidence. Adjacent layers are not identical, but their candidate sets often overlap substantially. As long as the sharing interval is not too long, later layers that reuse the candidates from the previous indexer run usually still cover most of the key information. For a small number of layers that are sensitive to indexing accuracy, the full indexer can be kept, or a shorter reuse interval can be used.

    4. Execution Flow and Pseudocode

    From the execution perspective, cross-layer index reuse works as a loop that periodically refreshes the index and then runs sparse attention within each layer. The following illustrative pseudocode focuses on the logic rather than specific interfaces:

    1. cache_topk = None
    2. reuse_interval = 4
    3. for layer_id, layer in enumerate(layers):
    4. q, k, v = layer.project(hidden_states)
    5. if layer_id % reuse_interval == 0 or cache_topk is None:
    6. scores = layer.indexer(q, k)
    7. cache_topk = topk_select(scores, k=sparse_k)
    8. else:
    9. # Reuse the candidate token indices selected by the most recent full indexer run
    10. pass
    11. attn_out = sparse_attention(q, k, v, indices=cache_topk)
    12. hidden_states = layer.update(hidden_states, attn_out)

    In this flow, the full indexer runs at a fixed layer interval, and the following layers read the candidate indices directly. In practice, the implementation must also handle several issues: different sequence lengths within a batch, cache invalidation, changes in representation scale across layers, and consistency between training and inference. For example, in batched inference, different samples have different effective context lengths, so shared indices must be maintained per sample. When sequences are truncated or processed in chunks, the index cache also needs a clear lifecycle.

    If training uses a similar mechanism, the indexer and the main attention path must be optimized jointly. A common approach keeps the indexer differentiable or uses a trainable approximate selection strategy. Training can also simulate the reuse interval to avoid noticeable distribution shift at inference time.

    5. Benefit Analysis: Reducing Repeated Indexing Overhead

    The direct benefit of cross-layer index reuse is fewer full indexer calls. If every layer runs the indexer, the total cost of scoring, sorting, and candidate selection grows with the number of layers. Refreshing the index every four layers reduces the call frequency significantly. For million-token contexts, this reduction matters because the indexer itself must process a large number of candidate tokens.

    Beyond compute cost, reuse can bring several additional benefits:

    1. Lower latency jitter. Indexers often include sorting or top-k selection. Over long contexts, these operations can create synchronization points and memory access pressure. Fewer calls simplify inference scheduling.

    2. Less repeated memory access. The indexer reads key or latent representations, so frequent calls add to memory bandwidth pressure. Cross-layer reuse keeps some index results in cache for longer.

    3. Better economics for long-context inference. On the same hardware, lower indexer overhead leaves more budget for attention computation, batching, or longer context windows.

    These benefits are not unconditional. A longer reuse interval means the index cache must cover more layers, and the candidate set may deviate more from what the current layer needs. The benefit analysis therefore has to go together with quality evaluation. Reduced indexer call counts alone do not show whether the approach works.

    6. Quality and Risk: Controlling Attention Drift Across Layers

    The main risk of cross-layer index reuse is attention distribution drift between layers. Different layers may focus on different historical information. Some layers need a local window, some need distant entities, and some need task instructions. If the shared index covers only the candidates that a previous layer attended to, later layers may miss tokens they actually need.

    Several measures can control this risk:

    First, set a sensible reuse interval. Every four layers is a common compromise, but it is not a fixed rule. For models with large differences in attention patterns across layers, a shorter interval may be appropriate. For models with high similarity between layers, a longer interval may work.

    Second, keep fallback candidates. In addition to the shared index, the system can retain a local window, special positions, or high-salience tokens. This lowers the risk of missed recall while keeping the attention sparse and giving later layers basic coverage.

    Third, refresh sensitive layers fully. For layers that require high retrieval precision, or for deeper layers with strong task relevance, the system can skip reuse and run the full indexer.

    Fourth, monitor candidate recall quality. In offline evaluation, teams can compare shared indices against per-layer indices on key tasks, measuring candidate overlap, answer hit rate, and performance on long-document question answering. In online services, teams can monitor index hit rate, sparsity, latency percentiles, and quality regression metrics.

    Fifth, maintain training-inference consistency. If training does not simulate cross-layer reuse, enabling it directly at inference may cause distribution shift. A safer approach is to introduce the reuse interval during training or fine-tuning so the model adapts to shared candidate sets.

    7. Engineering Recommendations

    To deploy cross-layer index reuse in an inference system, algorithm design and scheduling implementation should be considered separately. On the algorithm side, the team defines the reuse interval, the number of candidates, the fallback strategy, and quality thresholds. On the engineering side, the team handles index cache allocation, lifecycle management, batch alignment, and monitoring and alerting.

    Specifically, the following engineering points deserve attention:

    1. Maintain index caches per sample. Requests in a batch have different context lengths, so one global index cannot cover the whole batch. Cache keys should include the request ID, sequence position, and layer-interval information.

    2. Define cache invalidation clearly. When a sequence appends new tokens, is truncated, switches conversation turns, or triggers prefix cache reuse, the index cache must be invalidated or updated incrementally to avoid stale candidates.

    3. Reduce synchronization points in parallel scheduling. The full indexer usually includes top-k selection, which can introduce synchronization. Index refreshes can be scheduled during stages with lighter compute, or pipelined with attention computation in adjacent layers.

    4. Evaluate memory and bandwidth together. Reusing indices reduces indexer memory access, but the cached indices also take up space. Teams need to balance the number of candidates, the number of cached layers, and the batch size.

    5. Monitor both quality and cost. Beyond latency, throughput, and memory, teams should track index hit rate, candidate overlap, the actual coverage of sparse attention, and long-context task quality. For rollout, a gradual release with A/B evaluation is advisable. Validate quality on a small amount of traffic first, then expand the reuse interval.

    Deployments on Baidu AI Cloud can draw on general platform capabilities, such as container platforms, object storage, log services, and monitoring and alerting. These can be part of a unified operations system covering index cache management, quality regression testing, and latency monitoring. The key point is not dependence on any specific interface. Cross-layer index reuse should be treated as a configurable scheduling policy in the inference engine. Configurable intervals, gradual rollout switches, and rollback mechanisms help reduce deployment risk.

    8. Summary

    Cross-layer index reuse does not replace sparse attention. It refines how the indexer is scheduled. The technique keeps the basic framework of selecting candidates first and then running sparse attention. It runs the full indexer every few layers and lets later layers share the candidate token indices, which reduces repeated scoring and sorting overhead. Its technical value lies in isolating the repeated indexing cost within long-context inference and optimizing it directly. Reuse intervals, fallback candidates, refreshes for sensitive layers, and quality monitoring keep the risks under control.

    For large language model inference systems that handle million-token contexts, this design offers a practical compromise. It keeps the compute savings of sparse attention without letting the indexer become a new bottleneck. The outcome in practice depends less on the reuse interval itself than on understanding the differences between layers, continuously evaluating candidate quality, and providing full engineering support for caching, batching, and monitoring.