Cross-Layer Index Sharing for Long-Context Models: Architecture and Cost Analysis

    This article analyzes cross-layer index sharing as a cost-reduction technique for long-context language models. It explains why sparse-attention indexers become expensive at million-token scale, describes how index sharing can be layered on top of multi-head latent attention and sparse attention, outlines an illustrative pseudocode workflow, and discusses benefits, risks, implementation considerations, and evaluation practices.

    1. Background: Cost Bottlenecks in Million-Token Context Inference

    As large model applications expand from single-turn question answering to codebase understanding, long-document analysis, multi-turn agents, and retrieval-augmented generation, context length is moving from thousands of tokens toward millions of tokens. Longer context is not simply a matter of reading more text. It creates three pressures on the inference system: growth in attention computation, increased key-value cache usage, and additional overhead from sparse selection mechanisms. Traditional dense attention must compute relevance against all historical tokens in every layer. As sequence length grows, both computation and memory bandwidth costs rise significantly. Even with chunked computation, cache reuse, and batching optimizations, long context can still become a major bottleneck for inference throughput and latency.

    To mitigate this problem, a common industry approach combines multi-head latent attention with sparse attention. Multi-head latent attention reduces cache size and memory access pressure through low-rank compression or latent-space mapping of key-value representations. Sparse attention avoids having every query attend to all historical tokens. Instead, an indexer or scorer selects the top-k most relevant tokens, and attention computation is performed only on the selected set. Combined, these methods can keep costs acceptable for longer contexts. However, sparse attention is not free. To select top-k tokens, each layer typically runs an indexer to compute relevance scores between the query and candidate tokens. When context reaches millions of tokens, with many layers and large batch sizes, the indexer’s own computation and memory-access overhead can accumulate quickly and even become a new bottleneck.

    Therefore, the key to long-context architecture optimization is not only sparsification, but also how to generate sparse indices at lower cost. Cross-layer index sharing is an optimization approach aimed at this stage.

    2. Architectural Inheritance: Coordinating Multi-Head Latent Attention and Sparse Attention

    From an architectural design perspective, a class of long-context models inherits existing multi-head latent attention and sparse attention schemes. Multi-head latent attention primarily addresses key-value cache storage and bandwidth. Rather than storing full high-dimensional key-value representations for each head, it reduces cache dimensionality through latent representations and then restores or maps them into the required space during attention computation. This can reduce memory footprint and memory access during long-context decoding without an obvious loss of representational capacity.

    Sparse attention primarily addresses computation scope. For each query, only a small number of the most relevant tokens participate in attention computation. The selection process usually relies on a lightweight indexer that scores the query and candidate tokens, then takes the top-k. In this way, the attention matrix is no longer a fully dense matrix but is composed of multiple sparse blocks or sparse index sets. Multi-head latent attention and sparse attention can be combined orthogonally: the former optimizes what to store and how to read it, while the latter optimizes which positions to compute.

    However, this combination still has a hidden cost: the indexer must run again in every layer. For short contexts, indexer overhead may be masked by the main attention computation. For million-token contexts, the number of candidate tokens is enormous. Even a lightweight indexer can become significant when multiplied by layers, batch size, and sequence length. The core of cross-layer index sharing is to reduce repeated index computation.

    3. Core Idea of Cross-Layer Index Sharing

    The basic assumption behind cross-layer index sharing is that across several consecutive layers, the model’s judgment of which historical tokens are more important is strongly correlated. Shallow layers may focus on local syntax and entities, while deeper layers may focus on semantics and task-relevant information. However, between adjacent layers or adjacent layer groups, top-k token sets often overlap substantially. If every layer runs a complete indexer independently, a large amount of similar candidate scoring is repeated.

    The mechanism avoids recomputing the full sparse-attention top-k indexer in every layer. Instead, it periodically runs a complete indexer and lets subsequent layers reuse the selected token indices. For example, a typical strategy uses a period of four layers: every four layers, a complete indexer runs once to produce a shared top-k token index set, and the next three layers directly reuse that index set for sparse attention computation. It is important to emphasize that what is reused is the index set, not attention weights, key-value representations, or outputs. Each layer still retains its own query projection, attention parameters, and feed-forward network, so model expressiveness is not simply compressed into repeated computation of the same layer.

    From a cost perspective, this periodic reuse can reduce indexer calls to roughly one quarter of the original number, using a four-layer period as an example, while preserving the basic idea of sparse attention: attention is still computed only on a small number of tokens. For million-token inference, scoring and ranking over the full candidate set in the indexer is often an important part of long-context cost. Reducing repeated indexing can significantly lower overall inference cost. During training, shared indices can also reduce activation and computation-graph overhead, but training-inference consistency must be considered.

    4. Illustrative Workflow

    The pseudocode below illustrates the basic flow of cross-layer index sharing. The code expresses logic only and does not represent any real interface or parameter setting.

    1. def forward_with_shared_indices(hidden_states, layers, period=4):
    2. shared_topk_indices = None
    3. for layer_idx, layer in enumerate(layers):
    4. # Periodically run the full indexer to generate shared top-k token indices
    5. if layer_idx % period == 0:
    6. shared_topk_indices = layer.build_topk_indices(hidden_states)
    7. # Subsequent layers reuse the index set but use their own attention parameters
    8. layer_output = layer.sparse_attention(
    9. hidden_states=hidden_states,
    10. selected_indices=shared_topk_indices,
    11. kv_cache=layer.kv_cache
    12. )
    13. hidden_states = layer_output
    14. return hidden_states

    In actual implementations, index sharing can be designed at the layer-group, head-group, or block granularity. Coarser sharing granularity brings greater savings but higher quality risk. Finer sharing granularity is more stable in quality but offers lower benefits. Engineering decisions need to balance model scale, context length, batching patterns, and hardware memory-access characteristics.

    5. Benefit Analysis: Why It Reduces Million-Token Context Cost

    First, it reduces repeated indexer computation. An indexer usually needs to score a query against a large number of candidate tokens. Although a single scoring operation may be lightweight, accumulated FLOPs and memory access cannot be ignored with millions of candidates, dozens of layers, and large batch sizes. Periodically running the indexer can significantly reduce this repeated work.

    Second, it lowers memory bandwidth and scheduling overhead. The indexer not only consumes compute but also reads candidate representations and writes or sorts top-k results. Cross-layer reuse can reduce the generation, sorting, and synchronization of intermediate indices, which is especially beneficial during decoding.

    Third, it creates additive benefits with multi-head latent attention. Multi-head latent attention reduces key-value cache size, while cross-layer index sharing reduces sparse-selection cost. The two act on cache and selection respectively and can jointly improve long-context throughput.

    Fourth, it preserves the basic semantics of sparse attention. The mechanism does not degrade to dense attention or remove sparse selection. It changes the frequency at which indices are generated. Therefore, it is easier to combine with existing paged caching, continuous batching, quantized caching, and parallel inference strategies.

    6. Risks and Constraints: Shared Indices Cannot Be Used Unconditionally

    Cross-layer index sharing is not without cost. The primary risk is drift in attention patterns across layers. If some layers need to focus on tokens that differ clearly from those of the previous layer, reusing older indices may miss critical information, leading to degraded quality in long-document question answering, cross-file code references, or multi-hop reasoning. Second, different attention heads may have different attention patterns. If sharing granularity is too coarse, head diversity may be suppressed. Third, training-inference consistency needs validation. If shared indices are used during training, inference should maintain the same period and selection logic. If independent indices are used during training but shared indices during inference, distribution shift may occur.

    In addition, the stability of top-k indices deserves attention. When candidate scores are close, ranking results may jitter. Shared indices can amplify the impact of such jitter. Engineering mitigations include index smoothing, voting within layer groups, periodic full refresh, and retaining independent indices for critical layers. For security and privacy, the mechanism does not change data access boundaries, but shared index caches must still be prevented from leaking context information across requests.

    7. Implementation and Evaluation Recommendations

    At the implementation level, it is advisable to start with moderate periods and run ablations. For example, compare per-layer independent indexing, two-layer sharing, four-layer sharing, and eight-layer sharing in terms of quality and cost curves. At the same time, monitor index hit rate, top-k set overlap across layers, long-text task accuracy, first-token latency, and throughput. For variable-length batching, differences in index length across sequences need to be handled to avoid padding waste caused by shared indices. For cache layout, shared indices can be decoupled from the paged key-value cache, with index lifecycle maintained per layer group to reduce repeated allocation.

    At the evaluation level, coverage should include long-document question answering, multi-document summarization, cross-file code retrieval, and multi-turn agent memory. Report quality, latency, throughput, and memory usage together. Offline analysis can examine index similarity across layers. Online deployment can use canary releases and rollback strategies: when quality metrics are abnormal, automatically switch back to per-layer independent indexing. In deployment, general cloud environments can use container platforms, object storage, and monitoring and alerting to build evaluation and release pipelines, but actual benefits still need to be validated with real models and load tests.

    8. Conclusion

    The value of cross-layer index sharing lies in the fact that it does not change the basic paradigm of long-context sparse attention. Instead, it applies periodic reuse to an easily overlooked cost point: repeated indexer execution. Multi-head latent attention reduces cache cost, sparse attention reduces main computation cost, and cross-layer index sharing reduces selection cost. Together, these three can form a systematic optimization path for million-token inference. Developers should prioritize quality, validate through ablations, and use canary rollback. The sharing period should not be treated as a fixed constant; it should be continuously tuned according to model structure, task distribution, and hardware characteristics.