Evolution of Sparse Attention in LLMs: From Coarse-Grained Layers to Fine-Grained Token Selection

From Full Attention to Selective Focus: The Cost Dilemma of Attention Mechanisms

In the architecture design of Large Language Models (LLMs), the attention mechanism is the core component supporting long-range dependency modeling. However, the computational complexity of standard Multi-Head Attention (MHA) grows quadratically with the sequence length. When the context window expands to hundreds of thousands or even a million tokens, the computational and memory overheads become unsustainable. To break through this bottleneck, the industry has explored two primary optimization paths. One is linear attention (such as kernel-based or state-space variants), which reduces complexity to linear by approximating or rewriting the attention computation. The other is sparse attention, which preserves the original semantic computation of attention but uses specific strategies to ensure that each Query only attends to a subset of all Keys, effectively reducing the number of key-value pairs involved in the calculation.

Initially, many models adopted a module-level hybrid strategy, stacking two different types of attention layers within a single model. Some layers used linear attention for the coarse transmission of long-distance information, while others used standard (or gated) MHA for precise local detail modeling. While this coarse-grained division of labor alleviates cost pressure to some extent, its limitation lies in the static allocation of sparsity at the “layer” level. It cannot dynamically adjust “which historical information is currently most important” based on the input data content.

Fine-Grained Sparse Attention: From Layer Selection to Token Selection

Recently, a more dynamic sparse attention approach has garnered widespread attention. The core idea is to introduce a lightweight Indexer that quickly scores all historical key positions and then selects the Top-K most relevant key positions based on these scores. The subsequent attention computation is performed exclusively on these K selected positions. This mechanism is known as fine-grained sparse attention, typified by Gated Dynamic Sparse Attention (Gated DSA).

The architectural evolution of certain flagship open-source LLMs clearly illustrates the rationale behind this technological shift. In early versions, a model employed a hybrid configuration of 69 layers of linear attention (KDA) and 24 layers of Gated Multi-Head Linear Attention (Gated MLA). In the latest Pro version, developers retained the original 69 layers of linear attention but replaced the 24 layers of Gated MLA entirely with Gated DSA. This is not merely a component rename; it represents a fundamental leap in sparse strategy from “inter-layer selection” to “intra-layer dynamic token selection.”

Technical Breakdown: The Collaborative Workflow of Indexer and Top-K

The workflow of Gated DSA can be decomposed into three key phases:

1. Indexing and Filtering: In each layer, the hidden states of the input sequence are first fed into a parameter-efficient Indexer network. This network outputs an attention score for each historical position, reflecting its relevance to the current query position. Subsequently, only the Top-K positions with the highest scores (e.g., K=2048) are retained as candidate key-value pairs.
2. Gating and Weighting: The selected positions do not directly participate in the Softmax computation. Instead, they are first constrained by a gating mechanism. The gating unit typically consists of a linear layer and a Sigmoid activation function, which dynamically adjusts the importance weights of the selected positions based on the current query’s context, suppressing noise with weak relevance.
3. Sparse Computation: Finally, the attention computation is executed only on the filtered and gated sparse subset. Since the number of key-value pairs is strictly capped at K, the theoretical computational complexity of a single layer drops from O(N^2) to O(N * K). When the sequence length N is significantly larger than K, these savings are substantial.

The core advantage of this architecture is dynamic perception: even if a historical position is not a global hotspot, the Indexer can still retrieve it if it is highly semantically relevant to the token currently being generated. Compared to traditional sparse patterns (such as fixed windows or global + local), this approach is much closer to a “content-addressable” attention computation.

Why a Hybrid Architecture is the Current Optimal Compromise

The combination of 69 layers of KDA and 24 layers of Gated DSA reflects pragmatism and a focus on performance balance in model design. The rationality of this compromise is evident in three dimensions:

• Maximized Cost-Efficiency: Although linear attention (KDA) features low complexity, its expressive power is weaker than standard attention when handling complex relational reasoning and fine-grained semantic matching. Gated DSA, while retaining the high precision of standard attention, compresses the参与 scale to a fixed upper limit. Using the lowest-cost linear attention in 69 layers to consolidate the foundational information flow, and high-precision sparse attention in 24 layers to refine key logic, achieves an ideal balance between computational budget and model performance.
• Stable Scaling of Long-Context Capabilities: As the context window continues to stretch (e.g., exceeding 200K), relying solely on linear attention can lead to detail loss during “information distillation.” The Top-K mechanism of Gated DSA provides a “retrospective focus,” allowing the model to actively return to specific positions in the context for secondary confirmation. This significantly mitigates the “lost in the middle” phenomenon in long contexts.
• Predictability of Inference Performance: Because K in sparse attention is a fixed hyperparameter (e.g., 2048), the memory footprint and computation time during the inference phase no longer degrade linearly with context length. This brings immense convenience to deployment flexibility and computational scheduling during the Prefill phase.

Key Considerations from an Engineering Perspective

For R&D teams looking to replicate or adapt this approach in their own models, the following engineering details should be noted:

• Parameter Overhead of the Indexer: The Indexer must be sufficiently lightweight so that its additional overhead does not negate the benefits of sparsification. Typically, this can be implemented using a single-layer MLP or a small matrix mapping based on mean pooling.
• Top-K Selection Strategy: If K is too large, the significance of sparsification diminishes; if too small, critical information may be truncated. In practice, tuning is required based on the specific model scale and target application scenarios (e.g., code generation, long-document QA).
• Integration of Gating and Routing: The Gating mechanism acts as a “soft regulator” for sparse attention. It is necessary to avoid training instability caused by overly strong gating suppression. It is recommended to initialize the gating with smaller values initially to ensure that model performance does not suffer a cliff-drop when the sparse mechanism is introduced.

Technological Outlook: A Convergence Path Worth Watching

The architectural adjustments of these models actually send a crucial signal—in the context of massive parameters and ultra-long contexts, simply “compressing all attention” or “evenly distributing layer types” is no longer the optimal solution. Future evolution will likely no longer revolve around “which attention to use,” but rather “how to enable the model to learn sparse selection at the right time and position.” Fine-grained sparse attention like Gated DSA combines signal retrieval (Indexer) and content filtering (Gating), endowing models with a stronger “selective memory” capability over their own historical processing.

Although specific implementation details (such as the value of K or the Indexer network structure) vary across different models, the “hybrid linear layers + fine-grained sparse layers” approach has become the dominant design paradigm for advanced models processing massive information. This trend indicates that hardware compute bottlenecks will persist long-term, and structural optimization at the algorithm level remains the key engine driving the capability boundaries of LLMs. For practitioners, deeply understanding and mastering this evolutionary logic will be advantageous in future model design and inference optimization.