LLM Attention Evolution and a Selection Guide for the Long-Context Era
This article traces how attention mechanisms in large language models (LLMs) have evolved since the 2017 introduction of the Transformer and offers a practical framework for choosing among them. As feed-forward networks have converged across model families, the attention layer has become the main area of architectural divergence, giving rise to six technical routes: gated linear attention, linear-plus-global hybrid stacking, dynamic sparse attention, compression-plus-sparse fusion, sliding-window localization, and state-space model hybrids. The article explains the cost structure driving this divergence, provides a five-dimension comparison framework, debunks common misconceptions, and covers training, inference, and serving considerations. It closes with scenario-based selection recommendations for high-concurrency chat, long-document and code understanding, streaming, agent memory, and multimodal sequences.
Since the Transformer architecture was introduced in 2017, the structural design of large language models has gone through several rounds of refinement. Modules such as normalization, positional encoding, attention implementation, and feed-forward networks have all changed. Over the past two years, technical teams have largely converged on their feed-forward networks but have offered distinctly different answers for the attention layer: some retain global attention and add gated linear attention, some pair a small number of full-attention layers with a large number of sparse layers, and some embed gated attention inside state-space models. This article is abstracted from public technical reports and model cards, with vendors represented anonymously as A through I. Because model architectures iterate quickly, specific configurations should be confirmed against the official technical reports and the latest documentation.
1. The Evolution Trajectory: From Module Refinement to Attention-Layer Divergence
1.1 Phase One (2017–2022): Modular, Single-Point Refinement
The core logic of this phase was “keep the Transformer backbone unchanged and replace or optimize modules one by one.” Four kinds of change had the most far-reaching impact.
- Normalization: Shifted from post-normalization (“sub-layer → residual → normalization”) to pre-normalization (“normalization → sub-layer → residual”), which significantly improved training stability in deep networks; then, root-mean-square normalization without a mean term further reduced computation.
- Positional encoding: Learnable position embeddings had limited extrapolation ability and were gradually replaced by rotary position encoding and extrapolation-correction schemes, keeping models usable on contexts far longer than their training length.
- Attention implementation: From multi-head attention to grouped-query attention, and then to variants such as sliding window and block sparsity, always aimed at “reducing the key-value cache” and “reducing attention computation.”
- Feed-forward networks: Gated linear unit activations combined with a three-matrix structure became a common configuration; mixture-of-experts moved from sparsifying a few large experts to fine-grained experts, alleviating load imbalance and catastrophic forgetting.
The consensus of this phase: the structure was not overthrown; every refinement was engineering preparation for “larger models, longer contexts, and lower inference cost.”
1.2 Phase Two (2023–Present): Feed-Forward Convergence, Attention-Layer Divergence
As fine-grained mixture-of-experts gradually became mainstream, the feed-forward layer began to converge. The typical form is a large number of fine-grained routed experts plus a small number of always-on shared experts, where each token activates only a small fraction of parameters. Total parameters can be very large, while activated parameters make up only a small portion of the total. Training strategies also converged: dense pre-training first, then sparsification, and finally annealing.
Once the feed-forward layer no longer had obvious room for differentiation, vendors began treating the attention layer as the next target for optimization. To avoid confusion caused by version updates, vendor names are replaced with letters; this article abstracts the different vendors as A through I. The structures in the table only illustrate technical routes, and specific configurations should be confirmed against official technical reports.
| Vendor | Representative series | Attention structure route | Cross-generation combination | Evolution trend |
|---|---|---|---|---|
| Vendor A | A early / A new | 3× gated linear layers + 1× gated global layer | Linear + global | Linear → linear + global |
| Vendor B | B early / B new | 3× linear state layers + 1× global layer; later multiple linear layers + a few gated global layers | Linear + global | Linear → linear + global |
| Vendor C | C early / C new | Linear attention → full attention → block-sparse attention | Linear → full → sparse | Linear → full → sparse |
| Vendor D | D new | 7× linear attention + 1× global attention | Linear + global | Linear + global |
| Vendor E | E early / E new | Token-level dynamic sparsity → compression-and-sparsity fusion | Dynamic sparse → compression sparse | Dynamic sparse → compression sparse |
| Vendor F | F series | Dynamic sparse → sparse + index sharing → linear + sparse | Sparse → linear + sparse | Sparse → linear + sparse |
| Vendor G | G early / G new | 1× sliding window + 1× global → 3× gated sliding window + 1× sparse global | Window + global → window + sparse | Window + global → window + sparse |
| Vendor H | H new | 5× window attention + 1× global attention | Window + global | Window + global |
| Vendor I | I early / I new | State-space + grouped query → gated sparse + cross-layer index reuse | SSM hybrid → sparse | SSM hybrid → sparse |
After abstracting the attention schemes of 22 models from 9 vendors, several common judgments emerge.
- Global capability has not been abandoned. Almost all schemes retain some form of global or full-attention layer. In the mixed units explicitly given in the table, the proportion of global layers ranges roughly from 1/2 to 1/8; this is based on examples such as “1 global layer per 1 linear layer,” “3 linear layers per 1 global layer,” “5 window layers per 1 global layer,” and “7 linear layers per 1 global layer” from the table.
- Linear attention has returned. Multiple vendors use linear or state-space layers as the main body, backed by a small number of global layers.
- Sparsification has become the default option. Even the original full-attention layers are being replaced by dynamic sparsity, block sparsity, or gating.
- The combination ratio is a tunable hyperparameter. The ratio between linear layers and global layers, and the selection granularity of sparse layers, is becoming an architectural hyperparameter like the “number of experts / number of activations” in MoE.
- Cross-layer reuse is emerging. Some teams reduce repeated selection or index computation through index sharing and cross-layer index reuse.
1.3 Why Attention Became the New Divergence Point
The attention layer became a divergence point because the cost structure changed.
- Quadratic complexity: The time and memory cost of standard softmax attention both grow quadratically with sequence length n. When the context expands from thousands to hundreds of thousands or even millions of tokens, this overhead directly determines whether the model is usable.
- The key-value cache is a memory and bandwidth bottleneck: In autoregressive decoding, every generated token requires reading the entire historical key-value cache, so the decode phase is often “memory-bound” rather than “compute-bound.” The larger the cache, the lower the per-request throughput, and the higher the cost under high concurrency.
- Long context has become a hard requirement: Scenarios such as long-document question answering, code repository understanding, agent multi-turn memory, and multimodal long sequences all require the model to retrieve information precisely within extremely long contexts. Linear methods excel at cumulative summarization but are poor at precise localization, which is exactly the strength of softmax attention.
- Consistency between training and inference: How the sparse or linear behavior at inference time is simulated during training determines whether “saving at training time” translates into “saving at inference time.”
- Hardware affinity: Whether the attention operator is friendly to GPU matrix-multiply units directly affects real-world speed, not just theoretical complexity.
2. Six Technical Routes for the Attention Layer
2.1 Route One: Gated Linear Attention (Delta-Rule Family)
Core idea: Standard softmax attention is like “first writing out the complete key-value table, then retrieving from it one by one.” Linear attention instead compresses history into a fixed-size state matrix, updating the state each time a token is read, with a read cost independent of sequence length.
Let the dimensions of the query, key, and value be (q_t,k_t \in \mathbb{R}^{d_k}), (v_t \in \mathbb{R}^{d_v}), and let the state at step t be (M_t \in \mathbb{R}^{d_v \times d_k}). The update and output can be written as:
[
Mt = \alpha_t M{t-1} + \betat (v_t - M{t-1}k_t)k_t^\top,\quad o_t = M_t q_t
]
where (\alpha_t) is the gating coefficient that controls historical decay, and (\beta_t) is the write strength that controls the amount of new information written. After expansion, it can be rewritten in an “error-driven” form:
[
Mt = M{t-1}(\alpha_t I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top
]
Each token is first predicted once by the current state, and then only the prediction error is written. This is precisely a generalization of the Delta rule, aimed at improving the memory utilization of a fixed-capacity state.
The schematic code below is usually implemented in a block-parallel form during training:
def gated_delta_step(state, k, v, q, alpha, beta):"""Single-step update of the gated Delta rule (schematic)state: [d_v, d_k] memory state that stores "key -> value" associationsk, q: [d_k] query/key vectorsv: [d_v] value vectoralpha: gating coefficient that controls historical decaybeta: write strength that controls how much new information is written"""pred = state @ k # [d_v]delta = beta * (v - pred) # [d_v]state = alpha * state + torch.outer(delta, k) # [d_v, d_k]return state, state @ q # [d_v, d_k] * [d_k] -> [d_v]
Key design elements include:
- Gating mechanism: Similar to an LSTM forget gate, it lets the model decide by content whether to forget historical information. In practice, the forget factor is often parameterized to a form strictly less than 1 to maintain training stability.
- Normalization: The choice of kernel function and its corresponding normalization term determines training stability and expressive power.
- Engineering form: Linear attention can be written in matrix-multiply form, computed in blocks in parallel during training and updated recursively in O(1) state per step during inference—a “switchable” computation mode.
Applicability and costs:
- Suitable: Long sequences and tasks where state information can be compressed, such as long-document summarization, multi-turn dialogue, and streaming processing.
- Costs: The state capacity is fixed, so information can overwrite itself; for “needle-in-a-haystack” precise retrieval, it often relies on a few full-attention layers as backup.
2.2 Route Two: Linear + Global Hybrid Layer Stacking
Core idea: Since linear attention has a capacity limit, insert global attention layers at a fixed ratio so that “linear layers handle long-range accumulation, and global layers handle precise retrieval,” forming a multi-scale structure similar to “downsampling + pooling” in convolutional networks.
Taking a repeating unit of “3× gated linear layers + 1× gated softmax layer” as an example, it can be understood as: the model maintains a layer list and a set of global-layer indices; when iterating over each layer, if the current index belongs to the global-layer set, it calls the global attention module; otherwise, it calls the linear attention module. The pseudocode can be written as:
def hybrid_forward(hidden_states, position_ids, layer_list, global_indices):"""Forward pass for a linear + global hybrid layer stack (schematic)layer_list: list of layer modulesglobal_indices: index set of layers that use global attention"""for idx, module in enumerate(layer_list):if idx in global_indices:hidden_states = module(hidden_states, position_ids, mode="global")else:hidden_states = module(hidden_states, position_ids, mode="linear")return hidden_states
Key design elements include:
- Ratio: In the table data, the number of linear layers corresponding to “1 global layer” ranges from 1 to 7. A higher ratio saves more cache and computation on long sequences, but the long-range retrieval capability must be verified as sufficient; a lower ratio is the opposite.
- Inter-layer position: Whether global layers are placed in shallow layers, deep layers, or distributed evenly affects the functions each level undertakes. A common practice is to place the global layer at the end of the repeating unit.
- Gating on global layers: Adding a gate on the softmax attention output lets the model adjust, by content, the retention of “retrieval results,” reducing interference from irrelevant information. Such global layers may also use low-rank key-value compression, or grouped-query key-value sharing, to further compress the cache.
- Attention normalization: Some teams normalize queries and keys to stabilize training when mixing global and linear layers.
- Variants: The linear layer can also be replaced by other forms. One is a state-space model, such as a selective-scan SSM layer, which is naturally suited to alternating stacking with gated attention; another is gated linear attention plus a lightweight channel-mixing operation, forming a symmetric “token mixing + channel mixing” structure.
Multi-generation model comparisons show that “pure linear → linear + global → sparsification of global layers” is a clear iterative path.
2.3 Route Three: Dynamic Sparse Attention
Core idea: Rather than compressing history or changing the order of magnitude of computational complexity, let each query attend only to a small number of the “most relevant” keys. The typical process is:
- Block compression: Pool keys and values by block to obtain a summary representation of each block.
- Block scoring: Compute similarity between the query and block summaries to obtain block-level relevance scores.
- Block selection: Select the most relevant blocks according to the block-level scores.
- Token refinement: Continue with token-level top-k selection within the selected blocks to reduce long-context overhead.
- Aggregated computation: Aggregate keys and values by index, then perform attention computation.
Key design elements include:
- Two-stage coarse-filter-plus-refine structure: First use a shared block-level compression representation for coarse filtering, then perform token-level refinement within selected blocks. Both stages should be implemented with hardware-friendly operators as much as possible to avoid the irregular memory access caused by per-token search.
- Sharing and gating: Multiple query heads can share the same set of selected keys and values, and can differentiate their selection preferences via gating; they can also combine grouped-query attention with output gating to control retrieval strength.
- Hardware-aware selection: Whether previously computed cache blocks are hit, and the alignment of block size with grouped-query size, all affect real speed and need co-design with the inference kernel.
- Long-context extension: For scenarios trained short but inferred long, continued pre-training on long sequences can align the selection pattern of sparse attention with the length distribution.
In terms of evolution, some schemes have moved from token-level top-k selection to a “compression + sparse” combination, indicating that pure sparsity alone is insufficient for extremely long contexts and that compression is beginning to fuse with sparsity.
2.4 Route Four: Compression + Sparse Fusion Attention
Core idea: Compression handles “seeing the full history at low cost,” while sparsity handles “precisely retrieving key segments.” Specific approaches include: compressing continuous token blocks into summary representations; strided sampling to retain representative tokens; and introducing an additional high-compression channel into attention. Layering token-level selection or block sparsity on top of this forms a compression-sparse hybrid.
It can be abstracted as two attention outputs:
[
O = g \odot \text{Attn}(Q,[Kc;K_s],[V_c;V_s]) + (1-g)\odot \text{Attn}(Q,K{sel},V_{sel})
]
where (Kc,V_c) are the compressed keys and values, (K_s,V_s) are the sampled keys and values, (K{sel},V_{sel}) are the keys and values selected by top-k from the original tokens, and (g) is the gating coefficient.
Key design elements include:
- Fusion of selection granularity: Selection can be done on the compressed sequence or on the original tokens, forming a multi-granularity hybrid of “compression-aware + token selection.”
- Decoding-side consistency: At decoding time, the distribution of newly generated queries differs from the “history-only” queries at training time, so dedicated alignment methods are needed to narrow this gap.
- Evolution path: From pure linear, to linear plus global, and then to block-sparse or compression-sparse, indicating that sparsity and compression are becoming the mainstream backbone for long context, while linear methods increasingly serve the role of “efficient long-range accumulation.”
2.5 Route Five: Sliding-Window Localization
Core idea: Localize attention so that each token interacts only with neighbors within a fixed window, reducing complexity from (O(n^2)) to (O(n \cdot w)). After introducing the “half sliding window” concept, half the attention heads look only at the local window while the other half still see globally; combined with output gating and a few global layers, this forms a “window layer + global layer” combination.
Taking “3 gated sliding-window layers + 1 sparse global-query attention layer” as an example, the core process is:
- Construct a window mask based on position, masking out keys outside the window.
- Perform attention computation with output gating within the window.
- Periodically insert global layers to supplement long-range retrieval capability.
Key design elements include:
- Window size: A smaller window saves more but requires more frequent global layers or cross-layer information propagation to supplement the receptive field.
- The value of gating: Gating lets local attention adjust its output by content, alleviating the information loss from window truncation while improving training stability.
- The role of global layers: Global layers have gradually shifted from “full softmax” to “sparsified global-query attention,” making the cost of long-range retrieval controllable.
- Difference from dynamic sparsity: Sliding window is a “fixed, static, hardware-friendly” sparsity; dynamic sparsity is a “content-driven, non-fixed” sparsity. The two are often combined: static windows handle local, dynamic selection handles long-range.
2.6 Route Six: State-Space Model Hybrid
Core idea: Selective state-space models take the state-transition matrix as their core and achieve “content-aware memory” through an input-dependent selection mechanism. The update form is:
[
ht = A_t h{t-1} + B_t x_t,\quad y_t = C_t h_t + D x_t
]
Written in matrix form, it is naturally homologous with linear attention and can be trained efficiently in parallel with block scans.
Key design elements include:
- Hardware efficiency: State transitions can be organized as matrix multiplications, which are GPU-friendly and can be executed efficiently in both training and inference.
- Hybridization: SSM layers can be alternately stacked with attention layers, or SSM layers can replace some linear layers; both approaches have been practiced.
- Difference from linear attention: SSMs lean toward “sequential recursion + structured state transition,” while gated linear attention leans toward “error-driven key-value association writes”; the two differ in memory capacity and retrieval accuracy.
3. What Attention Schemes Are and How to Choose: A Unified Comparison Framework and Common Pitfalls
3.1 What: What Attention Schemes Actually Optimize
The core of an attention scheme is not simply “reducing the computation of a single attention operation,” but optimizing long-context capability, precise-retrieval capability, local-detail modeling capability, and inference throughput simultaneously, under limited cache, limited compute, and limited bandwidth. In other words, the attention structure answers one question: how to retain as much global information as possible with limited state.
3.2 How to Choose: A Five-Dimension Comparison Framework
The routes can be evaluated uniformly across five dimensions: computational complexity, cache form, long-range capability, training friendliness, and inference friendliness.
| Route | Computational complexity | Cache form | Long-range capability | Training friendliness | Inference friendliness | Use cases & costs |
|---|---|---|---|---|---|---|
| Global softmax attention | (O(n^2)) | Full KV cache | Strongest | Mature and stable | Memory-access heavy at decode | High-precision tasks; higher cost |
| Gated linear attention | (O(n)) | Linear state | Strong accumulation, weak precise localization | Needs stable gating and normalization | Low memory access, suits streaming | Long-document summarization, streaming; precise retrieval needs backup |
| Linear + global hybrid | Between linear and global | State + small KV cache | Fairly strong | Needs ratio search | Relatively balanced | Long documents, code understanding; long-range retrieval must be verified |
| Dynamic sparse attention | Roughly related to the selected amount | Selected KV or full KV cache | Depends on selection quality | Higher index-construction cost | Needs kernel co-design | Extremely long context; high training-inference consistency required |
| Compression + sparse | Compression part near-linear, sparse part depends on selection | Compressed KV + selected KV | Fairly strong | Needs to learn compression representations | Decode distribution needs alignment | Multimodal long sequences; higher system complexity |
| Sliding-window localization | (O(nw)) | Local KV cache | Strong local, weak global | Stable and easy to train | High throughput | Short-to-medium-long dialogue; global layers needed to fill in |
| State-space model hybrid | (O(n)) | State cache | Strong accumulation, weak precise retrieval | Hybrid design with attention | Hardware-friendly | Real-time streaming; precise retrieval needs global layers |
3.3 Common Pitfalls
- Looking only at theoretical complexity: Actual speed is affected by memory access, operator implementation, and batching strategy, so one cannot look only at (O(n)) or (O(n^2)).
- Looking only at training results: If full attention is used during training but sparse attention at inference, without simulating the inference mode during training, you may get good training metrics but poor online results.
- Looking only at model structure, not at cache and bandwidth: In long-context serving, the key-value cache often becomes a bottleneck before the attention computation itself.
- Assuming linear attention can fully replace global attention: Linear methods are good at accumulating information, but precise localization and strong-retrieval tasks still need global layers or sparse selection as backup.
- Assuming sparsity is free: Dynamic sparsity requires index construction, selection strategies, training-inference consistency, and inference-kernel support, so its engineering complexity is not low.
4. Practical Points for Training, Inference, and Serving
4.1 Training Side
- Mixed ratio as a hyperparameter search: The ratio between linear and global layers, the window size, the number of sparse blocks and tokens—all should be included in the architecture search space. It is recommended to validate at small-to-medium scale first, then scale up.
- Smooth transition from dense to hybrid: You can first train a full-attention model, then replace some layers with linear or sparse layers for continued training, avoiding the instability of training directly from scratch.
- Numerical stability: Normalize queries and keys; apply parameterization constraints to the decay and write coefficients of linear attention; use higher precision for state accumulation when training in bf16 or fp16.
- Length extrapolation: Combined with positional-encoding extrapolation strategies and long-sequence continued pre-training, let the sparse pattern and window size be fully learned at the target length.
- Training-inference consistency: If full attention is used during training and sparse attention at inference, the sparse pattern at inference must be simulated during training—for example, using the same block-selection logic—otherwise a “good at training, bad at inference” bias appears.
- Parallelism and distribution: Linear layers can be split by block and global layers can be split by sequence, so explicit parallelism strategies are needed for the hybrid structure; at the same time, be mindful of dependencies introduced by structures like cross-layer index reuse, to avoid pipeline bubbles.
4.2 Inference Side
- Cache estimation: The key-value cache size is approximately layers × number of key-value heads × head dimension × 2 × bytes × sequence length. Linear or compression structures can significantly reduce this value, but the specific gain must be measured against the actual configuration.
- Paged cache and continuous batching: Combined with paged cache management and continuous batching, this improves GPU utilization and reduces memory fragmentation for long-sequence requests.
- Prefill-decode separation: In long-context scenarios, the prefill phase is compute-intensive and the decode phase is memory-intensive; they can be deployed to different compute pools to improve overall throughput.
- Operator fusion and hardware adaptation: The selection process of sparse and compression attention involves irregular memory access and needs co-design with the inference kernel, including block-size alignment, index reuse, and grouped query.
- Streaming and long-document scenarios: Linear or state-space structures naturally support streaming decoding, and window structures can expand the effective receptive field through cross-layer propagation; these properties have direct value in agent long-memory, real-time dialogue, and similar scenarios.
- Serving and elasticity: General cloud-service capabilities can be combined with the above structures—for example, containerized deployment, asynchronous task queues, object storage, logging services, and monitoring/alerting—each handling elastic compute scheduling, task decoupling, asset and log retention, and observability, thereby supporting a “train–infer–evaluate” closed loop.
4.3 Evaluation and Selection
- Scenario-specific evaluation: For short-context tasks, focus on perplexity and downstream accuracy; for long-context tasks, focus on capabilities like “needle-in-a-haystack,” long-document QA, and multi-hop retrieval; for streaming scenarios, focus on first-token latency and throughput.
- Focus on long-range behavior: Check whether window size, global-layer ratio, and sparse-selection granularity cause information loss on long sequences; use “key-information insertion position” experiments to locate weak spots.
- Focus on training efficiency: Convergence speed, stability, and final results under the same compute budget for different structures often reflect engineering value better than a single-point metric.
- Focus on hardware cost: Include memory usage, bandwidth consumption, and operator utilization in selection metrics, rather than looking only at theoretical complexity.
5. Selection Decision Recommendations
For common business scenarios, the following paths can be referenced:
- Short dialogue and high concurrency: Prioritize a hybrid structure of sliding window plus a few global layers, or full attention with grouped-query key-value sharing; focus on optimizing cache and batching efficiency. The constraint is limited long-range retrieval capability, requiring global layers or sparse selection to supplement.
- Long documents and code repositories: Prioritize “linear + global” hybrids or “compression + sparse” structures; the former has a small cache and supports streaming, while the latter is more friendly to precise localization. The constraint is that precise retrieval of key information still needs to be verified through evaluation in extremely long contexts.
- Streaming and real-time scenarios: Prioritize linear or state-space hybrid structures, which naturally support incremental decoding and constant state. The constraint is weaker precise-localization capability, requiring global layers or retrieval-augmentation mechanisms as backup.
- Agent long memory: Prioritize a combination of “global-layer backup + sparse selection + cross-layer index reuse,” retaining retrieval capability while controlling cost. The constraint is that index reuse and cross-layer sharing increase training and inference system complexity.
- Multimodal long sequences: Prioritize a compression + sparse hybrid structure, using block compression to reduce the storage and computation overhead of visual or audio tokens. The constraint is that the token distributions of different modalities differ significantly, requiring finer evaluation and gating design.
It should be noted that these routes are not mutually exclusive. Judging from the evolution in technical reports, the three-layer combination of “linear or state-space for accumulation, global or sparse for retrieval, and compression for cost reduction” is becoming the mainstream backbone of long-context models.
6. Conclusion: The Methodology of Architectural Evolution
Looking back at the structural evolution since 2017, three methodological principles can be distilled:
- Bottleneck-driven: Structural innovation is often not “inventing a new module,” but targeted modification against the most prominent current bottleneck, such as long context, cache, or throughput.
- Combination over singularity: Pure linear, pure sparse, and pure compression all have obvious shortcomings; hybrid structures achieve balance through “different layers undertaking different functions.”
- Co-design of training and inference: The structure at training time, the kernel at inference time, and the deployment method at serving time must be considered together, otherwise “structural optimization” is hard to convert into “business gains.”
For developers and enterprise users, understanding the principles and trade-offs of these routes is more important than memorizing any one vendor’s scheme. Faced with a specific business, first clarify the four constraints of “context length, retrieval accuracy, concurrency, and cost budget,” then combine an appropriate structure from the above framework, and deploy it through the path of “small-scale validation → length extrapolation → inference-kernel adaptation → serving deployment.”
