Linear, Hybrid, and Sparse Attention: How LLM Architectures Are Evolving

    Softmax attention's O(N²) cost makes long contexts expensive. This article explains three improvement paths—linear attention, hybrid architectures, and sparse attention—covering mechanisms such as Gated DeltaNet, Lightning Attention-2, MLA, DeepSeek Sparse Attention, and NSA, along with practical guidance on architecture selection and deployment checks for memory fragmentation, operator fusion, and communication overhead in heterogeneous inference systems.

    Since the Transformer was introduced in “Attention Is All You Need” (2017), Softmax attention has become a standard building block of deep models. Its compute cost grows quadratically with sequence length N, i.e., $O(N^2)$, so both training and inference costs rise quickly as context gets longer. Over the past few years, researchers have improved on it along three main paths: linear attention to reduce complexity, hybrid architectures to retain global information, and sparse attention to compute only the most important connections. This article walks through the principles, representative work, and applicable boundaries of each path, and concludes with engineering considerations for cloud deployment.

    1. How Linear Attention Reduces Complexity to Linear—and What Gating Adds

    Softmax attention first computes similarity scores between each query vector and all key vectors, normalizes those scores, and then produces a weighted sum. Because the similarity matrix is N×N, compute grows quadratically with N.

    Linear attention rewrites the similarity as an inner product of two feature maps, $\phi(Q)\phi(K)^\top$. Using the associativity of matrix multiplication, you can first compute a fixed-size state matrix $\phi(K)^\top V$ and then have queries read from it. This makes compute linear in N, and at inference time the state can be updated step by step like an RNN. Katharopoulos et al. systematically described this framework in “Transformers are RNNs” (2020).

    The weakness of early linear attention is that the state keeps accumulating: old information gets overwritten by new information, and memory degrades over long sequences. Recent models address this with gating—input-dependent coefficients that control how much of the old state is retained and how much new information is written. Gated DeltaNet (proposed in 2024, published at ICLR 2025) combines two mechanisms: gated decay for forgetting, and the Delta rule for updating. The Delta rule first reads out the state’s current prediction for a given key, then writes only the prediction error, reducing repeated overwriting of information at the same position.

    Engineering implementation matters just as much. Linear attention has low theoretical complexity, but computing token by token serially yields poor GPU utilization. Lightning Attention-2 uses chunked computation: regular matrix operations within chunks and linear state passing between chunks, making memory access and parallelism more GPU-friendly. Its goal is to turn the theoretical linear advantage into actual throughput.

    Note that linear attention is still less stable than Softmax attention on retrieval tasks—for example, precisely recovering a specific string from a very long document. Models using only linear attention tend to fall short here, which is one reason hybrid architectures emerged.

    2. How Hybrid Architectures Arrange Linear and Global Layers

    Most large models do not fully replace Softmax with linear attention. Instead, they use linear attention in most layers while keeping global attention in a few. Global layers are typically Softmax attention or MLA (Multi-head Latent Attention).

    MLA was introduced in DeepSeek-V2 (2024). It projects keys and values into low-dimensional latent vectors and caches only those latent vectors, so the KV Cache (the historical key-value pairs stored during inference) shrinks considerably. How “considerable” depends on the specific configuration—refer to the paper for exact numbers.

    Layer ratios vary by model. According to the MiniMax-01 technical report (2025), that model uses Lightning Attention in most layers, interleaving one Softmax attention layer roughly every seven linear layers—about a 7:1 ratio. According to the Kimi Linear technical report (2025), that model interleaves Kimi Delta Attention (KDA) with MLA at roughly 3:1. These numbers come from the respective public reports; actual results depend on training data, parameter scale, and evaluation sets, so they should not be applied directly to other models.

    Mechanism Compute complexity (sequence length N) KV Cache footprint Long-range exact retrieval Implementation difficulty
    Softmax attention $O(N^2)$ Grows linearly with N Strong Mature, rich ecosystem
    Linear attention $O(N)$ Fixed-size state, independent of N Weaker; depends on gating design Requires custom chunked kernels
    MLA (global layer) Still $O(N^2)$ Significantly reduced after compression Strong Requires inference framework adaptation
    Sparse attention Depends on selection strategy, often $O(NK)$ Usually still requires full KV Depends on whether the selector hits key positions Requires handling irregular memory access

    The benefit of hybrid architectures comes from division of labor: linear layers handle most of the context efficiently, while global layers handle long-distance dependencies. There is no universally optimal layer ratio—it requires ablation experiments based on your target tasks’ length distribution and retrieval needs.

    3. Sparse Attention: From Fixed Patterns to Dynamic Selection

    Sparse attention does not change the Softmax computation itself; it reduces the number of key-value pairs that participate.

    Fixed-pattern sparse attention came first. Sparse Transformer (2019) used fixed local and strided connections; Longformer (2020) introduced sliding windows plus global tokens; BigBird (2020) added random connections. These methods are simple to implement with regular memory access, but their connection patterns are fixed in advance and cannot adapt to content, limiting the global view.

    Dynamic sparsity selects connections based on content. DeepSeek Sparse Attention (DSA), introduced in DeepSeek-V3.2-Exp (2025), uses a lightweight indexer to score each query’s relevance to historical tokens, then takes the Top-K key-value pairs for Softmax computation. According to its technical report, K is around 2048. The selection step is non-differentiable, so the indexer is typically trained with alignment signals against the full attention distribution—this is the main difficulty in training dynamic sparsity.

    Another line of work is Native Sparse Attention (NSA, 2025). It splits attention into three branches: a compression branch providing coarse-grained global summaries, a selection branch retaining important blocks, and a sliding-window branch preserving local detail, with the three outputs combined by weighting. This design is close to the “compress first, then sparsify” idea, but the exact structures of different works vary considerably—refer to the original papers.

    Dynamic sparsity also has failure modes. If critical information is spread across many positions and no single token scores highly, Top-K may miss it. The indexer itself adds compute and memory overhead: too small a K loses information, too large a K diminishes returns. Irregular sparse memory access also lowers GPU efficiency and requires dedicated operator implementations.

    4. How to Choose for Different Scenarios

    • Short sequences, high concurrency: Prefer Softmax attention or its mature optimized implementations, such as FlashAttention (an IO-aware exact attention implementation). When context is short, the gains from linear or sparse attention may not offset the implementation cost.
    • Long documents, codebase understanding, multi-turn long conversations: Prefer hybrid architectures. Linear layers control cost, while a few global layers preserve retrieval capability. Before launch, run length-bucketed retrieval tests.
    • Ultra-long context with only a few key pieces of information: Evaluate dynamic sparsity. Focus on testing whether Top-K misses critical positions.
    • Multimodal long sequences (e.g., long video frame sequences): Token counts often far exceed text. Consider local windows plus a few global layers, but verify that spatiotemporal structure is not broken.
    • Limited training resources: Prefer models with existing open weights and mature inference framework support; avoid training new architectures from scratch.

    5. What to Verify When Deploying Heterogeneous Architectures in the Cloud

    Heterogeneous architectures mix linear, global, and sparse operators, creating three main deployment challenges.

    Memory fragmentation: Requests with different sequence lengths produce KV Caches of varying sizes, and frequent allocation and deallocation leaves fragments. A common approach is paged KV Cache management (vLLM’s PagedAttention is one example) combined with length-bucketed scheduling. The fixed-size state of linear layers can be pre-allocated separately and managed apart from the global layers’ KV Cache.

    Operator fusion: Linear attention’s chunked kernels and sparse attention’s indexer typically require custom fusion. After fusion, run numerical regression—compare outputs and perplexity before and after fusion—to avoid trading accuracy for speed.

    Communication overhead: Tensor parallelism introduces all-reduce communication between layers; context parallelism splits along the sequence dimension and exchanges key-value pairs. For multi-node training, use collective communication libraries such as NCCL and overlap communication with computation as much as possible. RDMA networking can reduce cross-node latency, but whether it pays off depends on cluster topology and communication volume—it must be measured empirically.

    Pre-deployment checklist:

    1. Establish a baseline: Compare throughput, latency, and memory usage against a Softmax attention model at the same parameter scale and on the same data.
    2. Test across lengths: For example, at 512, 4K, 32K, and 128K tokens, record time-to-first-token and tokens generated per second.
    3. Retrieval regression: Use needle-in-a-haystack-style tests and record hit rates at different positions.
    4. Accuracy regression: Compare perplexity and downstream task scores before and after fusion.
    5. Monitoring metrics: Memory fragmentation rate, per-operator latency, and P50/P99 latency.
    6. Staged rollout and rollback: Keep the old model weights and service running, shift traffic gradually by percentage, and be able to switch back quickly on anomalies.

    6. Limitations to Keep in Mind

    • Public benchmarks and vendor reports are mostly self-reported with inconsistent evaluation setups. Reproduce results on your own data.
    • Models iterate quickly; versions and figures mentioned here may already be outdated. Check the latest technical reports before use.
    • The benefits of linear and sparse architectures depend heavily on training data and training budget; conclusions from small-scale experiments may not generalize.
    • Theoretical complexity does not equal real-world speed. Constant factors, memory access patterns, and hardware utilization all affect final performance.

    7. Summary and Outlook

    From Softmax attention to linear attention, and then to hybrid and sparse architectures, the common thread is reallocating resources between computational cost and long-range information utilization. Linear attention pushes cost down to linear, and gating improves memory retention; hybrid architectures add a few global layers to restore retrieval capability; sparse attention concentrates computation on the connections most likely to be useful.

    Looking ahead, attention architectures may become more tightly coupled with hardware design—for example, dedicated operators for sparse memory access, or attention implementations targeting new hardware paradigms such as compute-in-memory. Most of these directions are still in the research stage, and their effectiveness and adoption remain to be validated.

    For developers, the reliable approach is: first clarify your sequence length distribution and retrieval requirements, quantify the bottleneck with a baseline model, and then validate item by item using the principles and checklist in this article.

    References (refer to the original papers and technical reports):

    1. Vaswani et al., “Attention Is All You Need,” 2017
    2. Katharopoulos et al., “Transformers are RNNs,” 2020
    3. Yang et al., “Gated Delta Networks,” 2024
    4. Qin et al., “Lightning Attention-2,” 2024
    5. DeepSeek-AI, DeepSeek-V2 Technical Report (MLA), 2024
    6. MiniMax, MiniMax-01 Technical Report, 2025
    7. Moonshot AI, Kimi Linear Technical Report, 2025
    8. DeepSeek-AI, DeepSeek-V3.2-Exp Technical Report (DSA), 2025