The Evolution of Sparse LLM Architectures: From Hybrid Attention to Gated Residuals

In recent years, the parameter scale of large language models (LLMs) has continuously expanded. However, what truly determines inference cost and response latency is the number of parameters activated per token. To address this contradiction, the industry has gradually formed a clear technical roadmap: expanding total parameter capacity while minimizing the computational overhead per token. Using a recently open-sourced sparse MoE architecture as an entry point, this article systematically reviews its four core design components and discusses the evolutionary trends of current mainstream architectures.

1. Overall Design: Sparse MoE and Four-Component Synergy

This architecture is a typical sparse Mixture-of-Experts (MoE) model with a total parameter count of 125B, activating only about 6B parameters per token. Additionally, it mounts an n-gram embedding layer with 51B parameters. Measured by the ratio of activated parameters to total parameters, the sparsity is approximately 4.8%. For reference, the design approaches of several recent mainstream sparse models are as follows (describing parameter structures neutrally):

• Sparse Model A: 13B activated / 284B total, sparsity ~4.58%
• Sparse Model B: 18B activated / 320B total, sparsity ~5.63%
• Sparse Model C: 104B activated / 2800B total, sparsity ~3.71%

As observed, mainstream technical approaches are gradually converging on sparsity, generally falling within the 4% to 6% range. This convergence is not accidental but the inevitable result of balancing throughput, latency, and cost for Agent-based applications. When models need to frequently call tools, perform multi-step planning, and conduct long-context retrieval, a sparsity rate that is too low will slow down inference, while one that is too high may compromise expressive power.

Supporting this overall design are four architectural components corresponding to different bottlenecks:

1. Layer-by-layer mixing of gated linear attention and global attention: Recurrent layers compress the prefix into a fixed-size state at linear cost, while one out of every four layers retains full attention to maintain precise token-level retrieval capabilities.
2. Sparse attention mechanism: It follows the sparse route but scores the context at a micro-block granularity and introduces a compressed lightweight indexer, making the indexing overhead itself decrease as the sequence length grows.
3. Gated residuals: The residual stream is expanded into four branches, read via element-wise gating. This increases residual path capacity, dictates how the capacity is used, and provides the rescaling necessary for training stability.
4. n-gram embedding layer: It expands capacity outside the backbone network. The embedding table is prefetched from host memory, scaling parameter size with almost no increase in per-token FLOPs or latency.

Let’s break down each of these components.

2. Hybrid Attention: Why a 3:1 Ratio?

In long-context scenarios, the computational cost of full attention grows quadratically with sequence length. For a causal decoder, which only computes the lower triangular part, the magnitude is approximately:

FLOPs ≈ 2 · B · S² · D

Where B is the batch size, S is the sequence length, and D is the hidden dimension. Consider a specific example: when S=128K, D=8192, and B=1, the floating-point operations for just the attention matrix multiplication reach approximately 2 × 1 × (128K)² × 8192 ≈ 2.7 × 10¹⁴. If every layer used full attention, both VRAM and compute power would be unsustainable.

Therefore, the industry widely adopts an alternating layer-by-layer approach of “linear attention + full attention.” Linear attention (such as various gated DeltaNet variants) compresses historical prefixes into a fixed-size recurrent state at linear cost, with the trade-off being the inability to perfectly recall token-level information at any position. Full attention layers, however, retain direct retrieval capabilities. Alternating between the two controls overall costs while preserving critical capabilities.

Observing the layer configurations of several recent mainstream open-source models, an interesting phenomenon emerges: the ratio of linear attention layers to full attention layers generally revolves around 3:1. For example:

• One model uses linear_attention : full_attention = 3:1
• Another model achieves a similar rhythm through a compression rate sequence, inserting full attention every few layers
• Yet another model configures this as kda_layers : full_attn_layers = 3:1

Why 3:1 instead of 2:1 or 4:1? This can be understood from two perspectives. From a computational standpoint, if full attention accounts for 1/4 of the layers, the total attention overhead drops to roughly 1/4 of a pure full attention scheme (the linear layer portion is nearly negligible), bringing attention FLOPs down to an acceptable range for 128K contexts. From a capability perspective, retaining one full attention layer out of every four is sufficient to maintain necessary precise recall in most retrieval and reasoning tasks. If this ratio were further reduced to 1/5 or 1/6, tasks requiring precise localization would experience noticeable degradation. Thus, 3:1 has become a repeatedly validated balance point between cost and performance.

3. Sparse Attention and the Lightweight Indexer

Even with a 3:1 hybrid setup, full attention layers still need to process long sequences. To further reduce overhead, sparse attention becomes a natural choice. The core idea is that not all tokens are worth attending to; only parts relevant to the current query need to participate in the computation.

Traditional sparse attention often selects at the token granularity, where the indexing overhead itself grows linearly or even super-linearly with sequence length. New-generation approaches shift to micro-block granularity: dividing the context into smaller blocks, scoring the blocks first, and only applying fine-grained attention to high-scoring blocks. Meanwhile, the indexer itself is compressed into a lightweight structure, causing its overhead to decrease rather than increase as the sequence length grows. This way, the effective computation under long contexts is significantly compressed, while retrieval quality is preserved through block-level scoring.

This design complements the aforementioned hybrid attention: hybrid attention addresses “inter-layer” costs, while sparse attention addresses “intra-layer” costs. The combination of both is necessary to maintain acceptable inference overhead under ultra-long contexts.

4. Gated Residuals: Dual Considerations for Capacity and Stability

Residual connections are the cornerstone of stable training in deep networks. As models grow deeper and wider, the capacity of a single residual stream gradually becomes a bottleneck. The gated residual approach expands the residual stream into multiple branches (e.g., four), with each branch carrying different information pathways. Element-wise gating then determines the read ratio of each branch.

This brings two benefits. First, the residual path capacity is broadened, allowing the network to carry multiple types of information in parallel without mutual interference. Second, the gating mechanism provides adaptive rescaling capabilities. During the early stages of training or when gradient fluctuations are large, the gating can suppress the contribution of certain branches, thereby maintaining numerical stability. In other words, widening solves “how much it can hold,” while gating solves “how to use it” and “how to keep it stable.”

5. n-gram Embeddings: Low-Cost Expansion Outside the Backbone

The growth in parameter scale does not necessarily have to reside entirely within the backbone network. The n-gram embedding layer is a means to expand capacity outside the backbone: it maintains a large embedding table, looks up corresponding vectors based on n-gram fragments in the input, and injects them into the model. Because the table lookup itself does not involve massive matrix operations, the per-token FLOPs and latency barely increase.

The real engineering challenge lies in memory: the embedding table might be too large to fit entirely into accelerator VRAM. Therefore, it requires prefetching from host memory, paired with reasonable caching and scheduling strategies to ensure that table lookups do not become a bottleneck in most scenarios. The philosophy behind this design is worth noting—expanding parameter scale without significantly increasing computation, effectively decoupling “capacity” from “compute.”

6. Trend Observations: Sparsity Convergence and Agent Scenarios

Looking at these four components together, we can see they correspond to four types of bottlenecks: inter-layer attention cost, intra-layer attention cost, residual capacity and training stability, and the decoupling of parameter capacity from compute. This design approach—where one component addresses one specific bottleneck—is becoming a consensus for next-generation architectures.

On a broader scale, the sparsity of mainstream models is gradually converging to 4%–6%, and the hybrid attention ratio is converging to 3:1. This indicates that the industry has formed a relatively consistent cost model for Agent-based applications. Agent scenarios require models to frequently perform long-context understanding, tool calling, and multi-step planning. These applications are sensitive to throughput and latency, while having a rigid need for precise retrieval. Sparse MoE provides low-cost capacity, hybrid attention balances long-context overhead, sparse attention compresses intra-layer computation, gated residuals ensure deep-level stability, and n-gram embeddings offer further low-cost expansion. Together, these designs serve a single goal: enabling models to “remember accurately, retrieve precisely, and run quickly” at an acceptable inference cost.

For developers, understanding the trade-offs behind these components is more valuable than memorizing specific parameters. When you need to select or prune a model for your specific scenario, you can ask questions following the same logic: Is my bottleneck inter-layer or intra-layer? How long is my context? Does my task rely more on precise retrieval or fuzzy compression? Answering these questions often yields more practical benefits than blindly chasing parameter scale.