Next-Generation LLM Architectures: Sparse MoE and Hybrid Attention
This article covers how sparse mixture-of-experts (MoE) models separate total parameters from per-token active parameters, how a 3:1 mix of linear and global attention changes attention compute, and how micro-block sparse attention, gated residual streams, and external n-gram embeddings affect cost. It ends with a checklist for choosing an architecture.
1. Sparse MoE: Separating Total and Active Parameters
Sparse mixture-of-experts (MoE) models have very large total parameter counts, but each token activates only a subset of them. The table below shows four example models from the source. The data illustrates trends only and has not been verified by a third party.
Table 1: Example Total and Active Parameters from the Source
| Model | Total Parameters | Active Parameters per Token | Active Ratio |
|---|---|---|---|
| Model A | 125B | 6B | 4.8% |
| Model B | 284B | 13B | 4.58% |
| Model C | 320B | 18B | 5.63% |
| Model D | 2800B | 104B | 3.71% |
The active ratios of the four samples range from 3.7% to 5.6%, and most fall between 4% and 6%. Model D has the lowest ratio and also the largest total parameter count. This suggests that very large models can push the active ratio even lower.
This design, with large total parameters and small active parameters, keeps a large knowledge capacity while limiting the compute and memory access required per step. For agent applications that need multi-step tool calls and long reasoning chains, this balance matters in practice.
Parameter counting must use a consistent basis. The source cites one typical model with 51B n-gram embedding parameters. Lookup-style embedding layers usually do not count toward per-token active computation, but they still consume storage. When comparing two models, state whether the total parameter count includes the embedding table. Otherwise, the ratios cannot be compared directly.
Fewer active parameters does not mean lower memory requirements. All expert weights must still reside in GPU memory or be swapped in quickly. In deployment, memory capacity often becomes the limit before compute does.
2. Hybrid 3:1 Linear and Global Attention: How to Count Compute
Linear attention, such as recurrent layers like Gated DeltaNet, compresses the prefix into a fixed-size state. Compute therefore grows linearly with sequence length. Global attention keeps direct token-level retrieval, which a finite-state memory struggles to reproduce exactly. As a result, many models keep one global attention layer every few layers.
The source observes that some mainstream models have a linear-to-global attention ratio close to 3:1. This is an observed configuration, not a general rule.
The estimates below use the approximation adopted in this article. Attention compute for a causal decoder is about 2 × B × S² × D. Take S = 128K (131,072), D = 8192, and B = 1:
- One global attention layer: 2 × 131,072² × 8192 ≈ 0.28 PFLOPs
- All 60 layers using global attention: about 16.9 PFLOPs
- 15 global attention layers in the 3:1 mix: about 4.2 PFLOPs
Looking only at global attention layers, the 3:1 configuration costs about one quarter of the all-global scheme. This figure excludes the cost of the other 45 linear layers.
Linear-layer cost grows linearly with S. If the state width is 128, each layer costs about 2 × S × D × 128 ≈ 0.00028 PFLOPs. The 45 layers total about 0.012 PFLOPs, which barely affects the total. If the state width equals D, each layer costs about 2 × S × D² ≈ 0.018 PFLOPs. The 45 layers total about 0.79 PFLOPs, and the hybrid scheme totals about 5.0 PFLOPs, roughly 30% of the all-global scheme. Under both assumptions, the hybrid scheme’s attention cost is roughly one quarter to one third of the global scheme’s cost.
Feed-forward and projection layers have the same compute in both schemes, so the comparison above covers only the attention part.
3. Micro-Block Sparse Attention: How Indexing Cost Changes
Dense attention cost grows quadratically with sequence length. Micro-block sparse attention first splits the context into fixed-size micro-blocks. A lightweight indexer then scores each block, and exact attention runs only on the top-scoring blocks.
The indexing cost still grows with sequence length, but more slowly. Let b be the micro-block size and d_i the index dimension. Each query scores every context block, so the indexing compute is about S × d_i / b. Full attention compute per query is proportional to S × D. The ratio of the two slopes is about d_i / (b × D). With b = 64, d_i = 128, and D = 8192, this ratio is about 0.024%. These are illustrative parameters only. Actual values depend on the implementation.
This mechanism should not be understood as indexing cost falling as sequences get longer. A more accurate statement is this: with a fixed micro-block size and index dimension, indexing cost grows much more slowly than full attention, but total indexing cost still rises with context length.
This mechanism also has failure conditions. When key information is spread across several micro-blocks, the selected blocks may be incomplete. When the indexer mis-scores a block, relevant blocks may be missed. Micro-blocks that are too large or too small can also hurt quality.
When reporting results for this kind of method, state the micro-block size, the number of selected blocks, the index dimension, the sequence length, the hardware model, and the numerical precision. Speedups reported without these conditions cannot be reproduced.
4. Gated Residual Stream and External n-gram Embeddings
The gated residual stream expands the traditional single-channel residual into four independent branches, which are read through element-wise gating. The gates dynamically decide how capacity is allocated across branches, and rescaling helps keep training stable. The benefits of these designs need to be confirmed through ablation experiments, especially in deep models. The gating itself adds parameter reads and writes and extra bandwidth overhead.
The n-gram embedding layer is a large lookup structure that is independent of the backbone network. It can sit in host memory and be prefetched based on the current context. This approach can expand the parameter count without significantly increasing the compute per token.
The claim that this design uses almost no GPU compute holds only under certain conditions:
- The prefetch hit rate is high, so most needed embeddings are already in GPU memory or can arrive in advance.
- Data transfer overlaps with compute and does not block decoding.
- Host memory capacity is sufficient, and the NUMA overhead on the access path is manageable.
- CPU overhead for lookups and scheduling does not become a bottleneck.
The following example is illustrative. Assume an embedding dimension of 4096 in FP16, so each entry is 8 KB. If each decoding step has 1,000 missed entries, about 8 MB must be transferred. At an effective bandwidth of 50 GB/s, the transfer takes about 0.16 ms. Actual bandwidth depends on the PCIe version, the lane count, whether memory is pinned, and whether transfer and compute run concurrently. As batch size grows, the number of missed entries also grows, and transfer pressure rises with it.
The embedding table itself does not occupy GPU memory, but the prefetch cache does. This design therefore does not necessarily re
