Sparse Attention and Mixture-of-Experts Architectures in Long-Context LLMs
As long-context LLMs scale from 256K to 1M tokens, dense attention becomes prohibitively expensive. This article breaks down a representative next-generation design (roughly 770B total parameters, 49B active per token, 78 layers, 1M context, BF16 and FP8) and the key ideas behind it: Mixture-of-Experts (MoE) routing with an always-on shared expert, Multi-head Latent Attention (MLA) paired with a lightweight indexer that selects about 2,048 relevant positions per query, layer-wise index reuse, multi-path residual streams, multi-token prediction (MTP) for speculative decoding, and BF16/FP8 precision options. It also explains how to evaluate these models beyond leaderboard scores and how to put them into production.
1. Why Long Context Has Become the Main Battleground
Over the past two years, the central competition in large language models has shifted from raw parameter counts to effective context length. When the context window grows from 256K to 1M tokens, a model must take in roughly the equivalent of hundreds of thousands of words in a single pass. That matters for scenarios such as agent workflows, codebase-level understanding, and long-document question answering.
The naive approach, applying full attention across a million tokens, pushes compute and memory costs up quadratically and is close to impractical. So the core challenge for next-generation architectures is not simply “build a bigger model.” It is “how to spend compute where it matters most when the context is long.”
This article uses one representative next-generation architecture as a case study:
- Total parameters: ~770B
- Active parameters per token: ~49B
- Layers: 78
- Context length: 1M tokens
- Precision: BF16 and FP8
These are common engineering patterns across the industry, not capabilities exclusive to any single product.
2. Mixture-of-Experts (MoE): Decoupling Total from Active Parameters
In a conventional dense model, every token passes through all of the model’s parameters. MoE takes a different approach: keep the total parameter count large so the model has the capacity to store knowledge, but have each token activate only a small subset of experts so that inference cost stays low.
In this example, the model has ~770B total parameters and 256 routed experts. Each token activates only the Top-8 experts plus 1 shared expert that is always active, for a total of ~49B active parameters. The model’s knowledge capacity scales with 770B parameters, but its compute cost is closer to that of a 49B dense model.
The routing logic is illustrative:
def moe_layer(x, router, experts, shared_expert, top_k=8):logits = router(x) # [tokens, num_experts]topk_val, topk_idx = logits.topk(top_k)weights = softmax(topk_val)out = zeros_like(x)for k in range(top_k):out += weights[:, k:k+1] * experts[topk_idx[:, k]](x)# The shared expert carries general-purpose capability and is always# active, which dampens variation caused by different routing decisions.out = out + shared_expert(x)return out
The shared expert is a detail worth highlighting. When different tokens are routed to different experts and there is no common path through the network, training tends to become unstable. The shared expert provides a constant path for general-purpose capability, letting the model keep the capacity benefits of sparse activation without sacrificing consistency.
In practice, MoE also brings two major engineering challenges:
- Load balancing: Prevent popular experts from being overloaded while others sit idle.
- Communication overhead: Expert parallelism requires cross-device All-to-All communication.
For throughput, these often matter more than the parameter count itself.
3. Sparse Attention: Focus the Main Attention on the Most Relevant Positions
Moving from GQA (Grouped-Query Attention) to MLA (Multi-head Latent Attention) plus sparse selection is a key leap for long-context architectures. The approach has two parts:
- Low-rank compression (the MLA idea): Project keys (K) and values (V) into a lower-dimensional latent space, then cache the compressed latents instead of the full K/V. This substantially reduces KV Cache memory, which is especially valuable at 1M context lengths.
- Sparse selection: Instead of processing every token in a long history, a lightweight Indexer scores which tokens are most relevant to the current query. Main attention then processes only about 2,048 positions, rather than all million-plus tokens.
A sketch of the implementation:
def sparse_attn(q, k, v, indexer, top_n=2048, reuse_index=None):if reuse_index is None:scores = indexer(q, k) # [tokens, seq_len] lightweight relevance scoringidx = scores.topk(top_n).indiceselse:idx = reuse_index # Reuse token indices selected in earlier layers.return sdpa(q, gather(k, idx), gather(v, idx))
Sparse attention approximately turns quadratic attention cost into linear cost. It also keeps the model sensitive to key information, such as function definitions in code or named entities in documents, by selecting only the most relevant positions.
It pairs naturally with MLA’s low-rank cache: if you have to cache KV states anyway, store them in a more memory-efficient compressed form and spend detailed computation only on the selected positions.
4. Layer-Wise Index Sharing: Select Once, Reuse Across Layers
Index reuse is a pragmatic middle ground. In this example, all 78 layers use sparse attention, but only 21 of them (layers 0, 1, 5, 9, 13, …, 77) run the full Indexer to select relevant tokens. The other 57 layers reuse the token indices chosen by earlier layers.
The rationale is that attention patterns in adjacent layers are highly similar. If every layer selected tokens independently, the Indexer’s own overhead would add up into a meaningful cost. Sharing indices preserves the sparsity of the main attention while spreading the selection cost across a handful of layers.
The trade-off is coarser selection: layers that share the same set of indices cannot tailor token selection to their own characteristics. It is essentially a trade-off between compute cost and modeling precision, and a widely applicable optimization pattern worth borrowing.
5. Multi-Path Residual Streams and Multi-Token Prediction
Multi-path residual streams (as used in iHC-style designs): A standard residual connection provides a single main information highway, which can become a bottleneck in deep networks. Running 4 parallel residual streams gives gradients and features multiple redundant paths, which helps keep training stable and information flowing through deep networks. The cost is higher memory and bandwidth usage, so you have to balance capacity against cost.
Multi-Token Prediction (MTP): The model includes a larger native draft head (about 10B total parameters and 0.7B active parameters) that predicts multiple tokens at once during inference. Paired with speculative decoding, it can substantially improve time-to-first-token and decoding throughput. The draft head quickly proposes candidate tokens, and the main model verifies them in parallel. For online serving where low latency matters, it is a key technique for making the model feel faster.
6. Precision and Quantization: BF16 and FP8 Side by Side
The model offers BF16 for training and precision benchmarks, which helps keep results reproducible. It also provides an official FP8 option (such as MXFP8-style microscaling formats) for faster inference and lower memory usage.
FP8 introduces independent scaling factors for each block of data. That cuts bandwidth and compute requirements significantly while keeping quantization error within an acceptable range. For large-scale deployment, FP8 inference is often where cost-effectiveness starts to improve. However, precision sensitivity varies across different operators.
7. How to Evaluate These Models Without Being Misled by Leaderboards
When a new model lands, an engineering-focused evaluation matters far more than benchmark scores:
- Watch out for benchmark contamination: When training data overlaps heavily with evaluation sets, scores are systematically inflated, which makes them less useful as a reference. It is more reliable to compare models on private evaluation sets that reflect your own use cases.
- Look at both total and active parameters: Total parameters set the upper limit of knowledge capacity, while active parameters determine compute cost. Weigh them against your workload. Agent tasks, for instance, depend more on effective context length and instruction following than on parameter count alone.
- Do the total-cost math: Factor in input and output pricing, context length, and concurrent throughput together. For long-context models, price per token alone does not tell the full story. You also need to account for the actual token consumption and latency of a typical task.
- Evaluate beyond benchmark scores: Multimodal support, the reliability of tool calling, long-context retrieval accuracy, and precision regressions after FP8 deployment often decide whether a model can actually go into production.
8. Practical Recommendations
If you are choosing a model for long-context workloads, consider this sequence:
- Shortlist candidates using your private evaluation sets.
- Validate key capabilities such as long-context retrieval and tool calling.
- Stress-test inference with FP8 to measure cost per unit of work.
- Reduce latency further with techniques such as KV cache reuse, index sharing, and MTP-based speculative decoding.
At the architecture level, MoE, sparse attention, low-rank KV caching, and multi-path residuals have become a common combination in long-context LLMs. Understanding the specific problem each module solves is more valuable in the long run than memorizing any particular parameter count.
