MoE Large Model Architecture and Efficient Attention: Sparse Activation, MLA KV Cache Compression, and Long-Context Engineering
As the focus of new-generation large models shifts from scaling parameters to improving efficiency, this article explains MoE's sparse activation, MLA's low-rank KV Cache compression, Muon-style orthogonalized matrix optimizers, multi-token prediction, and sparse attention for long context. It clarifies the difference between total and activated parameters, provides memory estimation formulas and scenario-based selection advice, and closes with a go-live checklist for MoE deployment and a set of FAQs.
Introduction: From More Parameters to Better Efficiency
The focus of competition among new-generation large models is shifting from simply scaling up parameter counts to improving efficiency. MoE raises knowledge capacity through sparse activation, MLA cuts KV Cache overhead through low-rank compression, matrix-based optimizers improve training stability, multi-token prediction raises sample efficiency, and sparse attention reduces the cost of long context. Around these core designs, this article provides practical selection criteria, a memory estimation framework, and an engineering checklist.
1. Why Are New-Generation Models Moving from Dense to MoE?
1.1 The Bottleneck of Dense Models
In a conventional dense architecture, every token activates all parameters on each forward pass. As parameter count grows, per-step compute grows at roughly the same rate. When scale moves from tens of billions to hundreds of billions of parameters, training and inference costs rise quickly, while marginal returns do not necessarily improve at the same rate. This is a structural contradiction that the dense route cannot avoid.
1.2 The Core Idea of MoE: Sparse Activation
MoE (Mixture of Experts) does not use one feed-forward network for all tokens. Instead, it defines multiple expert subnetworks and uses a routing mechanism to select a few experts to participate in computation for each token:
def moe_layer(x, experts, router, top_k=2):weights = router(x)top_idx = topk(weights, k=top_k)# Real implementations usually also re-normalize the weights of the selected expertsselected_weights = normalize(weights[top_idx])out = 0for idx, w in zip(top_idx, selected_weights):out += w * experts[idx](x)return out
Real engineering also involves capacity factors, token dropping, and expert parallel communication, so it cannot be reduced to “just run two experts”.
1.3 Parameters vs. Activated Parameters: Not Fully Decoupled
An illustrative configuration gives a sense of the magnitudes: about 700B total parameters, about 40B activated per token, hundreds of experts across dozens of layers. This is illustrative only and does not correspond to a specific model. When citing a real model, you must go back to the official model card or paper and state the exact definition, including whether the activated count covers shared experts, attention, embeddings, and so on.
A more accurate statement: total parameters determine knowledge capacity and weight memory footprint, while activated parameters determine compute per token. The two are related but not fully decoupled. MoE’s sparse activation shortens the per-token compute path but does not remove all costs — in particular, all expert weights must either stay resident in memory or be spread across devices through expert parallelism, offloading, and similar techniques.
2. How MLA Compresses KV Cache with Low-Rank Projection
2.1 Why KV Cache Is the Long-Context Bottleneck
Multi-head attention must cache the Keys and Values of historical tokens during autoregressive decoding. KV Cache size can be approximated as:
KV memory ≈ layers × heads × head dim × 2 × sequence length × bytes per element
The “2” accounts for the two caches, K and V. For example, with 80 layers, 128 heads, head dim 128, a 32K context, and BF16 storage, this illustrative calculation alone reaches the order of hundreds of GiB. Configurations differ greatly across models, so always treat the official configuration as the source of truth.
2.2 The Low-Rank Idea Behind MLA
MLA (Multi-head Latent Attention) does not cache the full K and V. Instead, it projects them into a lower-dimensional latent vector, caches that vector, and projects back up when it is needed:
Full cache: K/V dimension is tied directly to the number of heads and the head dimLow-rank cache: latent vector dimension is far smaller than the total K/V dimension
This substantially reduces KV Cache size. Using FP8 or lower precision can compress it further, but accuracy regression must be evaluated. How much MLA compresses relative to GQA depends on the rank, the head-sharing strategy, precision, and the specific implementation, so no fixed gain can be claimed without the configuration.
3. Matrix Optimizers: Why Orthogonalize the Update?
In large-scale pre-training, adaptive optimizers remain the mainstream, but a class of optimizers built on matrix structure, such as Muon-style methods, has started to attract attention. A verifiable description is this: the momentum matrix of a two-dimensional weight matrix is approximately orthogonalized — for example via Newton-Schulz iteration — so that the singular values of the update matrix become closer to uniform. This reduces scale differences between update directions and improves stability at high learning rates.
Boundaries to keep in mind:
- Such optimizers typically apply only to two-dimensional matrix weights;
- One-dimensional parameters such as embeddings, the output head, LayerNorm, and bias usually still use AdamW;
- Distributed training requires communicating momentum matrices across the data-parallel group;
- Whether it beats conventional optimizers should be validated by ablation experiments across model scale, parallelism strategy, and data distribution.
Talking about the “best optimizer” without the concrete training configuration is meaningless.
4. Multi-Token Prediction: Raising the Training Ceiling, Cashing Out at Inference?
Multi-Token Prediction (MTP) makes the model predict not only the next token but also several future tokens.
During training, it provides a denser supervision signal, improves sample efficiency, and helps the model learn longer-range dependencies. At inference, it can serve as the draft module for speculative decoding: guess several tokens in parallel, then have the main model verify them.
Boundaries that must be added:
- Predicting multiple tokens during training does not mean the model can serve as a draft model directly; a dedicated MTP module or training adaptation is usually required;
- When a draft is rejected, the extra tokens that were computed get discarded;
- In large-batch, high-throughput scenarios, the gains may diminish;
- “No accuracy loss” holds only under standard verification mechanisms such as speculative sampling, not for every implementation.
5. Saving Both Memory and Compute in Long Context
Long context has two distinct bottlenecks:
- Prefill: standard attention compute grows quadratically with sequence length, so this is mainly a compute cost;
- Decode: every step reads the historical KV Cache, so this is mainly a memory capacity and memory-access bottleneck.
Sparse attention — such as DSA (in common implementations, DeepSeek Sparse Attention, though other dynamic sparse schemes exist) — works by using a lightweight indexer to score historical tokens and computing attention over only a subset of key tokens. Common practice includes:
- Approximate scoring of historical tokens;
- Keeping only the top-k tokens;
- Scaling top-k with context length, for example a fixed budget, logarithmic growth, or hierarchical selection;
- The indexer itself carries compute and memory overhead, which must be included in the evaluation.
Sparse attention is usually paired with MLA: MLA compresses the representation of each token, and sparse attention reduces the number of tokens to process. Stacked together, they help make long context both usable and cost-controlled.
6. Engineering: Scenario Selection, Memory Estimation, and a Go-Live Checklist
6.1 Scenario Selection
| Scenario | Preferred options | Watch out for |
|---|---|---|
| High knowledge capacity, distributed training affordable | MoE + MLA | Expert load, EP communication, weight memory |
| Low latency, single-machine deployment | Dense small models, distillation, quantization | Don’t look only at activated parameter count |
| Long-context applications | MLA + sparse attention | Accuracy regression, top-k strategy |
| Small teams shipping fast | Dense or distilled models | Avoid MoE’s engineering complexity |
6.2 MoE Memory Estimation
MoE deployment cannot be judged by activated parameters alone. At inference, all expert weights usually need to stay resident in memory unless expert parallel sharding or expert offloading is used. The memory floor depends mainly on total parameter count, not on activated parameters per token. For single-machine deployment with limited memory, MoE is not necessarily better than a dense model.
A reference formula:
Total memory ≈ weight shard per device + KV Cache + activation and communication buffers
The weight shard is affected by EP, TP, and PP splitting; KV Cache is affected by MLA, sparse attention, context length, and precision; activation and communication buffers are affected by batch size, sequence length, and parallelism strategy.
6.3 MoE Go-Live Checklist
- During training, monitor expert load distribution, routing entropy, expert collapse, and capacity drop rate;
- At inference, account for full weight memory, KV Cache memory, and communication buffers;
- Before enabling MLA or sparse attention, run accuracy regression and long-context evaluation;
- Inference acceleration can combine multi-token prediction, speculative decoding, and quantization;
- Architecture changes should be validated on standard benchmark suites and human preference comparisons, not tuned by intuition.
FAQ
Q1: What is MoE?
MoE is the Mixture of Experts architecture. A router selects a few experts to participate in computation for each token, which controls per-token compute while total parameter count remains large.
Q2: What is the difference between MLA and GQA?
MLA compresses KV Cache through low-rank latent vectors, while GQA reduces the cache by sharing some Key/Value heads. MLA has greater compression potential, but is more complex to implement and tune.
Q3: How do I estimate MoE inference memory?
At minimum, account for all expert weights, KV Cache, and activation and communication buffers. If expert parallelism is used, estimate per-device shards and also check communication overhead.
Q4: What are common MoE mistakes?
Common mistakes include: looking only at activated parameters while ignoring total weight memory; ignoring expert load imbalance; applying activated-parameter conclusions to single-machine deployment; skipping long-context accuracy regression; assuming sparse attention only saves memory, or only saves compute.
Conclusion
The main thread of next-generation model architecture can be summarized as: use sparsity to buy scale, use compression and sparsity to buy efficiency, and use multi-target prediction to gain benefits in both training and inference. What really matters is not memorizing a particular set of parameters, but understanding this “sparse–compress–predict” design paradigm and making engineering choices based on memory, compute, latency, and evaluation metrics.
