Ultra-Large-Scale MoE Language Models: Architecture Optimization Explained
This article explains how ultra-large-scale Mixture of Experts (MoE) language models are engineered: sparse expert routing and expert parallelism, load balancing and capacity factors, MLA for KV cache compression, Muon with per-head orthogonalization, multi-token prediction, sparse attention, and the pre-training and mid-training data stages. It also covers how model capability is measured and gives practical guidance for memory planning and deployment.
1. Why MoE Became the Go-To Architecture for Ultra-Large Models
In a traditional dense architecture, every token passes through all of the model’s parameters on every forward pass, which ties parameter count almost directly to compute cost. As a result, making a model “smarter” also makes it slower and more expensive, and scaling up quickly runs into hard limits on compute and memory.
Mixture of Experts (MoE) takes a different path. It places a large number of expert subnetworks side by side inside the model, but during inference a routing mechanism lets only a small fraction of them participate in each computation. This sparse activation decouples parameter capacity from per-step compute cost: total parameters can be made very large to carry more knowledge, while the parameters actually used in computation remain manageable.
From an engineering perspective, evaluating an MoE model usually comes down to two distinct metrics:
- Total parameters, which set the model’s knowledge ceiling and overall capacity;
- Activated parameters, which determine the real compute and memory cost of each forward and backward pass.
A typical configuration for a new generation of ultra-large models is a few hundred billion total parameters (for example, around 700 billion), with only about 5%–6% activated per inference (for example, around 40 billion). This “high capacity, low activation” combination is a key enabler for such models to approach or even match the capabilities of top-tier closed-source models under limited compute.
2. Expert Routing and Parallelism: A Simplified View
At the core of an MoE model is a gating network. It computes a score for how suitable each expert is for a given token, and usually only the top K experts are selected. The illustrative code below captures the essence of that sparsity:
import torchimport torch.nn.functional as Fdef moe_forward(x, experts, gate, top_k=2):# x: [batch, seq, dim]logits = gate(x) # scores each token against all expertstopk_val, topk_idx = logits.topk(top_k, dim=-1)weights = F.softmax(topk_val, dim=-1) # normalize over the selected expertsout = torch.zeros_like(x)for k in range(top_k):idx = topk_idx[..., k]w = weights[..., k].unsqueeze(-1)for e in range(len(experts)):mask = (idx == e)if mask.any():out[mask] += w[mask] * experts[e](x[mask])return out
The real engineering challenges lie not in routing itself but in two other areas. The first is load balancing: without it, a few experts can be selected too often while the rest are “starved.” The second is communication overhead: when experts are spread across different devices (expert parallelism), All-to-All communication can become the throughput bottleneck. For this reason, modern MoE training commonly adds an auxiliary load-balancing loss and a capacity factor that caps how much each expert can process.
3. Attention Mechanisms: MLA and KV Cache Compression
Beyond expert computation, the KV cache of attention is often another major source of memory consumption in long-text scenarios. During autoregressive generation, the model must cache the keys and values of all previous tokens. The longer the sequence, the larger the cache, which makes long-context scenarios especially demanding.
Multi-head Latent Attention (MLA) takes a different approach. Instead of caching the full K and V, it projects them into a lower-dimensional “latent vector,” caches that vector, and decompresses it to reconstruct K and V when computing attention. This substantially reduces the memory footprint of the KV cache, improving long-text throughput and the context length the model can support. The trade-off is extra projection and decompression computation, so it comes down to balancing memory savings against one additional step of computation.
4. Optimizers and Orthogonalization: Muon Split by Attention Head
In large-scale training, the choice of optimizer also affects convergence quality. Optimizers such as Muon apply orthogonalization to the momentum associated with each parameter matrix. Intuitively, this produces a more even singular value distribution and a better condition number, which leads to more stable update directions.
In practice, an interesting pattern emerged. When Muon was combined with compressed attention such as MLA, orthogonalizing the entire projection matrix in one pass performed worse than simpler approaches such as grouped-query attention (GQA). The team subsequently found an improvement based on divide and conquer. Instead of stacking multiple attention heads together and orthogonalizing them as one block, they applied orthogonalization separately for each attention head, an approach that can be understood as Muon Split. The reasoning is that different heads already carry different semantic subspaces. A single, unified orthogonalization artificially couples them and disrupts their individual learning behavior. Separating them improved both convergence stability and final performance.
5. Multi-Token Prediction (MTP)
Standard training predicts only the next token at each position, so the supervision signal is relatively sparse. Multi-token prediction (MTP) has the model predict several future tokens at once from the same position. It brings two benefits. First, the training signal is denser, which often improves sample efficiency and representation quality on the same amount of data. Second, it naturally fits speculative decoding: the multi-step prediction ability learned during training can serve as a draft model to speed up inference-time generation.
6. Sparse Attention: Tackling Quadratic Complexity at Very Long Contexts
Standard self-attention has quadratic complexity in sequence length, so compute grows sharply as the context gets longer. Sparse attention reduces this complexity by attending only to a carefully chosen subset of tokens rather than all of them. The key design challenge is the selection strategy: it needs to cover the genuinely important long-range dependencies, while keeping the selection itself differentiable or learnable, or inexpensive. Sparse attention is often used together with the KV cache compression discussed above to jointly support long-context capabilities.
7. Data Scale and Training Stages
Model capability comes not only from architecture but also from data. Training for this class of ultra-large models is typically split into two stages:
- Pre-training: carried out on a massive, multi-source corpus at the trillion-token scale (for example, tens of trillions of tokens) to build general knowledge and language ability;
- Mid-training / continued pre-training: injects more high-quality curated data, such as long-text, code, reasoning, and math corpora, to deliberately strengthen specific capabilities.
It is worth noting that scaling up data (for example, from around 23T to around 28T tokens) works hand in hand with architectural optimization. A better architecture improves how efficiently the model absorbs each unit of data, while more and higher-quality data gives the new architecture room to reach its potential.
8. How to Objectively Measure Model Capability
To evaluate models of this class, the industry typically looks at several complementary dimensions, since a single metric is not enough to tell the full story:
- General-purpose capability benchmarks: a standardized set of problems that produces a comparable “intelligence index” score, making side-by-side comparison easier;
- Crowd-sourced arena-style evaluations: preference votes from real users on blind, head-to-head comparisons (such as text conversations and code generation), which better reflect the actual user experience;
- Open-source vs. closed-source positioning: whether open-source models have caught up to the level of top closed-source models.
For the open-source ecosystem, an important milestone is when an open-source model first reaches around 50 points on a mainstream general capability index and performs in the same tier as top closed-source models on arena-style evaluations. This signals that open-source models are rapidly closing the gap with closed-source ones.
9. Practical Tips for Developers
If you are planning to evaluate or deploy this class of ultra-large MoE models in your business, here are a few points worth attention:
- Plan memory first: whether through KV cache compression or activated-parameter optimization, inference for models this large still requires considerable memory. On platforms such as Baidu AI Cloud, GPU compute instances and container platforms can be used for elastic scaling and orchestration.
- Use managed supporting services: model weights and datasets can be archived centrally through object storage. For online services, integrate logging and monitoring with alerting to continuously track latency, throughput, and error rates.
- Choose specs to fit your needs: among models that are similarly “large and comprehensive,” a lower activated-parameter count means a lower per-step cost. Still, you need to weigh that against the capability ceiling set by total parameters rather than focusing on a single number.
- Prioritize attention design for long-context scenarios: the benefits of optimizations such as MLA and sparse attention are significantly amplified in use cases like long-document processing and codebase understanding.
Conclusion
MoE’s sparse activation, MLA’s KV cache compression, Muon’s per-head orthogonalization, multi-token prediction, and sparse attention do not exist in isolation. They work together around the same core goal: expanding the boundaries of model capability while keeping inference costs under control. Understanding the trade-offs behind these designs is more valuable than memorizing specific parameter values. It determines whether you can choose the right model and use it effectively under real-world compute constraints.
