Large-Scale MoE Model Architecture: Core Advantages and Key Technical Design
This article breaks down the architecture of large-scale Mixture of Experts (MoE) models and explains why decoupling parameters from compute has become the main line of large language model evolution. It covers MoE sparse routing, fine-grained and shared experts, load balancing, capacity control, low-rank KV cache compression for long-context efficiency, sparsity scaling laws, Muon-style optimizers, and large-scale parallelism strategies. It also offers a framework for reading model architecture from technical reports, along with practical guidance on enterprise selection, deployment, and cost estimation.
1. Introduction: Decoupling Parameters from Compute Is the Main Line of Architecture Evolution
Over the past few years, large language models have evolved through roughly three stages. The first stage relied on dense architectures, where capability improved mainly by “piling on parameters.” The second stage introduced Mixture of Experts (MoE) structures, which separated total parameter count from the compute used per token. The third stage—the direction widely adopted by today’s leading large models—builds on sparsity and pushes further into “precision engineering,” optimizing attention cost, routing efficiency, and training stability at the same time.
A widely accepted conclusion in the industry is that, under a fixed training compute budget, allocating more parameters together with more data usually pays off better than simply making a dense network deeper or wider. MoE is precisely the key engineering means for achieving that allocation. Taking a trillion-parameter-class MoE as an example, the ratio of total parameters to activated parameters per token can reach tens to one. A representative model of this class has the following specifications:
- Hidden dimension: 7168
- Attention heads: 64, with 128 dimensions per head
- Attention: low-rank compressed KV cache
- Experts: 384 routed experts, plus dedicated shared experts
- Each routed expert: intermediate dimension 2048 (about 0.29× the model’s hidden dimension)
- Activation strategy: top-k routed experts per token, with k typically 6–8
- Pretraining corpus: 15.5T tokens
Based on these numbers, we can estimate:
- Parameters per routed expert: With a gated structure, each expert has 3 weight matrices, roughly 3 × 7168 × 2048 ≈ 4.4×10⁷ (44 million). Across 384 experts, this totals about 1.7×10¹⁰ (~16.9 billion)—only about 1.7% of the total parameters.
- Parameters consumed by the backbone: Attention, shared experts, token embeddings, and the output layer make up the bulk, totaling roughly 8×10¹¹ (800 billion).
- What the hidden dimension buys: 7168 divided by 128 dimensions per head equals 56, while the actual head count is 64. This shows that the projection dimension (n_h × d_h = 8192) is independent of the hidden dimension. A larger hidden dimension strengthens the representational capacity of the backbone, while a smaller per-expert intermediate dimension keeps each expert “light”—making it easy to raise overall capacity by increasing the number of experts and expanding the combinatorial space.
From this, the logic of current mainstream large-model architecture design can be summarized at three levels:
- Parameter level: Use MoE to decouple parameters from compute, trading sparsity for total capacity.
- Attention level: Use low-rank compression to reduce the KV cache and control long-context cost.
- Training level: Use load balancing, optimizers, and large-scale parallelism strategies to ensure stable convergence of trillion-parameter models.
The following sections break down each of these three layers, and the article closes with an approach for evaluating enterprise selection and deployment.
2. MoE Sparse Architecture: Separating Parameter Scale from Inference Compute
2.1 Basic Principles
In a standard Transformer, the feed-forward network (FFN) usually accounts for about two-thirds of the parameters in a layer, and it is computed in full for every token. MoE replaces a single FFN with E parallel “expert” subnetworks, then uses a gating network to select the Top-K experts for each token. Only the selected experts participate in computation; the parameters of the remaining experts sit “idle” for that forward pass.
Expressed as formulas:
# h_t: hidden representation of the t-th token at the current layer# s_t: gating output, dimension E (total number of experts)# g_t: normalized gate weights, g_t = TopKSoftmax(s_t, k)# Normalization (along the expert dimension)denom = sum_{j in TopK(s_t)} exp(s_t[j])g_t[i] = exp(s_t[i]) / denom if i in TopK(s_t), else 0# MoE layer outputy_t = sum_{i=1}^{E} g_t[i] * E_i(h_t)
This way, total parameter count is determined by E, while per-token compute is determined only by k—so the two are decoupled. Note that MoE reduces compute, not memory footprint. All expert weights must either stay resident in memory or be distributed across nodes. Therefore, total parameters determine memory cost, while activated parameters determine compute cost—the basis for the cost analysis below.
2.2 Fine-Grained and Shared Experts: Two Ways to Increase Capacity
Two common approaches have emerged in the industry’s widely adopted practice of scaling capacity through sparsity:
- Fine-grained partitioning: Shrink each expert’s intermediate dimension (e.g., to 2048) while increasing the expert count (e.g., from 256 to 384). With a comparable total parameter count, the number of selectable combinations grows from C(256, k) to C(384, k), letting the model express finer-grained knowledge combinations.
- Shared experts: Set up a small number of experts that are always active for every token, carrying general knowledge and common patterns, while routed experts focus on learning differentiating features.
The rationale is that gating-based routing is essentially a discrete choice and tends to oscillate between “general knowledge” and “specialized knowledge.” Fixing the general part gives routing a stable foundation, reducing routing jitter and improving training stability.
2.3 Load Balancing: The First Challenge in MoE Training
Without constraints on routing, the gating network quickly falls into a “winner-take-all” regime: a few experts absorb large numbers of tokens, while the rest receive almost no gradient, sharply reducing parameter utilization. Two common solutions exist in practice:
- Auxiliary loss balancing. Let f_i be the fraction of tokens in a batch actually received by expert i, and P_i the average gating probability assigned to it. The balancing loss is:
L_aux = alpha * N_e * sum_i (f_i * P_i) # N_e: total experts, alpha: balancing coefficient
The meaning is: the more an expert is used and the more the gate favors it, the larger the penalty.
- Lossless balancing. Maintain a dynamically adjusted bias term for each expert that is used only for routing selection and does not enter the normalization of the gate weights. If an expert is overloaded, lower its bias; if underloaded, raise it. This achieves balancing without interfering with the main-task gradient.
2.4 Capacity Factor and Token Dropping
To keep the computation graph regular on each device, implementations typically set an “expert capacity”:
capacity = ceil(capacity_factor * T * k / N_e) # T: number of tokens in the batch
Tokens within capacity are computed normally; tokens exceeding it are dropped or passed straight through via shared experts. A larger capacity factor means fewer dropped tokens but more wasted memory and compute; a smaller one improves training efficiency but may affect convergence quality. This is an engineering variable that must be tuned continuously during training.
2.5 Sample Code: Core MoE Routing Logic
import torchimport torch.nn.functional as Fdef moe_forward(x, gate, experts, top_k=8, capacity_factor=1.25,aux_coef=1e-2, num_experts=384):"""x: [T, d] input token representationsgate: [d, E] gating weightsexperts: module list, each expert is a 2-layer FFN (intermediate dim 2048)Returns: MoE output [T, d] and auxiliary loss (scalar)"""T, _ = x.shape# 1) Score with the gate and take Top-Kscores = x @ gate # [T, E]topk_scores, topk_idx = torch.topk(scores, top_k, dim=-1)# 2) Normalize within Top-K to obtain sparse gate weightsweights = F.softmax(topk_scores, dim=-1) # [T, k]# 3) Compute the load-balancing auxiliary loss: f_i is the dispatch ratio, P_i is the mean gate probabilityone_hot = F.one_hot(topk_idx, num_experts).float().sum(1) # [T, E]f = one_hot.mean(dim=0) # actual dispatch ratioP = scores.softmax(dim=-1).mean(dim=0) # mean gate probabilityaux_loss = aux_coef * num_experts * (f * P).sum()# 4) Dispatch tokens by capacity (illustrative: tokens exceeding capacity pass through via residual)cap = int(capacity_factor * T * top_k / num_experts)out = x.clone() # tokens not routed to any expert keep their original valuefor e_idx, expert in enumerate(experts):# Collect the (token_id, slot) pairs that selected this expertmask = (topk_idx == e_idx)tok_ids, slot_ids = mask.nonzero(as_tuple=True)if tok_ids.numel() == 0:continue# Each expert processes at most cap tokens; the rest pass through via residualtok_ids, slot_ids = tok_ids[:cap], slot_ids[:cap]h = expert(x[tok_ids]) # expert computationw = weights[tok_ids, slot_ids].unsqueeze(-1) # gate weightingout.index_add_(0, tok_ids, w * h)return out, aux_loss
In real systems, this logic is replaced by implementations based on grouped routing, All-to-All communication, and fused operators, but the core idea is the same: Top-K sparse activation + normalized weighting + load-balancing constraints.
3. Attention Mechanism: KV Cache Compression Determines Long-Context Cost
3.1 The Problem: Memory and Bandwidth Bottlenecks in Long Contexts
During autoregressive decoding, generating each new token requires reading the Key/Value cache of all historical tokens. The KV cache size for standard multi-head attention is roughly:
# L: number of layers, H: number of KV heads, d_h: dim per head, S: context length, bytes: bytes per elementKV_bytes = 2 * L * H * d_h * S * bytes # 2 for one copy each of K and V
When the context grows from 4K to 128K, the cache size increases 32× linearly. For a server with 80GB of memory per card, long-context scenarios often hit a wall not from insufficient compute, but from memory capacity and access bandwidth reaching their limits first.
3.2 Low-Rank Compressed Attention
To ease this problem, the industry has developed several classes of low-rank compressed attention. The common idea is: instead of directly caching the full K and V for every head, jointly compress each token’s Key and Value into a low-dimensional “latent vector.” At decode time, an up-projection matrix reconstructs the per-head K and V from the latent vector.
A sketch of the implementation:
class LatentKVCache(nn.Module):"""Low-rank compressed KV cache (illustrative implementation; RoPE positional encoding omitted)"""def __init__(self, d_model, n_heads, d_head,d_latent=512, d_latent_rope=64):super().__init__()self.n_heads, self.d_head = n_heads, d_head# Down-projection: d_model -> d_latent (joint KV compression)self.W_dkv = nn.Linear(d_model, d_latent, bias=False)# Decoupled position-related subspace (only this small part is cached)self.W_kr = nn.Linear(d_model, d_latent_rope, bias=False)# Up-projection: d_latent -> n_heads * d_head (one set each for K and V)self.W_uk = nn.Linear(d_latent, n_heads * d_head, bias=False)self.W_uv = nn.Linear(d_latent, n_heads * d_head, bias=False)# Register as cache buffersself.register_buffer("cache_latent", None) # compressed latent vectorsself.register_buffer("cache_k_rope", None) # position-related partdef forward(self, h):c = self.W_dkv(h) # [T, d_latent] one copy cached per tokenkr = self.W_kr(h) # [T, d_latent_rope]k = self.W_uk(c).view(-1, self.n_heads, self.d_head)v = self.W_uv(c).view(-1, self.n_heads, self.d_head)# In a real implementation, K is concatenated with kr before RoPE, and history cache is appended as neededreturn k, v, c, kr
Comparing cache sizes. With d_latent = 512, d_latent_rope = 64, and FP16 precision:
- Cache per token per layer is about (512 + 64) × 2 ≈ 1.1KB;
- Estimating roughly 60 layers, cache per token is about 66KB;
- A 64K context uses about 4GB, and 128K about 8GB.
By contrast, with ordinary multi-head attention—even after compressing KV heads down to 8 (i.e., GQA)—a 64K context still requires about 15GB. The gap is more than 3×; compared with uncompressed multi-head attention, the gap can reach an order of magnitude.
Joint K/V compression. In traditional designs, K and V are stored independently. Joint compression lets the two share information in a low-dimensional subspace, reducing information loss at high compression ratios. The cost is an extra up-projection at decode time—trading “a bit more compute” for “much less storage.”
3.3 The Trade-off on the Number of Attention Heads
In ordinary multi-head attention, KV cache size is proportional to the number of KV heads, so reducing the head count directly lowers the cache. With low-rank compression, however, cache size is largely independent of the head count, and the meaning of the head count shifts:
- Benefit: At decode time, each head must up-project the latent vector, so halving the head count roughly halves that compute, improving decode throughput accordingly.
- Cost: In compute-intensive phases such as prefill, reducing heads weakens multi-head attention’s ability to model in parallel across different subspaces.
Thus, whether to reduce the head count is essentially a trade-off between “decode efficiency” and “modeling capacity.” The mainstream approach is to moderately reduce heads while keeping per-head dimension fixed (e.g., from 128 down to 64), and to compensate for the loss in representational capacity using the backbone capacity that a large hidden dimension provides.
3.4 Comparing Attention Schemes
| Scheme | Cached content | Cache size | Main advantage | Main limitation |
|---|---|---|---|---|
| MHA (multi-head attention) | Independent K, V per head | Largest | Strong expressiveness, simple to implement | High memory cost in long contexts |
| MQA / GQA | A few KV heads shared | Medium | Significant cache reduction, mature and stable | Sharing limits multi-head expressiveness |
| Low-rank compressed attention | Low-dim latent vector + positional subspace | Smallest | High compression ratio, cost-efficient for long context | Requires up-projection at decode, more complex to implement |
4. Sparsity Scaling Law: Why Experts Should Be “Many and Narrow”
The gains from sparsification are not unlimited. The prevailing conclusion from Scaling Law research is that, under a given compute budget, increasing the number of experts usually brings sizable performance gains, but the returns gradually diminish and are constrained by two factors:
- Routing quality: The more experts there are, the harder it is for the gate to learn a clear division of labor, and the more likely it is to fall into “routing noise.”
- Communication overhead: When experts are distributed across different devices, All-to-All traffic grows with the number of experts and devices. Beyond a certain scale, the compute gains from adding experts are eaten up by communication cost.
Taking a representative model as an example, increasing the expert count from 256 to 384 is a 50% increase; the per-expert intermediate dimension is 2048, only about 0.29× the model’s hidden dimension. This “many and narrow” design is essentially trading combinatorial complexity for parameter efficiency: only when each expert is small enough can more experts be stacked under the same compute; only when the combinatorial space is large enough can the model learn fine-grained specialization.
This also answers a common question: why does model performance still improve as total parameters grow, even though activated parameters are only a fraction of the total? Because MoE performance comes from “the sum of knowledge across all experts,” not the portion used in a single forward pass. Routing only decides “which part to use”; knowledge capacity is always determined by the total parameters.
5. Training Strategy: Muon-Style Optimizers and Large-Scale Parallelism
5.1 Optimizer Evolution: From Element-Wise Adaptive to Matrix Orthogonalization
Mainstream training has long relied on element-wise adaptive optimizers such as AdamW: each maintains first- and second-order momentum for every parameter, with updates determined by each parameter’s own gradient statistics. Its limitation is that this update ignores the internal correlations within parameter matrices—the update direction of each dimension is independent of the others.
In recent years, a class of optimizers represented by Muon (the name comes from MomentUm Orthogonalized by Newton-Schulz) has drawn attention. Its core idea is to first orthogonalize the momentum matrix (in practice often approximated via Newton-Schulz iterations) before using it to update the parameters. You can think of it as a “matrix-level” optimizer: it treats a matrix as a whole, keeps singular values balanced during updates, and avoids some directions being updated too aggressively while others barely move. Related research indicates that, in large-scale training, it can reach comparable or even better convergence with fewer training steps.
Two engineering considerations:
- Extra overhead: Newton-Schulz iterations require multiple matrix multiplications, adding compute overhead; this is typically controlled via low-precision implementation.
- Mixed usage: A common practice is to use it to update hidden-layer weight matrices while still using adaptive optimizers for token embeddings, the output layer, and gating—balancing stability and efficiency.
def newton_schulz_orthogonalize(G, steps=5):"""Approximately orthogonalize the gradient/momentum matrix G (core Muon step, illustrative)After convergence, G's singular values approach 1, equivalent to the orthogonal part of the polar decomposition"""X = G.bfloat16()X = X / (X.norm() + 1e-7) # Normalize by Frobenius norma, b, c = (3.4445, -4.7750, 2.0315) # Coefficients of the fifth-order approximating polynomialfor _ in range(steps):A = X @ X.T # A = X X^TB = b * A + c * (A @ A) # Polynomial accelerationX = a * X + B @ X # Update Xreturn X.float()# Illustration: momentum buffer -> orthogonalize -> update parametersdef muon_step(param, buf, lr, weight_decay):buf.mul_(0.95).add_(param.grad, alpha=1.0) # Momentum accumulationupdate = newton_schulz_orthogonalize(buf)param.data.mul_(1 - lr * weight_decay) # Decoupled weight decayparam.data.add_(update, alpha=-lr) # Parameter update
5.2 Pretraining Data Scale and Composition
Taking a pretraining scale of 15.5T tokens as an example, its magnitude sits at the upper-middle range of the pretraining scale common to mainstream dense models (a few trillion to just over ten trillion tokens). More important than the total is the composition and quality:
- The ratio of general corpora to code, math, and other reasoning-oriented corpora directly determines the model’s reasoning ability;
- The coverage of multilingual corpora affects the model’s international capabilities;
- Data deduplication and quality filtering significantly reduce memorization and overfitting;
- The later stage of training is often paired with annealing and upsampling of high-quality data to further improve results within a limited number of steps.
5.3 Large-Scale Parallelism Strategies
Training a trillion-parameter MoE requires combining multiple parallelism strategies:
- Data parallelism (DP): Each device holds a full model replica, with gradient synchronization.
- Tensor parallelism (TP): Split single-layer weight matrices across cards, suited to scenarios with heavy per-layer compute.
- Pipeline parallelism (PP): Partition by layer across devices to form a pipeline execution.
- Expert parallelism (EP): Different experts are placed on different devices; tokens are dispatched to the device holding the target expert via All-to-All.
- Sequence / context parallelism: During long-context training, partition along the sequence dimension.
The parallelism challenge for MoE concentrates on All-to-All communication: each MoE layer forward requires two global dispatches (one before entering experts, one to aggregate on return), and communication volume is proportional to the number of tokens in the global batch, the hidden dimension, and the number of activated experts. Common engineering mitigations include:
- Overlapping communication with computation;
- Grouped routing limits, restricting tokens to being routed only to experts on the same node;
- Pipeline bubble optimization.
6. Horizontal Comparison: How to Read Model Architecture from Technical Reports
An interesting phenomenon has emerged in the industry: multiple models adopt highly similar architectures in their technical reports (e.g., MoE + low-rank compressed attention) and provide detailed parameter comparisons against models in the same category. This does not mean architectural convergence is without value; on the contrary, it shows that sparsity plus attention compression has become a de facto industry standard, and the focus of competition is shifting from “architectural prototype innovation” toward “training efficiency, data quality, and engineering execution.”
For enterprises, the following dimensions deserve attention when reading technical reports:
| Dimension | What to focus on | Impact on business |
|---|---|---|
| Total / activated parameters | The ratio of the two determines memory and compute needs | Affects deployment cost and throughput |
| Attention mechanism | Whether KV compression and long context are supported | Affects cost-efficiency in long-document scenarios |
| Number and granularity of experts | Total experts, per-expert dimension | Affects performance on fine-grained tasks |
| Training data scale | Total tokens and composition | Affects domain adaptability |
| Optimizer and parallelism strategy | Training efficiency and stability | Affects ability to iterate continuously |
| Context length | Maximum supported tokens | Affects RAG and long-document analysis scenarios |
7. Enterprise Selection and Deployment Recommendations
- Define the business scenario first: Long-document understanding, multi-turn dialogue, code generation, and structured extraction place very different demands on model capability, so establish an evaluation set before selecting a model.
- Do cost estimation upfront: Memory footprint is roughly “total parameters × bytes per precision unit + KV cache,” which lets you estimate the concurrency a single machine can handle; compute needs relate to activated parameters and input/output length.
- Pay attention to long-context cost-efficiency: Models using low-rank compressed attention have clearly lower cache overhead in contexts above 64K, making them well suited to long-document analysis workloads.
- Choose the deployment form flexibly: You can quickly validate results via a public cloud API; for data-sensitive workloads, use dedicated instances or private deployment; you can also use container platforms for elastic scaling, combined with logging services and monitoring/alerting to ensure service stability. For example, Baidu AI Cloud’s Qianfan platform provides access to and fine-tuning for a range of mainstream large models, and supports meeting enterprise needs through multiple options such as public cloud APIs and private deployment.
- Build an evaluation loop: Combine general-capability evaluations with business metrics (accuracy, latency, cost) rather than looking at leaderboards alone.
8. Summary
The core advantage of the MoE architecture lies in “decoupling”: separating parameter scale from per-token compute lets a model hold far greater knowledge capacity under a controllable compute budget. Low-rank compressed attention solves the memory and bandwidth bottleneck in long contexts; load balancing and lossless routing ensure the training stability of trillion-parameter models; and optimizer innovation together with large-scale parallelism continues to improve training efficiency.
The parameter comparisons in technical reports also show that sparsity plus attention compression is becoming the de facto standard across leading large models, and the focus of competition has shifted toward training efficiency, data quality, and engineering execution. For enterprises, understanding these designs is the key to making more rational decisions in selection, deployment, and cost estimation.
