Large-Scale MoE Model Architecture: Core Advantages and Key Technical Design

    This article breaks down the architecture of large-scale Mixture of Experts (MoE) models and explains why decoupling parameters from compute has become the main line of large language model evolution. It covers MoE sparse routing, fine-grained and shared experts, load balancing, capacity control, low-rank KV cache compression for long-context efficiency, sparsity scaling laws, Muon-style optimizers, and large-scale parallelism strategies. It also offers a framework for reading model architecture from technical reports, along with practical guidance on enterprise selection, deployment, and cost estimation.

    1. Introduction: Decoupling Parameters from Compute Is the Main Line of Architecture Evolution

    Over the past few years, large language models have evolved through roughly three stages. The first stage relied on dense architectures, where capability improved mainly by “piling on parameters.” The second stage introduced Mixture of Experts (MoE) structures, which separated total parameter count from the compute used per token. The third stage—the direction widely adopted by today’s leading large models—builds on sparsity and pushes further into “precision engineering,” optimizing attention cost, routing efficiency, and training stability at the same time.

    A widely accepted conclusion in the industry is that, under a fixed training compute budget, allocating more parameters together with more data usually pays off better than simply making a dense network deeper or wider. MoE is precisely the key engineering means for achieving that allocation. Taking a trillion-parameter-class MoE as an example, the ratio of total parameters to activated parameters per token can reach tens to one. A representative model of this class has the following specifications:

    • Hidden dimension: 7168
    • Attention heads: 64, with 128 dimensions per head
    • Attention: low-rank compressed KV cache
    • Experts: 384 routed experts, plus dedicated shared experts
    • Each routed expert: intermediate dimension 2048 (about 0.29× the model’s hidden dimension)
    • Activation strategy: top-k routed experts per token, with k typically 6–8
    • Pretraining corpus: 15.5T tokens

    Based on these numbers, we can estimate:

    • Parameters per routed expert: With a gated structure, each expert has 3 weight matrices, roughly 3 × 7168 × 2048 ≈ 4.4×10⁷ (44 million). Across 384 experts, this totals about 1.7×10¹⁰ (~16.9 billion)—only about 1.7% of the total parameters.
    • Parameters consumed by the backbone: Attention, shared experts, token embeddings, and the output layer make up the bulk, totaling roughly 8×10¹¹ (800 billion).
    • What the hidden dimension buys: 7168 divided by 128 dimensions per head equals 56, while the actual head count is 64. This shows that the projection dimension (n_h × d_h = 8192) is independent of the hidden dimension. A larger hidden dimension strengthens the representational capacity of the backbone, while a smaller per-expert intermediate dimension keeps each expert “light”—making it easy to raise overall capacity by increasing the number of experts and expanding the combinatorial space.

    From this, the logic of current mainstream large-model architecture design can be summarized at three levels:

    1. Parameter level: Use MoE to decouple parameters from compute, trading sparsity for total capacity.
    2. Attention level: Use low-rank compression to reduce the KV cache and control long-context cost.
    3. Training level: Use load balancing, optimizers, and large-scale parallelism strategies to ensure stable convergence of trillion-parameter models.

    The following sections break down each of these three layers, and the article closes with an approach for evaluating enterprise selection and deployment.

    2. MoE Sparse Architecture: Separating Parameter Scale from Inference Compute

    2.1 Basic Principles

    In a standard Transformer, the feed-forward network (FFN) usually accounts for about two-thirds of the parameters in a layer, and it is computed in full for every token. MoE replaces a single FFN with E parallel “expert” subnetworks, then uses a gating network to select the Top-K experts for each token. Only the selected experts participate in computation; the parameters of the remaining experts sit “idle” for that forward pass.

    Expressed as formulas:

    1. # h_t: hidden representation of the t-th token at the current layer
    2. # s_t: gating output, dimension E (total number of experts)
    3. # g_t: normalized gate weights, g_t = TopKSoftmax(s_t, k)
    4. # Normalization (along the expert dimension)
    5. denom = sum_{j in TopK(s_t)} exp(s_t[j])
    6. g_t[i] = exp(s_t[i]) / denom if i in TopK(s_t), else 0
    7. # MoE layer output
    8. y_t = sum_{i=1}^{E} g_t[i] * E_i(h_t)

    This way, total parameter count is determined by E, while per-token compute is determined only by k—so the two are decoupled. Note that MoE reduces compute, not memory footprint. All expert weights must either stay resident in memory or be distributed across nodes. Therefore, total parameters determine memory cost, while activated parameters determine compute cost—the basis for the cost analysis below.

    2.2 Fine-Grained and Shared Experts: Two Ways to Increase Capacity

    Two common approaches have emerged in the industry’s widely adopted practice of scaling capacity through sparsity:

    • Fine-grained partitioning: Shrink each expert’s intermediate dimension (e.g., to 2048) while increasing the expert count (e.g., from 256 to 384). With a comparable total parameter count, the number of selectable combinations grows from C(256, k) to C(384, k), letting the model express finer-grained knowledge combinations.
    • Shared experts: Set up a small number of experts that are always active for every token, carrying general knowledge and common patterns, while routed experts focus on learning differentiating features.

    The rationale is that gating-based routing is essentially a discrete choice and tends to oscillate between “general knowledge” and “specialized knowledge.” Fixing the general part gives routing a stable foundation, reducing routing jitter and improving training stability.

    2.3 Load Balancing: The First Challenge in MoE Training

    Without constraints on routing, the gating network quickly falls into a “winner-take-all” regime: a few experts absorb large numbers of tokens, while the rest receive almost no gradient, sharply reducing parameter utilization. Two common solutions exist in practice:

    1. Auxiliary loss balancing. Let f_i be the fraction of tokens in a batch actually received by expert i, and P_i the average gating probability assigned to it. The balancing loss is:
    1. L_aux = alpha * N_e * sum_i (f_i * P_i) # N_e: total experts, alpha: balancing coefficient

    The meaning is: the more an expert is used and the more the gate favors it, the larger the penalty.

    1. Lossless balancing. Maintain a dynamically adjusted bias term for each expert that is used only for routing selection and does not enter the normalization of the gate weights. If an expert is overloaded, lower its bias; if underloaded, raise it. This achieves balancing without interfering with the main-task gradient.

    2.4 Capacity Factor and Token Dropping

    To keep the computation graph regular on each device, implementations typically set an “expert capacity”:

    1. capacity = ceil(capacity_factor * T * k / N_e) # T: number of tokens in the batch

    Tokens within capacity are computed normally; tokens exceeding it are dropped or passed straight through via shared experts. A larger capacity factor means fewer dropped tokens but more wasted memory and compute; a smaller one improves training efficiency but may affect convergence quality. This is an engineering variable that must be tuned continuously during training.

    2.5 Sample Code: Core MoE Routing Logic

    1. import torch
    2. import torch.nn.functional as F
    3. def moe_forward(x, gate, experts, top_k=8, capacity_factor=1.25,
    4. aux_coef=1e-2, num_experts=384):
    5. """
    6. x: [T, d] input token representations
    7. gate: [d, E] gating weights
    8. experts: module list, each expert is a 2-layer FFN (intermediate dim 2048)
    9. Returns: MoE output [T, d] and auxiliary loss (scalar)
    10. """
    11. T, _ = x.shape
    12. # 1) Score with the gate and take Top-K
    13. scores = x @ gate # [T, E]
    14. topk_scores, topk_idx = torch.topk(scores, top_k, dim=-1)
    15. # 2) Normalize within Top-K to obtain sparse gate weights
    16. weights = F.softmax(topk_scores, dim=-1) # [T, k]
    17. # 3) Compute the load-balancing auxiliary loss: f_i is the dispatch ratio, P_i is the mean gate probability
    18. one_hot = F.one_hot(topk_idx, num_experts).float().sum(1) # [T, E]
    19. f = one_hot.mean(dim=0) # actual dispatch ratio
    20. P = scores.softmax(dim=-1).mean(dim=0) # mean gate probability
    21. aux_loss = aux_coef * num_experts * (f * P).sum()
    22. # 4) Dispatch tokens by capacity (illustrative: tokens exceeding capacity pass through via residual)
    23. cap = int(capacity_factor * T * top_k / num_experts)
    24. out = x.clone() # tokens not routed to any expert keep their original value
    25. for e_idx, expert in enumerate(experts):
    26. # Collect the (token_id, slot) pairs that selected this expert
    27. mask = (topk_idx == e_idx)
    28. tok_ids, slot_ids = mask.nonzero(as_tuple=True)
    29. if tok_ids.numel() == 0:
    30. continue
    31. # Each expert processes at most cap tokens; the rest pass through via residual
    32. tok_ids, slot_ids = tok_ids[:cap], slot_ids[:cap]
    33. h = expert(x[tok_ids]) # expert computation
    34. w = weights[tok_ids, slot_ids].unsqueeze(-1) # gate weighting
    35. out.index_add_(0, tok_ids, w * h)
    36. return out, aux_loss

    In real systems, this logic is replaced by implementations based on grouped routing, All-to-All communication, and fused operators, but the core idea is the same: Top-K sparse activation + normalized weighting + load-balancing constraints.

    3. Attention Mechanism: KV Cache Compression Determines Long-Context Cost

    3.1 The Problem: Memory and Bandwidth Bottlenecks in Long Contexts

    During autoregressive decoding, generating each new token requires reading the Key/Value cache of all historical tokens. The KV cache size for standard multi-head attention is roughly:

    1. # L: number of layers, H: number of KV heads, d_h: dim per head, S: context length, bytes: bytes per element
    2. KV_bytes = 2 * L * H * d_h * S * bytes # 2 for one copy each of K and V

    When the context grows from 4K to 128K, the cache size increases 32× linearly. For a server with 80GB of memory per card, long-context scenarios often hit a wall not from insufficient compute, but from memory capacity and access bandwidth reaching their limits first.

    3.2 Low-Rank Compressed Attention

    To ease this problem, the industry has developed several classes of low-rank compressed attention. The common idea is: instead of directly caching the full K and V for every head, jointly compress each token’s Key and Value into a low-dimensional “latent vector.” At decode time, an up-projection matrix reconstructs the per-head K and V from the latent vector.

    A sketch of the implementation:

    1. class LatentKVCache(nn.Module):
    2. """
    3. Low-rank compressed KV cache (illustrative implementation; RoPE positional encoding omitted)
    4. """
    5. def __init__(self, d_model, n_heads, d_head,
    6. d_latent=512, d_latent_rope=64):
    7. super().__init__()
    8. self.n_heads, self.d_head = n_heads, d_head
    9. # Down-projection: d_model -> d_latent (joint KV compression)
    10. self.W_dkv = nn.Linear(d_model, d_latent, bias=False)
    11. # Decoupled position-related subspace (only this small part is cached)
    12. self.W_kr = nn.Linear(d_model, d_latent_rope, bias=False)
    13. # Up-projection: d_latent -> n_heads * d_head (one set each for K and V)
    14. self.W_uk = nn.Linear(d_latent, n_heads * d_head, bias=False)
    15. self.W_uv = nn.Linear(d_latent, n_heads * d_head, bias=False)
    16. # Register as cache buffers
    17. self.register_buffer("cache_latent", None) # compressed latent vectors
    18. self.register_buffer("cache_k_rope", None) # position-related part
    19. def forward(self, h):
    20. c = self.W_dkv(h) # [T, d_latent] one copy cached per token
    21. kr = self.W_kr(h) # [T, d_latent_rope]
    22. k = self.W_uk(c).view(-1, self.n_heads, self.d_head)
    23. v = self.W_uv(c).view(-1, self.n_heads, self.d_head)
    24. # In a real implementation, K is concatenated with kr before RoPE, and history cache is appended as needed
    25. return k, v, c, kr

    Comparing cache sizes. With d_latent = 512, d_latent_rope = 64, and FP16 precision:

    • Cache per token per layer is about (512 + 64) × 2 ≈ 1.1KB;
    • Estimating roughly 60 layers, cache per token is about 66KB;
    • A 64K context uses about 4GB, and 128K about 8GB.

    By contrast, with ordinary multi-head attention—even after compressing KV heads down to 8 (i.e., GQA)—a 64K context still requires about 15GB. The gap is more than 3×; compared with uncompressed multi-head attention, the gap can reach an order of magnitude.

    Joint K/V compression. In traditional designs, K and V are stored independently. Joint compression lets the two share information in a low-dimensional subspace, reducing information loss at high compression ratios. The cost is an extra up-projection at decode time—trading “a bit more compute” for “much less storage.”

    3.3 The Trade-off on the Number of Attention Heads

    In ordinary multi-head attention, KV cache size is proportional to the number of KV heads, so reducing the head count directly lowers the cache. With low-rank compression, however, cache size is largely independent of the head count, and the meaning of the head count shifts:

    • Benefit: At decode time, each head must up-project the latent vector, so halving the head count roughly halves that compute, improving decode throughput accordingly.
    • Cost: In compute-intensive phases such as prefill, reducing heads weakens multi-head attention’s ability to model in parallel across different subspaces.

    Thus, whether to reduce the head count is essentially a trade-off between “decode efficiency” and “modeling capacity.” The mainstream approach is to moderately reduce heads while keeping per-head dimension fixed (e.g., from 128 down to 64), and to compensate for the loss in representational capacity using the backbone capacity that a large hidden dimension provides.

    3.4 Comparing Attention Schemes

    Scheme Cached content Cache size Main advantage Main limitation
    MHA (multi-head attention) Independent K, V per head Largest Strong expressiveness, simple to implement High memory cost in long contexts
    MQA / GQA A few KV heads shared Medium Significant cache reduction, mature and stable Sharing limits multi-head expressiveness
    Low-rank compressed attention Low-dim latent vector + positional subspace Smallest High compression ratio, cost-efficient for long context Requires up-projection at decode, more complex to implement

    4. Sparsity Scaling Law: Why Experts Should Be “Many and Narrow”

    The gains from sparsification are not unlimited. The prevailing conclusion from Scaling Law research is that, under a given compute budget, increasing the number of experts usually brings sizable performance gains, but the returns gradually diminish and are constrained by two factors:

    1. Routing quality: The more experts there are, the harder it is for the gate to learn a clear division of labor, and the more likely it is to fall into “routing noise.”
    2. Communication overhead: When experts are distributed across different devices, All-to-All traffic grows with the number of experts and devices. Beyond a certain scale, the compute gains from adding experts are eaten up by communication cost.

    Taking a representative model as an example, increasing the expert count from 256 to 384 is a 50% increase; the per-expert intermediate dimension is 2048, only about 0.29× the model’s hidden dimension. This “many and narrow” design is essentially trading combinatorial complexity for parameter efficiency: only when each expert is small enough can more experts be stacked under the same compute; only when the combinatorial space is large enough can the model learn fine-grained specialization.

    This also answers a common question: why does model performance still improve as total parameters grow, even though activated parameters are only a fraction of the total? Because MoE performance comes from “the sum of knowledge across all experts,” not the portion used in a single forward pass. Routing only decides “which part to use”; knowledge capacity is always determined by the total parameters.

    5. Training Strategy: Muon-Style Optimizers and Large-Scale Parallelism

    5.1 Optimizer Evolution: From Element-Wise Adaptive to Matrix Orthogonalization

    Mainstream training has long relied on element-wise adaptive optimizers such as AdamW: each maintains first- and second-order momentum for every parameter, with updates determined by each parameter’s own gradient statistics. Its limitation is that this update ignores the internal correlations within parameter matrices—the update direction of each dimension is independent of the others.

    In recent years, a class of optimizers represented by Muon (the name comes from MomentUm Orthogonalized by Newton-Schulz) has drawn attention. Its core idea is to first orthogonalize the momentum matrix (in practice often approximated via Newton-Schulz iterations) before using it to update the parameters. You can think of it as a “matrix-level” optimizer: it treats a matrix as a whole, keeps singular values balanced during updates, and avoids some directions being updated too aggressively while others barely move. Related research indicates that, in large-scale training, it can reach comparable or even better convergence with fewer training steps.

    Two engineering considerations:

    • Extra overhead: Newton-Schulz iterations require multiple matrix multiplications, adding compute overhead; this is typically controlled via low-precision implementation.
    • Mixed usage: A common practice is to use it to update hidden-layer weight matrices while still using adaptive optimizers for token embeddings, the output layer, and gating—balancing stability and efficiency.
    1. def newton_schulz_orthogonalize(G, steps=5):
    2. """
    3. Approximately orthogonalize the gradient/momentum matrix G (core Muon step, illustrative)
    4. After convergence, G's singular values approach 1, equivalent to the orthogonal part of the polar decomposition
    5. """
    6. X = G.bfloat16()
    7. X = X / (X.norm() + 1e-7) # Normalize by Frobenius norm
    8. a, b, c = (3.4445, -4.7750, 2.0315) # Coefficients of the fifth-order approximating polynomial
    9. for _ in range(steps):
    10. A = X @ X.T # A = X X^T
    11. B = b * A + c * (A @ A) # Polynomial acceleration
    12. X = a * X + B @ X # Update X
    13. return X.float()
    14. # Illustration: momentum buffer -> orthogonalize -> update parameters
    15. def muon_step(param, buf, lr, weight_decay):
    16. buf.mul_(0.95).add_(param.grad, alpha=1.0) # Momentum accumulation
    17. update = newton_schulz_orthogonalize(buf)
    18. param.data.mul_(1 - lr * weight_decay) # Decoupled weight decay
    19. param.data.add_(update, alpha=-lr) # Parameter update

    5.2 Pretraining Data Scale and Composition

    Taking a pretraining scale of 15.5T tokens as an example, its magnitude sits at the upper-middle range of the pretraining scale common to mainstream dense models (a few trillion to just over ten trillion tokens). More important than the total is the composition and quality:

    • The ratio of general corpora to code, math, and other reasoning-oriented corpora directly determines the model’s reasoning ability;
    • The coverage of multilingual corpora affects the model’s international capabilities;
    • Data deduplication and quality filtering significantly reduce memorization and overfitting;
    • The later stage of training is often paired with annealing and upsampling of high-quality data to further improve results within a limited number of steps.

    5.3 Large-Scale Parallelism Strategies

    Training a trillion-parameter MoE requires combining multiple parallelism strategies:

    • Data parallelism (DP): Each device holds a full model replica, with gradient synchronization.
    • Tensor parallelism (TP): Split single-layer weight matrices across cards, suited to scenarios with heavy per-layer compute.
    • Pipeline parallelism (PP): Partition by layer across devices to form a pipeline execution.
    • Expert parallelism (EP): Different experts are placed on different devices; tokens are dispatched to the device holding the target expert via All-to-All.
    • Sequence / context parallelism: During long-context training, partition along the sequence dimension.

    The parallelism challenge for MoE concentrates on All-to-All communication: each MoE layer forward requires two global dispatches (one before entering experts, one to aggregate on return), and communication volume is proportional to the number of tokens in the global batch, the hidden dimension, and the number of activated experts. Common engineering mitigations include:

    • Overlapping communication with computation;
    • Grouped routing limits, restricting tokens to being routed only to experts on the same node;
    • Pipeline bubble optimization.

    6. Horizontal Comparison: How to Read Model Architecture from Technical Reports

    An interesting phenomenon has emerged in the industry: multiple models adopt highly similar architectures in their technical reports (e.g., MoE + low-rank compressed attention) and provide detailed parameter comparisons against models in the same category. This does not mean architectural convergence is without value; on the contrary, it shows that sparsity plus attention compression has become a de facto industry standard, and the focus of competition is shifting from “architectural prototype innovation” toward “training efficiency, data quality, and engineering execution.”

    For enterprises, the following dimensions deserve attention when reading technical reports:

    Dimension What to focus on Impact on business
    Total / activated parameters The ratio of the two determines memory and compute needs Affects deployment cost and throughput
    Attention mechanism Whether KV compression and long context are supported Affects cost-efficiency in long-document scenarios
    Number and granularity of experts Total experts, per-expert dimension Affects performance on fine-grained tasks
    Training data scale Total tokens and composition Affects domain adaptability
    Optimizer and parallelism strategy Training efficiency and stability Affects ability to iterate continuously
    Context length Maximum supported tokens Affects RAG and long-document analysis scenarios

    7. Enterprise Selection and Deployment Recommendations

    1. Define the business scenario first: Long-document understanding, multi-turn dialogue, code generation, and structured extraction place very different demands on model capability, so establish an evaluation set before selecting a model.
    2. Do cost estimation upfront: Memory footprint is roughly “total parameters × bytes per precision unit + KV cache,” which lets you estimate the concurrency a single machine can handle; compute needs relate to activated parameters and input/output length.
    3. Pay attention to long-context cost-efficiency: Models using low-rank compressed attention have clearly lower cache overhead in contexts above 64K, making them well suited to long-document analysis workloads.
    4. Choose the deployment form flexibly: You can quickly validate results via a public cloud API; for data-sensitive workloads, use dedicated instances or private deployment; you can also use container platforms for elastic scaling, combined with logging services and monitoring/alerting to ensure service stability. For example, Baidu AI Cloud’s Qianfan platform provides access to and fine-tuning for a range of mainstream large models, and supports meeting enterprise needs through multiple options such as public cloud APIs and private deployment.
    5. Build an evaluation loop: Combine general-capability evaluations with business metrics (accuracy, latency, cost) rather than looking at leaderboards alone.

    8. Summary

    The core advantage of the MoE architecture lies in “decoupling”: separating parameter scale from per-token compute lets a model hold far greater knowledge capacity under a controllable compute budget. Low-rank compressed attention solves the memory and bandwidth bottleneck in long contexts; load balancing and lossless routing ensure the training stability of trillion-parameter models; and optimizer innovation together with large-scale parallelism continues to improve training efficiency.

    The parameter comparisons in technical reports also show that sparsity plus attention compression is becoming the de facto standard across leading large models, and the focus of competition has shifted toward training efficiency, data quality, and engineering execution. For enterprises, understanding these designs is the key to making more rational decisions in selection, deployment, and cost estimation.