Sparse MoE LLM Architecture Evolution and Training Optimization Explained

    This article analyzes the architecture and training optimizations behind a sparse mixture-of-experts (MoE) large language model. It explains how sparsity decouples model capacity from compute, then examines three key changes: low-rank compressed attention with per-head orthogonalized optimization, multi-token prediction, and sparse attention. The article also covers memory estimation, deployment considerations, and evaluation practices for selecting and deploying large language models.

    Introduction: From the Parameter Race to Sparsity

    Open-source models are iterating rapidly. Recently, several open-source models have reached the same tier as leading closed-source models on public evaluations. One roughly 40-page technical report stands out: in its third-party composite intelligence index—a score formed by weighting dozens of math, coding, reasoning, and instruction-following benchmarks—the model scored 50, becoming the first open-source model to reach that mark. In large crowdsourced human-preference arenas, it also ranks near the top among open-source models in both text and coding categories, placing it alongside some of the strongest closed-source models.

    Scores are only the outcome. What developers should study is the technical path behind them. The report focuses on a sparse mixture-of-experts (MoE) model with 744 billion total parameters and 40 billion parameters activated per token. It discloses three major changes compared with the previous generation:

    1. Low-rank compressed attention paired with a modified matrix-orthogonalization optimizer;
    2. Multi-token prediction (MTP);
    3. Sparse attention (DSA).

    This article first builds the background needed to understand these changes, then breaks down each design motive, technical principle, and engineering cost. It concludes with practical recommendations for memory estimation, deployment, and evaluation.

    1. Foundation: How Sparse MoE Decouples Capacity from Compute

    1.1 Core Design: Trading Parameter Capacity for Compute Efficiency

    In a traditional dense model, every token passes through all parameters. Model capacity and per-token compute are roughly proportional. MoE splits the feed-forward layers into many parallel expert networks and uses a gating router to select only a few experts for each token. This allows the model to store knowledge in a large parameter count while each token incurs only a small fraction of the computation.

    First, compare key configurations across the two generations:

    Dimension Previous generation Current generation
    Total parameters 355 billion 744 billion
    Activated parameters 32 billion 40 billion
    Experts / layers — 256 / 80
    Training tokens 230T 285T (270T pre-training + 15T mid-training)
    Sparsity (activated / total) about 9.0% about 5.4%
    Capacity / activation ratio about 11.1x about 18.6x

    Three signals stand out from this table:

    • Parameters roughly double, but per-token compute increases by only 25%: Activated parameters rise from 32 billion to 40 billion, an increase of 8 billion. Even if a substantial part of this increase lands in dense computation such as attention, the growth rate is still much lower than the doubling of total parameters.
    • Sparsity decreases significantly: The proportion of parameters actually used per token drops from about 9% to about 5%. This means the model can gain much more parameter capacity at inference costs closer to the previous generation.
    • Data scale expands in parallel: Training tokens increase from 230T to 285T, about 24% growth, roughly matching the 25% growth in activated parameters. This suggests parameter expansion follows a path of “more sparsity plus more data,” rather than simply increasing compute.

    1.2 Capacity Ceiling: Sparsity Is Not a Free Lunch

    To understand the benefit of sparsity, recall an important lesson from the dense-model era: under a fixed compute budget, model parameters and training tokens should scale together. If parameters grow too quickly while data does not keep up, the model may quickly memorize training data, and generalization can degrade.

    MoE changes this balance. Because total parameters are not used in every token’s computation, far more parameters can be accommodated under the same compute budget. Knowledge capacity and compute cost are no longer measured by the same ruler. As a result, “how many parameters are activated and how many tokens were trained” becomes the core metric for MoE inference cost, while total parameters mainly correspond to memory and deployment thresholds.

    The costs are also clear:

    • Expert distribution within a batch can be severely imbalanced, with popular experts overloaded and cold experts idle.
    • Inconsistent routing across devices can amplify communication overhead in distributed training.
    • With more and finer-grained experts, the model is more likely to degrade into a situation where a small number of experts dominate.

    Therefore, mainstream approaches combine load-balancing loss, capacity factor, expert parallelism (EP), and improved routing algorithms. When evaluating sparse models, it is not enough to look only at total parameters and activated parameters. One must also examine whether expert granularity, routing strategy, and balancing mechanisms are designed together.

    1.3 Capability Verification: Objective Metrics and Subjective Preferences

    To judge the actual level of a model, three complementary types of evidence are usually needed:

    1. Objective evaluations: Automatically scored benchmarks for math, coding, scientific reasoning, and similar tasks measure reasoning capability. A third-party composite intelligence index can also help observe overall standing.
    2. Subjective preferences: Crowdsourced arenas rely on large-scale anonymous pairwise comparisons and statistical models to produce rankings. They are often closer to real-world usage experience.
    3. Domain-specific capabilities: Separate leaderboards for coding completion, function generation, and similar scenarios reflect competitiveness in vertical domains.

    For this report, the model ranks first among open-source models on the third-party composite intelligence index with a score of 50, and also enters the open-source top tier in both the text arena and the coding arena. The two types of evidence support each other, suggesting that its advantage is broad rather than limited to gaming a single question type. Of course, every benchmark carries some risk of overfitting. Production selection should still be validated against business-specific evaluation sets.

    2. Change 1: Low-Rank Attention × Per-Head Orthogonalized Optimization

    2.1 Starting Point: KV Cache Is the Memory Bottleneck for Long Contexts

    During large-model inference, generating each new token requires reviewing all prior context. To avoid repeated computation, systems cache the Keys and Values of historical tokens (KV Cache). Under standard self-attention, the KV cache size for a single sequence is:

    1. KV_bytes = 2 (K and V) × number of layers × number of KV heads × head dimension × sequence length × bytes per element

    With 80 layers, 128-dimensional heads, and BF16 precision: if the number of KV heads is 64 (standard multi-head attention, one group per head), the cache is about 2.5 MB per 1K tokens. If the number of KV heads is 8 (grouped-query attention, GQA-8), it drops to about 320 KB, an 8x reduction. For a 128K-token context and a single request, the former requires about 320 GB, while the latter requires about 40 GB. KV cache can easily become the main bottleneck limiting throughput and concurrency.

    2.2 Three Generations of Trade-offs: From MHA to GQA, Then to Low-Rank Compression

    To relieve this pressure, attention mechanisms have evolved through several stages:

    • MHA (Multi-Head Attention): Each query head has an independent set of K/V. It has the strongest expressiveness but the largest cache overhead.
    • MQA (Multi-Query Attention): All query heads share one set of K/V. Cache size is greatly reduced, but modeling capability drops noticeably.
    • GQA (Grouped-Query Attention): Query heads are divided into groups that share K/V within each group. It is the most common compromise between expressiveness and efficiency.
    • Low-rank compressed attention: Used in the current model and similar to techniques in some other open-source models. Instead of caching K/V for every head directly, it first uses a down-projection matrix to compress the input into a lower-dimensional latent vector and stores that latent vector in the cache. When attention is needed, an up-projection matrix reconstructs the per-head Keys and Values. Caching only the latent vector significantly reduces memory usage.

    An often-overlooked detail is that the up-projection matrix can be “absorbed” into adjacent weight matrices through matrix fusion during inference, meaning there is no need to reconstruct the full K/V. This allows the memory savings from compression to carry through to inference, rather than remaining only an intermediate trick during training.

    In addition, this type of attention often decouples positional encoding from K/V and applies it separately to queries and Keys. This keeps the position of the up-projection matrix consistent with matrix fusion.

    2.3 Unexpected Training Issue: Optimizer and Forward Structure Mismatch

    The team encountered a problem when training low-rank attention with a matrix-orthogonalization optimizer such as Muon, which approximately orthogonalizes parameter update matrices through Newton–Schulz iterations: no matter how learning rates and other hyperparameters were tuned, results consistently lagged behind the simpler GQA-8 baseline.

    The reason lies in different implicit assumptions about weight matrices:

    • Orthogonalization optimizer assumption: Each matrix to be updated is treated as a homogeneous whole, and updates should be distributed as evenly as possible across all directions of that matrix.
    • Low-rank attention structure: Each attention head has its own down-projection, up-projection, and bias, and the feature structures learned by different heads can vary greatly.
    • Conflict: When the entire matrix is orthogonalized as a single block, updates for each head are pulled toward the same direction. Some heads end up with reduced effective rank, which appears as insufficient fitting.

    This is a typical “optimizer–architecture coupling” problem: a change that works in the forward pass may alter the statistical distribution of gradients in the backward pass and weaken the optimizer’s effective range.

    2.4 Solution: Muon Split, Per-Head Orthogonalization

    The solution is called “Muon Split”: split the projection matrix by attention head, orthogonalize each head independently, then concatenate them back into the full matrix. Illustrative code:

    1. import torch
    2. def muon_split_update(W, head_dim, lr, momentum=0.95, ns_steps=5):
    3. """
    4. Split by attention head, orthogonalize each head independently, then concatenate back.
    5. W: [d_out, d_in]; d_out is divisible by head_dim.
    6. """
    7. buf = torch.zeros_like(W)
    8. reshaped = W.view(-1, head_dim, W.shape[1]) # [num_heads, head_dim, d_in]
    9. out = torch.empty_like(reshaped)
    10. for i in range(reshaped.shape[0]):
    11. G = reshaped[i] # Update matrix for a single head
    12. buf[i].mul_(momentum).add_(G) # Momentum accumulation, maintained per head
    13. X = buf[i] / (buf[i].norm() + 1e-7) # Normalize first for stable iteration
    14. for _ in range(ns_steps):
    15. A = X @ X.transpose(-1, -2) # Newton–Schulz iteration
    16. B = b_0 + b_1 * A + b_2 * A @ A # Fifth-order polynomial approximation of orthogonalization
    17. X = a_0 * X + a_1 * (B @ X)
    18. out[i] = 0.2 * lr * X * (buf[i].norm() + 1e-7) # Coefficients depend on the specific optimizer implementation
    19. return out.view_as(W)

    This splitting works for at least three reasons:

    1. It respects head heterogeneity: Each head can adjust update magnitude according to its own scale and is not averaged with other heads.
    2. It improves matrix shape: Multi-head projection matrices usually have a number of rows that is a multiple of the number of heads. After splitting, each sub-matrix is closer to square, making orthogonalization more stable and improving conditioning.
    3. It expands the effective range: The part of the gradient that truly carries low-rank attention information is no longer diluted.

    The practical lesson is: optimizers are not plug-and-play components; they need to be designed jointly with the forward structure. When using non-standard attention structures, it is important to keep controlled comparisons against standard baselines such as GQA rather than assuming mature optimizers will work with default settings.

    3. Change 2: Multi-Token Prediction (MTP)

    3.1 Basic Idea

    Standard language models use a causal objective that predicts one token at a time. MTP adds several lightweight prediction modules after the main prediction head to predict the second, third, and further future tokens. The overall structure is shown below:

    1. Input token sequence → Backbone Transformer (80 layers) → Main output head → Predict t+1
    2. Hidden state h_t
    3. │
    4. ▼
    5. MTP Module 1 → Predict t+2
    6. │ (fuse h_t with the representation of t+1)
    7. ▼
    8. MTP Module 2 → Predict t+3

    Each MTP module usually contains only a small number of layers and shares the input embedding layer and output head with the backbone. Therefore, the additional parameters and computation are relatively limited.

    3.2 Training Benefits

    MTP provides at least three benefits at the training level:

    • Denser supervision signals: Each position must predict not only the next token but also farther tokens. This encourages the model to plan ahead rather than only learning local collocations.
    • Improved representation quality: Long-horizon prediction targets are usually harder, so the model must encode longer-term dependencies. Representation learning therefore becomes more structured.
    • Reduced training–inference inconsistency: Training a separate small draft model often creates distribution differences from the main model. MTP modules share representations with the backbone and are trained together, naturally providing stronger consistency.

    3.3 Inference Acceleration: Self-Speculative Decoding

    The most direct use of MTP modules is self-speculative decoding: the main model produces the main token and several draft tokens in one pass, then a single forward pass verifies whether those drafts are correct. Accepted prefixes can be confirmed at once, reducing the number of sequential decoding rounds.

    In practice, the distribution of the number of tokens accepted per inference pass is critical. If draft hit rates are too low, verification overhead may offset the benefits. Therefore, MTP module depth, training weights, and sharing with the backbone need to be tuned for the target scenario, such as dialogue, coding, or math reasoning.

    3.4 Separation of Training and Inference Roles

    It is important to note that MTP’s training objective and inference acceleration can be designed separately. Teams can choose to keep or disable MTP modules depending on whether speculative decoding is enabled and what acceptance-length goals are desired. This suggests that architectural benefits are often not one-time gains but modular items that can be tailored as needed.

    4. Change 3: Sparse Attention (DSA)

    4.1 Motivation: Quadratic Growth of Attention Complexity

    Standard attention compute and memory costs grow quadratically with sequence length. When contexts expand to hundreds of thousands of tokens, even with chunking and IO-aware implementations such as FlashAttention, computation still grows rapidly. The core assumption of DSA is simple: only a small portion of historical content truly needs fine-grained attention.

    4.2 Learnable Indexer + Top-k Selection

    DSA introduces a lightweight “indexer”: it computes a relevance score for each historical position and keeps only the top-k positions for attention computation. Illustrative code:

    1. # Fragment: sparse attention with an indexer
    2. def sparse_attention(q, k_full, v_full, indexer, topk=2048):
    3. """
    4. q: [batch, heads, q_len, d]
    5. k_full: [batch, heads, kv_len, d]
    6. v_full: [batch, heads, kv_len, d]
    7. indexer: lightweight scoring network that outputs a score for each historical position
    8. """
    9. scores = indexer(q, k_full) # [batch, heads, q_len, kv_len]
    10. idx = scores.topk(topk, dim=-1).indices # Keep top-k positions for each query
    11. k_sel, v_sel = gather_by_index(k_full, v_full, idx)
    12. return softmax(q @ k_sel.transpose(-1, -2) @ v_sel)

    Three points are key:

    1. The indexer must be trainable: It is trained jointly with the backbone rather than selected post hoc by heuristic rules.
    2. It composes orthogonally with compressed attention: DSA decides “which historical segment to attend to,” while low-rank compression decides “how much memory that segment occupies.” The two can be combined.
    3. It depends on efficient kernels: Sparse attention after top-k requires fused kernels. Otherwise, the theoretical computation savings can be consumed by gather and indexing overhead.

    4.3 Engineering Costs

    Sparse attention is not free:

    • The indexer itself requires scoring computation, so its dimension must be controlled.
    • Dynamic gather is unfriendly to hardware memory-access patterns and requires specialized kernels.
    • Sparse selection introduces training–inference inconsistency because different batches may select different top-k sets. Training strategies must be designed carefully.

    5. Synergy Among the Three Changes

    These three changes are not isolated additions. They serve the same goal: substantially increasing total parameters while keeping effective per-token compute within an acceptable range.

    • Low-rank compression reduces the storage cost of historical information, making long contexts economically feasible.
    • Sparse attention reduces the compute cost of historical information, preventing attention from growing quadratically with length.
    • Multi-token prediction increases effective supervision density without increasing inference cost and can also serve as a rehearsal mechanism for inference acceleration.

    At the same time, per-head optimizer orthogonalization solves the compatibility problem between low-rank structure and orthogonalized optimization. This architecture is a combination of structural changes, optimizer adaptation, and training-objective expansion. Missing any one part reduces the overall effect.

    6. Memory and Cost Estimation: An Essential Engineering Task

    The claim that sparse models are “cheap” applies only to inference, not deployment. Consider a simplified memory estimate:

    1. # Example: 80 layers, 256 experts, hidden dimension about d, BF16 weights
    2. Weight memory ≈ 744e9 × 2 bytes ≈ 1.46 TB (excluding optimizer states and gradients)

    Adding KV cache, activations, and temporary buffers, a single GPU—even one with 80 GB of memory—obviously cannot host the model. Tensor parallelism and expert parallelism are required. This explains why sparse models attract attention in the open-source community but still have high deployment barriers in practice.

    Practical recommendations:

    1. Estimate memory before discussing throughput: Model memory as “weights + KV cache × concurrency + activations” to avoid discovering bandwidth or memory shortages after launch.
    2. Separate the resource profiles of training and inference: Training is compute-intensive and communication-intensive, while inference is memory-intensive and bandwidth-intensive.
    3. Decide whether to enable MTP acceleration by scenario: Structured-output scenarios such as code completion and agent tool calls often benefit more.
    4. For long-context scenarios, evaluate sparse attention and cache compression together instead of simply adding more memory.

    7. Evaluation Methods and Selection Recommendations

    When reviewing a technical report, pay attention to both “scores” and the “evidence chain”:

    • Composite intelligence index: Useful for horizontal comparison, but weighting and question distribution can affect conclusions.
    • Arena-style human preference: Closer to real experience, but subject to style bias and prompt sensitivity.
    • Vertical leaderboards: More targeted for coding, math, and other scenarios, but also more prone to overfitting.
    • Self-built business evaluation sets: The final decision basis. It is recommended to prepare 100–300 real business samples during model selection, covering typical input distributions and long-tail cases.

    Selection can follow a four-step method:

    1. Define constraints: memory budget, concurrency target, context length, latency SLA.
    2. Validate with a small sample: Evaluate candidate models on the business set rather than relying only on leaderboards.
    3. Run load tests and cost analysis: Calculate memory, throughput, and cost per token together.
    4. Continue regression testing: Re-run evaluations after model iterations to avoid capability regression.

    General deployment recommendations include:

    • Use containerized deployment for inference services to support elastic scaling.
    • Manage model weights and intermediate artifacts with object storage.
    • Cover key metrics such as latency, throughput, and memory usage with logging and monitoring/alerting.
    • Combine tensor parallelism and expert parallelism, and pay attention to all-to-all communication overhead.
    • On general cloud platforms, these common cloud capabilities can help build a complete inference service pipeline and use elastic resources to handle traffic fluctuations.

    Conclusion

    The value of this technical report is not only that an open-source model reached a certain score threshold. More importantly, it presents a reusable engineering methodology for sparse models:

    • Use sparsity to decouple capacity from compute.
    • Use low-rank compression and sparse attention to control the memory and compute cost of long contexts.
    • Use multi-token prediction to increase supervision density and support inference acceleration.
    • Use structural optimizer changes to make forward-architecture innovations truly work during training.

    For developers, understanding the motives, costs, and boundary conditions of these designs is more meaningful than remembering a single score. Next time you face a decision about “whether to switch models” or “whether to adopt long context,” you can return to three questions:

    1. Is my bottleneck memory, bandwidth, or compute?
    2. Does my scenario really require very long context, or would retrieval augmentation be more cost-effective?
    3. Do I have an evaluation set that can quickly falsify a choice?

    Once these three points are clear, technical selection is no longer about following leaderboards but about making rational decisions based on your own constraints.