A Complete Guide to Large-Scale MoE LLM Architecture and Training

    This article walks through a 40-page technical report on large-scale sparse (Mixture-of-Experts) model training and turns it into a practical playbook for engineers. It covers the parameter-activation decoupling at the core of modern MoE design, three key architectural changes—low-rank compressed attention (MLA) with a head-split orthogonalized optimizer (Muon Split), multi-token prediction, and native sparse attention—and the fully asynchronous agentic reinforcement learning loop used in post-training. For each change it explains the motivation, the engineering trade-offs, and the transferable lessons, then closes with a phased deployment roadmap and a checklist of common pitfalls. Readers will learn how to read benchmark scores critically, where the real cost bottlenecks sit in large-scale training, and how to reproduce these techniques under their own data, hardware, and latency constraints.

    1. Why a 40-Page Report Is Worth Reading Page by Page

    For engineering readers, the value of this report is not in its benchmark scores but in the fact that it reads like a set of construction blueprints. The four technical points that dominate community discussion—trainable native sparse attention, a per-head orthogonalized optimizer, multi-token prediction, and fully asynchronous agentic reinforcement learning—happen to cover the three most expensive stages of large-model training: the attention overhead introduced by sequence length, the match between the optimizer and the parallel structure, and the throughput bottleneck during the rollout phase. Understand these four items, and you have essentially understood the blueprint for training today’s large-scale sparse models.

    One thing to state up front: every number cited below comes from the report’s own disclosures and has not been independently reproduced by a third party. The architectural and engineering conclusions reflect general industry understanding and can be adapted, but the concrete gains must be re-measured under your own data distribution, compute setup, and latency constraints.

    2. The Foundation: Decoupling “Capacity” from “Inference Cost”

    2.1 The parameter-to-activation ratio is central to this generation of design

    The value of a sparse Mixture of Experts (MoE) model lies in a simple idea: total parameters determine knowledge capacity, while activated parameters determine the compute for a single forward pass. Once the two are decoupled, model capacity can grow with the compute budget while the per-token inference cost stays within a controllable range.

    The report discloses the following figures:

    Dimension Previous generation This generation
    Total parameters 355B 744B
    Activated parameters per token 32B 40B
    Experts / Layers — 256 / 80
    Training tokens 23T 28.5T (pre-training 27T + mid-training 1.5T)

    Two observations are worth remembering. First, while total parameters doubled, activated parameters rose by only 25%, which shows the team prioritized capacity over compute. Second, pre-training and mid-training are explicitly metered separately: the former lays down scale at the trillion-token level, while the latter typically handles tasks such as upsampling high-quality data, extending context length, weighting code and reasoning samples, and learning-rate annealing. Treating “mid-training” as a distinct phase, rather than folding it into a single monotonic pre-training loop, makes it easier to run mixing-ratio experiments and to roll back on failure.

    2.2 How to read the evaluation coordinates

    The position the report claims is this: a score of 50 on some third-party composite intelligence index, ranking first among open-source models; and in public human-preference head-to-head evaluations, first place among open-source models in both the text and code tracks, on par overall with the two top overseas models.

    A caution for readers here: a composite index is a weighted aggregation across multiple task sets, and a single score can hide the underlying capability distribution. Differences in areas such as long-context retrieval, structured tool calling, and multi-turn instruction following are nearly invisible inside a 50-point total. A realistic selection process should instead draw the 200–500 most representative samples from your own business, build an internal evaluation subset, and then make a weighted decision that also factors in throughput, time-to-first-token latency, context length, and the cost of on-premises deployment.

    3. Architecture Change One: Low-Rank Compressed Attention with a Per-Head Orthogonalized Optimizer

    3.1 Why the KV cache must be compressed

    In long-context and agent scenarios, the real cost driver is the KV cache, whose memory footprint can be approximated as:

    1. KV_bytes = 2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_elem

    Here the factor of 2 accounts for both K and V. This expression is linear in both seq_len and batch, meaning that a tenfold increase in context length brings a tenfold increase in memory and bandwidth pressure for long-text inference. The idea behind low-rank joint compression attention (commonly known in the industry as MLA) is to first project the key/value into a low-dimensional latent space much smaller than head_dim × heads and cache that compressed representation, restoring the full dimension only when needed, while splitting rotary position embeddings into a separate path to preserve positional information. The result is a significantly smaller cache dimension, making long-text processing more memory-efficient and faster than traditional multi-head schemes. It is orthogonal to sparse MoE: MoE saves FFN compute, while MLA saves attention-side memory and bandwidth.

    3.2 An unexpected finding during training: optimizer-structure mismatch

    During training, the team observed a counterintuitive phenomenon: with a matrix-orthogonalization optimizer (commonly referred to in the industry as the Muon family), the MLA structure could not match a more plain baseline—8-head grouped-query attention (GQA-8). This suggests that such optimizers orthogonalize the update of two-dimensional weight matrices and, by default, treat the entire projection matrix as a single semantically complete update unit; but MLA’s KV projection is itself a composite structure of “reduce to low rank, then restore dimension,” so orthogonalizing the whole block flattens this structural prior and leaves the optimization geometry mismatched with the actual parameter structure.

    3.3 The solution: pushing orthogonalization granularity down to the attention head

    The report’s approach is summarized as Muon Split: instead of orthogonalizing the entire projection matrix, orthogonalize each attention head separately. The pseudocode is:

    1. def newton_schulz_orthogonalize(G, steps=5):
    2. # G: gradient/momentum matrix of shape [out, in]
    3. X = G / (G.norm() + 1e-7)
    4. a, b, c = 3.4445, -4.7750, 2.0315 # stable coefficients for five iterations
    5. for _ in range(steps):
    6. XtX = X @ X.T
    7. X = a * X + b * (XtX @ X) + c * (XtX @ XtX @ X)
    8. return X
    9. def split_orthogonalize(G_qkv, n_heads):
    10. # Key change: split along the head dimension and orthogonalize per head, then reassemble
    11. shards = G_qkv.view(n_heads, -1, G_qkv.shape[-1])
    12. out = [newton_schulz_orthogonalize(h) for h in shards]
    13. return torch.cat(out, dim=0).view_as(G_qkv)

    This change takes little engineering effort but yields a clear payoff, which shows that an optimizer is not a plug-and-play component.

    3.4 Three transferable lessons

    First, the interaction between a new structure and the optimizer must be validated with ablation experiments; you cannot carry over the previous generation’s configuration. Second, the minimal-change principle often beats starting from scratch—adjust the optimizer’s orthogonalization granularity before considering a structural redesign. Third, whenever you introduce a new low-rank decomposition or grouping structure, check whether the gradient scale is balanced across the sub-spaces, and add per-group learning-rate scaling where necessary.

    4. Architecture Change Two: Multi-Token Prediction

    Multi-Token Prediction (MTP) works by attaching one or two lightweight prediction modules after the main output head, so the model predicts the second and third future tokens in the same forward pass, rather than only the immediately adjacent next token.

    There are several implementation points. The prediction modules typically reuse the backbone’s hidden states and the shared input embeddings and output vocabulary projection, adding just one Transformer block in between to keep the extra overhead controllable. During training, the auxiliary loss is given a small weight coefficient so it does not interfere with the main objective. At inference time, these modules can be discarded to return to standard autoregressive decoding, leaving the deployment shape unchanged.

    The benefits show up in two places. On the training side, the supervision signal per position grows from 1 to n, raising signal density under the same token budget—a particular advantage for tasks that need long-range planning and code completion. On the inference side, these modules can naturally serve as draft heads for self-speculative decoding, folding the verification of multiple tokens into a single forward pass to reduce latency. One caveat: the acceptance rate of speculative decoding depends on how closely the draft head’s distribution matches the main model’s. If the auxiliary weight is set too small during training, the actual speedup will fall well short, so this must be measured on your own workload rather than copied from the paper’s numbers.

    5. Architecture Change Three: Native Sparse Attention and Its Data Efficiency

    Attention computation is quadratic in sequence length, which is the fundamental bottleneck for long contexts and long chain-of-thought inference. There are two routes to sparsification. One is to sparsify only at inference time—using sliding windows, block sparsity, or retrieval-based selection to prune the attention range. The other is to sparsify from the start of training, so the model learns within a sparse topology and the architecture and weights are natively adapted to it.

    The report takes the second route and states that the data-efficiency gain is substantial: roughly 20B tokens of training is enough to match the dense-attention baseline trained on about 943.7B tokens—a gap of nearly 47×. The engineering implication of such a result is that sparse attention does more than save compute; it may actually change the optimization path itself, letting the model converge earlier to useful inductive biases.

    The conceptual skeleton can be understood through the following pseudocode:

    1. def sparse_attention(q, k, v, block=128, top_k_blocks=16):
    2. # 1) Aggregate representations per block (e.g., block mean/max), far cheaper than per-token scoring
    3. k_blocks = k.view(-1, block, k.shape[-1])
    4. block_scores = (q @ k_blocks.transpose(-1, -2)).max(dim=-1).values
    5. # 2) Force-keep the current and neighboring blocks for locality
    6. block_scores[..., -1] = float("inf")
    7. # 3) Each query head selects top-k blocks and builds a block-level mask
    8. idx = block_scores.topk(top_k_blocks, dim=-1).indices
    9. mask = build_block_mask(idx, seq_len, block)
    10. # 4) Compute exact attention only within the mask-permitted region
    11. return masked_softmax_attention(q, k, v, mask)

    What is genuinely hard in practice is the engineering, not the formula. Block size must trade off selection accuracy against memory-access efficiency. Block-level top-k creates uneven compute across heads and batches, and load imbalance directly lowers whole-device utilization, so it needs to be paired with dynamic load balancing or a capacity factor. The sparse pattern and the KV cache layout must be designed together, or cache hits will degrade. Beyond that, you must consider compatibility with sequence parallelism and context-parallel sharding, plus correctness verification of custom operators—numerical alignment against a dense implementation at small sizes is the bare minimum.

    6. Post-Training: Fully Asynchronous Agentic Reinforcement Learning

    6.1 Three bottlenecks of the synchronous on-policy paradigm

    The traditional synchronous RL loop is “batch-generate trajectories → compute rewards → update policy → generate again.” Applied to agent scenarios, this loop has three fatal problems. First, severe long-tail effects: within the same batch, some trajectories finish in a few dozen steps while others require hundreds of tool calls, and the synchronization barrier forces the whole batch’s compute to wait for the slowest sample, leaving many devices idle. Second, high variance in environment latency: the response times of external tools, code sandboxes, and retrieval services are unpredictable and often become the bottleneck for the entire pipeline. Third, sparse and delayed rewards: for multi-turn tasks, the reward often comes only at the end, so sample utilization is low.

    6.2 Key design points of fully asynchronous

    The core of a fully asynchronous design is to completely decouple rollout from training, turning them into a producer–consumer pair:

    1. # Producer: rollout worker continuously pulls the latest weights and pushes trajectories into an experience stream
    2. while running:
    3. weights = weight_store.latest(ack_policy=True) # accept that weights may be slightly stale
    4. traj = rollout(agent, env, weights) # multi-turn tool calls, variable length
    5. stream.push(traj) # do not wait for other workers
    6. # Consumer: training worker pulls in batches from the stream and never stalls on long-tail samples
    7. while budget_left:
    8. batch = stream.pull(min_size=B) # take once enough is collected, no same-generation requirement
    9. adv = compute_advantage(batch) # group normalization, etc.
    10. ratio = pi_theta.logp(batch) - batch.behavior_logp
    11. loss = -min(ratio * adv,
    12. clip(ratio, 1-e, 1+e) * adv).mean() # clipping suppresses off-policy bias
    13. optimizer.step()
    14. weight_store.publish(theta) # hot weight update

    There are four key engineering points. First, staleness control: asynchronous designs inevitably introduce off-policy data, so training stability must be preserved through importance-sampling ratios combined with clipping, plus a cap on the maximum number of generations by which the weight version may lag. Second, handling partial trajectories: whether an interrupted, incomplete trajectory is discarded, truncated, or resumed directly determines sample efficiency, and it typically requires trajectory-level resumption and state snapshots. Third, servicifying rewards and environments: turn tool calls, sandbox execution, and retrieval into independently horizontally scalable services and use queues to smooth out peaks, so environment latency no longer blocks training. Fourth, observability: you must simultaneously monitor throughput, average staleness, clip-trigger rate, reward distribution, and entropy—otherwise it is very hard to localize a degradation when asynchronous training goes wrong.

    7. Infrastructure: Why These Models Often Need a Custom RL Framework

    This generation’s report specifically mentions a self-developed reinforcement learning infrastructure, and the reason is not mysterious. General-purpose open-source frameworks are already mature enough for supervised fine-tuning, but RL—especially agentic RL—places additional demands on the system shape: the training engine and the inference engine must be co-resident and exchange weights frequently; the generation side needs high-throughput continuous batching while the training side needs a different parallelization strategy; hot weight synchronization must reach minute-level or faster; reward computation over millions of trajectories needs to scale independently; and long-running jobs spanning weeks require checkpoint-resume and fault isolation.

    The corresponding general pattern for landing this on the cloud is clear: use container orchestration to host heterogeneous rollout and training task pools and elastically scale them by queue; use object storage to hold checkpoints and dataset versions; use a message queue to carry the decoupled pipeline of trajectories and rewards; and use logging services and monitoring/alerting to build dashboards and threshold alarms for throughput, staleness, and failure rate. For enterprises that want to reproduce this kind of training and inference pipeline in their own environment, building resource orchestration and observability on general-purpose cloud infrastructure—such as the container platform, object storage, message queue, logging service, and monitoring/alerting capabilities offered by platforms like Baidu AI Cloud—is a pragmatic path. It should be emphasized that specific capabilities and quotas are subject to the official documentation and commercial terms; this article makes no service commitment for any individual case.

    8. Deployment Roadmap and Common Pitfalls

    To reproduce a scheme like this from scratch, it is advisable to advance in four phases rather than changing everything to the final state at once:

    1. Establish a baseline first. Use a mature dense or MoE architecture to get the data–training–evaluation loop working end to end, and lock down the internal evaluation set and throughput benchmarks. Every subsequent change must be measured against this baseline.
    2. Then run single-item ablations. Prioritize validating the memory savings and accuracy loss of low-rank compressed attention, along with the additional training signal from multi-token prediction. Give each change its own on/off switch for easy rollback.
    3. Jointly tune the optimizer and structure. When introducing a matrix-orthogonalization optimizer, be sure to split the orthogonalization granularity per attention head and check the gradient scale group by group—the payoff from this step is often underestimated.
    4. Finally, bring in sparse attention and asynchronous RL. These two changes place the highest demands on custom operators, parallel sharding, and experience-stream consistency, so they should be advanced only after the first three items are stable.

    Common pitfalls checklist: numerical divergence between the sparse and dense implementations on long sequences (small-size alignment and end-to-end regression are mandatory); load imbalance from block-level top-k eating up half the theoretical speedup; reward collapse caused by out-of-control weight staleness in asynchronous RL; degradation of the main task from an overly large auxiliary-loss weight in multi-token prediction; and evaluation-set contamination—when building an internal evaluation set, reserve a portion of continuously refreshed private samples as an honesty check.

    9. Conclusion

    The most valuable thing about this report is not the “50-point” number but the way it lays out a complete pipeline: sparse MoE to decouple capacity from cost, low-rank compressed attention to lower long-context memory, multi-token prediction to raise signal density, native sparse attention during training to trade for data efficiency, and finally fully asynchronous reinforcement learning with a custom infrastructure to keep post-training throughput high. There is no magic among these five changes—each is an engineering choice made on a clearly identified bottleneck, one that is ablatable and reversible. For a development team, the truly replicable capability is this methodology of “locate the bottleneck → make the minimal change → ablate rigorously → evaluate independently.” As for parameter scale, corpus mixing ratios, and compute shape, those must be re-solved within your own constraints.