Large-Scale MoE LLM Training: A Deep Technical Breakdown

    This article dissects the training decisions behind a modern open-source-style Mixture-of-Experts (MoE) recipe, using a generalized sample configuration of roughly 744B total parameters, ~40B activated parameters, 256 experts, 80 layers, and ~28.5T training tokens. It explains how total and activated parameter counts trade capacity against cost, how Multi-head Latent Attention (MLA) shrinks the KV cache, why orthogonalized optimizers such as Muon-style methods can conflict with MLA and how per-head orthogonalization resolves it, what Multi-Token Prediction (MTP) does and does not buy you, when sparse attention actually pays off, and how a fully asynchronous agentic reinforcement learning stack raises GPU utilization in post-training. Engineers who work on model architecture, long-context training, or RL post-training will find practical selection guidance for parameter sizing, optimizer-structure alignment, KV cache and attention cost accounting, and async RL infrastructure requirements.

    1. Why It Is Worth Breaking Down an Open-Source MoE Recipe

    The gap between open-source large models and leading closed-source models has been a recurring industry talking point. For engineering teams, the more valuable signal is not a single benchmark ranking but the reusable training decisions behind these models: how to trade off parameter scale, how to compress long-context memory, how to co-design the optimizer with the network structure, and how to improve post-training throughput.

    This article uses a next-generation open-source-style MoE training recipe as a sample: roughly 744B total parameters, about 40B activated parameters per forward pass, 256 experts, 80 layers of depth, and about 28.5T pretraining tokens. Because the source material does not provide a verifiable model name, report link, evaluation date, or complete leaderboard screenshot, the sample is treated here as a generic engineering case study rather than promotion of a specific product, and no unverifiable ranking claims are repeated.

    Five questions are worth focusing on:

    1. How do total and activated parameter counts affect cost and quality?
    2. How does Multi-head Latent Attention (MLA) reduce the KV cache?
    3. Why can Muon-style orthogonalized optimizers conflict with MLA?
    4. How do Multi-Token Prediction (MTP) and sparse attention change training efficiency?
    5. How does fully asynchronous agentic reinforcement learning improve GPU utilization in post-training?

    2. How to Choose Total and Activated Parameters in an MoE

    The sample uses a Mixture of Experts (MoE) architecture. The core idea is simple: only a subset of experts is activated in each forward pass, so total parameter capacity can grow while per-inference compute stays controlled.

    The key configuration can be summarized as roughly 744B total parameters, about 40B activated parameters per inference, 256 experts, 80 layers, and about 28.5T pretraining tokens, of which roughly 27T is pretraining and about 1.5T is mid-training. Compared with the previous generation, total parameters grew from about 355B to 744B, activated parameters from 32B to 40B, and the data volume from 23T to 28.5T.

    Dimension Previous-gen sample Current sample
    Total parameters ~355B ~744B
    Activated parameters ~32B ~40B
    Number of experts Not disclosed 256
    Network depth Not disclosed 80 layers
    Pretraining data volume ~23T ~28.5T

    The point of MoE design is not simply “bigger.” It is understanding three variables:

    • Total parameters determine knowledge capacity, and also determine weight storage and memory footprint.
    • Activated parameters determine the cost of a single forward pass, directly affecting inference compute and latency.
    • The sparse activation ratio determines the balance point between capacity and efficiency.

    The current sample activates roughly 40B of 744B parameters, an activation ratio of about 5.4%.

    Two engineering cautions apply. If the activation ratio is too low, routing becomes overly sparse and experts are under-trained. If it is too high, the MoE gradually degenerates into a dense model with high compute cost. In addition, inference latency depends not only on activated parameter count but also on memory bandwidth, expert communication, routing overhead, and parallelism strategy. MoE selection therefore cannot be made on parameter count alone; it must weigh quality, latency, throughput, and cost requirements of the workload together.

    3. How MLA Compresses the KV Cache

    The sample uses Multi-head Latent Attention (MLA). MLA is not about adding more attention heads; it reconstructs how keys and values are stored through low-rank compression.

    In standard attention, KV cache size is roughly proportional to sequence length, number of attention heads, and per-head dimension. In long-context scenarios the KV cache quickly becomes the memory bottleneck. A simplified relationship:

    1. Standard KV cache ≈ batch × sequence × heads × head_dim × bytes_per_dtype × 2

    MLA first projects K and V into a lower-dimensional latent space and caches that low-dimensional latent at inference time instead of the full K/V. One point deserves emphasis: MLA normally does not re-materialize the full K/V during inference. Through low-rank projection and weight absorption, the query attends directly against the low-dimensional latent and the necessary positional encoding information. Materializing the full K/V first would cancel out the memory savings.

    MLA also typically caches a decoupled RoPE key separately, since that key does not participate in the low-rank compression. The KV cache therefore consists mainly of two parts: the low-dimensional latent c_kv and the decoupled positional-encoding key.

    Simplified pseudocode:

    1. # Standard attention: cache full K/V for every head
    2. K = x @ W_k # [batch, seq, heads, head_dim]
    3. V = x @ W_v # [batch, seq, heads, head_dim]
    4. # MLA: cache the low-rank latent and the decoupled positional key
    5. c_kv = x @ W_down_kv # [batch, seq, latent_dim]
    6. k_pe = rope(x @ W_k_pe) # [batch, seq, rope_dim]
    7. # At inference, the query attends directly over c_kv and k_pe;
    8. # the full K/V is never materialized.

    A rough illustration of the benefit: suppose a standard model has 128 heads with 128 dimensions each, so a single token requires caching about 128 × 128 = 16,384 KV dimensions. If MLA compresses the latent to 512 dimensions and caches a separate 64-dimensional decoupled RoPE key, per-token cache size drops to about 576 dimensions. This is only an illustration, not a specific model configuration, but it shows the mechanism: as sequence length grows, KV cache cost shifts from “heads × head_dim” to “low-rank latent plus a small number of positional dimensions,” and memory pressure falls sharply.

    4. Why Orthogonalized Optimizers Conflict With MLA, and How to Fix It

    The sample also involves a class of optimizers centered on matrix orthogonalization, such as the Muon-style methods often discussed in the community. These optimizers orthogonalize weight matrices so that updates are more uniform, and they perform well on some dense models and training tasks. Combined with MLA, however, they can underperform simpler alternatives.

    The usual root cause is a mismatch in granularity. The mathematical assumptions behind orthogonalized optimizers typically apply orthogonalization over a whole projection matrix, while MLA organizes its attention logic around individual attention heads as independent units. Once the whole matrix is compressed and rearranged into the latent space, whole-matrix orthogonalization can break the update geometry each head expects, degrading convergence.

    The common team-level fix is per-head orthogonalization: instead of orthogonalizing the full projection matrix at once, split along attention head boundaries and orthogonalize each head’s submatrix separately. If the weight matrix follows the common Muon convention of [out_dim, in_dim], split by output heads:

    1. # Assume W_full has shape [out_dim, in_dim].
    2. # After splitting by output head, orthogonalize each block separately.
    3. update = orthogonalize(W_full)
    4. for h in range(num_heads):
    5. sl = slice(h * head_dim, (h + 1) * head_dim)
    6. update[sl, :] = orthogonalize(W_full[sl, :])

    If the weight layout is instead [in_dim, out_dim], the split must be adjusted to columns. Orthogonalized optimizers are also generally better suited to two-dimensional hidden-layer matrices; for embedding layers, output layers, biases, or one-dimensional parameters, applicability should be evaluated carefully.

    The broader engineering lesson is highly transferable: whenever you introduce a structured component such as MLA, grouped experts, sparse routing, or an MTP head, the optimizer’s inductive bias must align with the granularity at which the structure is organized. Tuning cannot focus on learning rate alone; it must also verify that optimizer granularity, parameter shapes, matrix layout, and attention head partitioning are consistent. Otherwise an optimizer that is theoretically stronger can end up performing worse in practice.

    5. Is Multi-Token Prediction Worth Adding?

    The sample also uses Multi-Token Prediction (MTP). A conventional language model predicts only the next token at each position, while MTP predicts several future tokens in a single forward pass.

    MTP delivers value in two main ways. First, the training signal is denser: each position provides more than a single next-token target, offering richer prediction objectives and improving sample efficiency. Second, the representations are more foresighted: multi-step prediction forces the model to learn longer-range dependencies, which may benefit planning, reasoning, and code generation.

    One clarification is needed on the engineering side: MTP does not inherently reduce training cost. It adds compute and memory overhead on the training side, because it requires extra prediction heads and more complex target construction. Its benefits lie in sample efficiency, representation quality, and training stability, not in directly reducing training resources.

    At inference time, if only the main prediction head is used, MTP need not increase online inference cost. If it is used for speculative decoding, then the dependency between draft heads and the main model, acceptance rate, sequential generation, and system scheduling all have to be considered. Whether to enable MTP should therefore be evaluated against training resources, task type, and inference architecture.

    6. How Sparse Attention Reduces Long-Sequence Cost

    The sample also discusses a dynamically selectable sparse attention mechanism. To avoid confusion with Self-Attention, it can be described as native or dynamic sparse attention, for example Native Sparse Attention (NSA).

    The core observation is that the token interactions that actually matter in the attention matrix are sparse, while a large share of computation goes to low-contribution connections. Sparse attention computes attention only over the small set of tokens most relevant to the current token, bringing long-sequence attention complexity down from roughly quadratic.

    Boundary conditions matter, though. Whether complexity actually drops depends on whether the number of selected tokens K grows with sequence length. If K is fixed, complexity approaches O(S × K) and is close to linear; if K grows linearly with sequence length, it can still approach quadratic. Sparse attention also introduces selection overhead: token importance scoring, index generation, block scheduling, and sparse kernel implementation. If that overhead is too high, end-to-end speedup may fall short of expectations.

    The source text contained a claim along the lines of “about 20B tokens of training matching about 943.7B tokens.” Without model scale, control baseline, data mixture, evaluation methodology, and implementation details, such a number can easily mislead engineering readers, so it is not treated here as a general conclusion. A more reliable judgment is this: in long-context training, if attention computation accounts for a large share of end-to-end cost and selection overhead is manageable, sparse attention can substantially cut compute; if the model’s bottleneck lies in the FFN, communication, or the data pipeline, the benefit will be limited.

    MLA and sparse attention are complementary. MLA mainly addresses the memory pressure caused by an oversized KV cache, while sparse attention mainly addresses the compute pressure caused by expensive attention. Combining the two is what gets closer to an efficient long-context large-model recipe.

    7. How Fully Asynchronous Agentic RL Improves Throughput

    Pretraining sets the floor for a model; post-training determines how well it completes tasks. The sample highlights two post-training priorities: a fully asynchronous agentic reinforcement learning training framework, and in-house reinforcement learning training infrastructure.

    In conventional RL post-training, sampling and training often run serially. During sampling, the model generates trajectories and the trainer waits; during training, the trainer updates parameters and the sampler sits idle. This waste already exists for ordinary dialogue tasks, but for agentic tasks it is worse: the model needs multi-step tool calls and environment interaction, trajectories are longer and sampling is slower, so GPU idling in a serial architecture becomes more severe.

    The fully asynchronous approach decouples sampling from training and pipelines them:

    1. # Serial mode: sampling and training wait on each other
    2. for step in steps:
    3. trajectories = rollout(policy) # sampling phase, trainer idles
    4. policy = train(policy, trajectories)
    5. # Asynchronous mode: the sampler continuously produces experience and the learner continuously consumes it
    6. # Actor: continuously samples and writes to an experience queue
    7. # Learner: continuously consumes experience, updates parameters, syncs weights periodically

    Asynchrony raises throughput substantially because the sampler and the learner can each saturate their own resources. But it introduces new risks.

    • Policy staleness. The sampler’s weights may lag the learner’s. If the synchronization interval is too long, the off-policy gap deepens and stability suffers.
    • Synchronization interval trade-off. Syncing too often adds communication overhead; syncing too rarely weakens policy consistency.
    • Checkpointing and resumption. Large-scale asynchronous RL needs reliable checkpoints, experience queue persistence, and failure recovery.

    Fully asynchronous RL is therefore not just “turning on parallelism.” It requires supporting infrastructure: parameter synchronization mechanisms, experience queue monitoring, sampling quality evaluation, training metric alerting, weight version management, and recovery procedures. Only when those engineering details are in place does asynchrony convert into stable throughput rather than merely a higher theoretical utilization number.

    8. What This Recipe Means for Engineering Choices

    The value of this kind of MoE recipe lies not in any single trick but in system-level coordination. Practical recommendations for training and selection teams:

    1. Treat parameters as layered. Distinguish total parameters, which govern capacity and memory, from activated parameters, which govern inference cost and latency.
    2. Check optimizer-structure alignment. After introducing MLA, grouped experts, or sparse routing, confirm that the granularity at which an orthogonalized optimizer operates matches the structure.
    3. For long context, evaluate both costs. Look at KV cache memory and attention compute time together; MLA and sparse attention are best evaluated in combination.
    4. Use an asynchronous pipeline for post-training. Agentic tasks are expensive to sample, and an asynchronous sampling-training framework improves GPU utilization, but policy staleness and checkpoint/resume must be handled at the same time.
    5. Treat architecture changes as system problems. Real gains come from coordinated decisions across attention, optimizer, data scale, parallelism strategy, and post-training infrastructure.

    9. FAQ: Common Questions About MoE and MLA

    How should total and activated parameters be chosen in an MoE?

    If the workload values knowledge capacity and a higher quality ceiling, total parameters can grow. If it values latency and inference cost, activated parameters and routing efficiency should be controlled instead. A higher activation ratio is not automatically better; the decision has to weigh training stability and expert load balancing.

    How does MLA save KV cache memory?

    MLA uses low-rank compression to turn full K/V into a cached low-dimensional latent, and caches the decoupled positional-encoding key separately. At inference time the full K/V is never materialized; the query attends directly against the low-dimensional representation.

    Why might an orthogonalized optimizer conflict with MLA?

    The common reason is that orthogonalization is applied across a whole projection matrix, whereas MLA is organized around single heads or structured submodules. The usual fix is to split by head and orthogonalize each head separately, after confirming that the weight matrix layout supports that split.

    Does sparse attention always speed things up?

    Not necessarily. It depends on how large a share of end-to-end cost attention represents, whether the number of selected tokens grows with sequence length, whether selection overhead is controllable, and how efficient the sparse kernels are.

    Can asynchronous RL reduce training stability?

    It can. Asynchronous sampling makes training data relatively older with respect to the current policy, so the synchronization interval must be set sensibly, policy divergence must be monitored, checkpoints must be retained, and issues such as experience queue backlog and training failures must be handled.