Large-Scale MoE LLM Architecture and Key Training Techniques Explained

    A structured walkthrough of the engineering techniques behind a recently published open-source large model technical report: sparse MoE scaling, Multi-head Latent Attention (MLA), per-head orthogonalization in optimizer design, Multi-Token Prediction, sparse attention, two-stage pretraining and mid-training data mixing, and fully asynchronous agentic reinforcement learning. Each section explains what the technique does, why it matters, and what it means for developers choosing models, optimizing inference, or running secondary training.

    1. Background: Open-Source Models Are Closing in on the Closed-Source Frontier

    Over the past year, the capability gap between open-source LLMs and the leading closed-source models has narrowed steadily. Third-party leaderboards tell the same story: one flagship closed-source model is still widely regarded as the “smartest,” but the distance between it and the best-performing open-source model directly behind it has shrunk noticeably. This is not the result of simply stacking parameters and data — it comes from a coherent body of architectural and training innovations.

    This article uses a recently published open-source model technical report as a sample. Rather than focusing on the product itself, it isolates the generalizable engineering techniques that carry real value and breaks them down one by one. For developers working on model selection, inference optimization, or secondary training, these points are directly actionable.

    2. MoE Sparse Architecture: Rethinking “Total Parameters” vs. “Activated Parameters”

    The model described in the report uses the now-mainstream Mixture of Experts (MoE) architecture. The core idea: the model contains a large number of “expert” subnetworks, but at each inference step a router activates only a small subset of them while the rest stay silent.

    The sample configuration is roughly:

    • Total parameters: ~744B
    • Activated parameters per inference: ~40B
    • Experts: 256
    • Layers: 80

    Compared with the previous generation (~355B total, ~32B activated), total parameter count has roughly doubled while activated parameters rose only modestly. The key mental model here:

    Total parameters mainly determine the capability ceiling; activated parameters mainly determine inference cost.

    MoE has become the default choice at scale precisely because it decouples “model knowledge capacity” from “compute per forward pass.” You can grow capacity by adding experts while keeping activated parameters relatively controlled — gaining inference speeds close to a much smaller dense model while approaching the expressiveness of a large dense one.

    MoE also introduces new engineering challenges:

    1. Load balancing: routers tend to funnel tokens to a handful of experts, overloading some while leaving others idle. In practice, a load balancing loss is typically added to constrain the routing distribution.
    2. Memory footprint: although only some experts are active per token, all expert parameters must remain resident in GPU memory, raising capacity requirements.
    3. Communication cost: in distributed training, experts are spread across devices, so tokens must be routed between devices, incurring all-to-all communication overhead.

    3. The Attention Dilemma: Memory Efficiency vs. Training Stability

    At the architecture level, this generation makes three substantial changes relative to the last: attention mechanism optimization, multi-token prediction, and sparse attention. Start with attention.

    3.1 Multi-head Latent Attention (MLA): Compressing the KV Cache

    The model uses Multi-head Latent Attention (MLA), whose central goal is to reduce the GPU memory footprint of the KV cache in long-context settings.

    Standard attention must cache the keys and values of historical tokens during inference. As the context grows, the KV cache grows linearly and becomes the primary memory bottleneck for long-context inference. MLA stores the KV in a low-dimensional latent space and projects it back up when needed, approximating the full attention effect with a much smaller cache representation. It preserves the expressiveness of multi-head attention while substantially reducing memory and bandwidth pressure during long-context inference.

    An illustrative comparison (pseudocode, for explanation only, not a real implementation):

    1. # Standard attention: cache the full K and V
    2. # cache_shape = [batch, seq_len, num_heads, head_dim] * 2 (K and V)
    3. # MLA: cache the compressed latent vector c_kv
    4. # c_kv = W_down @ [K; V] # down-projection: compress into a low-dim latent space
    5. # cache_shape = [batch, seq_len, latent_dim] # cache only latent_dim
    6. # Project back up at use time
    7. # [K; V] ≈ W_up @ c_kv # up-projection: approximate reconstruction

    3.2 Orthogonalized Optimizers and the Per-Head Solution

    During training, the team hit a very typical engineering problem: pairing a matrix-orthogonalization-based optimizer (the class commonly referred to as orthogonalized optimizers) with MLA caused the model to underperform simpler, more mature designs such as GQA-8.

    The root cause is worth spelling out. Matrix-orthogonalization-based optimizers orthogonalize a parameter matrix as a whole, aiming to make weight updates more uniform across directions and thereby improve training stability and convergence. In MLA, however, the projection matrices are responsible for the “compress–reconstruct” operation. Orthogonalizing an entire projection matrix as one block destroys the originally independent structural semantics of the individual attention heads, forcibly coupling their expressiveness and hurting quality.

    The team’s fix, described as per-head orthogonalization (Muon Split): instead of orthogonalizing the whole projection matrix, orthogonalize each attention head separately.

    The engineering lesson is clear:

    When a general-purpose optimizer conflicts with a specific architectural structure, don’t reflexively swap out the attention mechanism. Instead, check whether the optimizer’s coupling granularity matches the parameter structure. Refining orthogonalization from “the whole matrix” to “each attention head” retains the stability benefits while avoiding damage to structural semantics.

    4. Multi-Token Prediction: A Training Objective That Improves Sample Efficiency

    The second architectural change is Multi-Token Prediction (MTP).

    Conventional autoregressive training predicts only the next token. Multi-token prediction has the model predict several future tokens during training rather than just the immediate next one. It delivers value in two ways:

    1. Better training sample efficiency: a single forward pass yields learning signal at multiple positions, making the training objective denser and data utilization more efficient.
    2. A path to inference acceleration: the multi-head prediction structure introduced during training can be reused for speculative decoding, letting the model draft several tokens and then verify them.

    Note that multi-token prediction is a training objective design. Its core benefits show up in training and inference efficiency, not as a direct increase in the capability ceiling.

    5. Sparse Attention: Controlling Long-Context Inference Cost

    The third change is a sparse attention mechanism. Where MLA addresses the storage density of the KV cache, sparse attention addresses the compute density of attention itself.

    In standard full attention, every token attends to all historical tokens, and compute grows quadratically with sequence length. Sparse attention lets each token attend to only a subset of historical tokens rather than all of them, using carefully designed sparsity patterns to cut long-sequence compute with almost no loss in quality.

    One data point from the report stands out and is discussed repeatedly: sparse attention allowed the model to match the quality of a dense approach trained on ~943.7B tokens using roughly 20B tokens. The magnitude of that gap makes the point plainly: in long-context training, sparsification not only saves compute but also sharply improves token efficiency — which matters enormously for models whose training costs are already high.

    6. Data Mixture: Dividing Labor Between Pretraining and Mid-Training

    On the data side, this generation uses about 28.5T tokens in total: roughly 27T in the pretraining stage and about 1.5T in mid-training.

    This two-stage pattern — large-scale pretraining followed by targeted mid-training — is becoming an industry consensus:

    • Pretraining: dominated by massive, general, deduplicated corpora, aimed at building broad world knowledge and language ability.
    • Mid-training: a further targeted pass on top of pretraining using high-quality data weighted toward particular domains or capabilities (code, math, reasoning, long context), aimed at “refining” general capability into specific capability.

    Compared with blending large amounts of low-quality data into pretraining, this division of labor makes the data mixture clearer and more controllable, and easier to iterate on experimentally.

    7. Post-Training: Fully Asynchronous Agentic Reinforcement Learning

    In post-training, the report’s highlight is a fully asynchronous agentic reinforcement learning training framework.

    Understanding its value starts with the pain points of Agent RL. Conventional RL training is a serial “generate → score → update” pipeline: the model produces trajectories (rollouts), a reward model or environment scores them, and then policy parameters are updated. In agentic settings, trajectories are typically multi-step, long-chain, and involve tool calls or interaction with an external environment, so generating a single high-quality trajectory takes far longer than an ordinary dialogue turn. A serial pipeline leaves GPUs idle at scale.

    The core idea of full asynchrony is to decouple and parallelize trajectory generation from policy updates: one pool of compute continuously generates trajectories and posts them to a queue, while another continuously pulls from the queue to perform training updates, with neither side waiting on the other. This substantially raises training throughput and reduces idle time on expensive compute.

    The engineering challenge is data freshness (the off-policy problem): trajectories used for training may have been produced by a policy several steps old, introducing policy lag. In practice, this typically requires importance sampling corrections, limits on policy version drift, or weighting of stale data — striking a balance between throughput and training stability.

    8. The Engineering Foundation: An RL Training Framework

    Underpinning that asynchronous Agent RL is a purpose-built reinforcement learning training infrastructure. The point worth emphasizing: in the LLM era, the training framework matters as much as the training algorithm.

    A mature RL training framework generally needs to cover:

    • Distributed rollout scheduling: dispatching generation work efficiently across an inference cluster;
    • Training/inference parameter synchronization: pushing the latest policy parameters to the generation side efficiently;
    • Sample buffering and scheduling: managing the trajectory queue and controlling the degree of off-policyness;
    • Fault tolerance and elasticity: handling node failures, memory overflows, and similar anomalies during long runs.

    Many teams find their algorithmic innovations ultimately constrained by the maturity of this engineering foundation. It is also one reason open-source models can close the gap with closed-source leaders so quickly: open and mature infrastructure raises the floor for everyone building on top of it.

    9. Evaluation: Reading Capability Beyond the Leaderboard

    Technical reports usually include results across several third-party benchmarks. In this sample, the model ranks first among open-source models on a composite intelligence index, and places in the same tier as leading closed-source models on text and code arenas.

    For developers, though, a level-headed reading of leaderboards matters just as much:

    1. Rankings are a necessary, not sufficient, condition: a high placement means the model is strong in general capability, not that it is optimal for your specific workload.
    2. Look at activated parameters, not just total parameters: total parameters determine capacity; activated parameters determine your actual inference cost and latency.
    3. Validate with a private evaluation set: the most reliable approach is to build an eval set close to your own business, then compare candidate models head-to-head on real performance and cost.

    10. Practical Takeaways for Developers

    Pulling the report together, a few conclusions have direct value for practitioners:

    • Architecture choice is about trade-offs: MoE trades total parameters for capability and uses activated parameters to control cost, so evaluate capacity and inference overhead together.
    • Long context requires two separate calculations: MLA saves on KV cache storage, sparse attention saves on attention compute; only together do they form a complete long-context inference cost optimization.
    • Training objectives are efficiency levers: designs like multi-token prediction improve sample efficiency and lay groundwork for inference acceleration.
    • Post-training bottlenecks are engineering problems: the value of Agent RL depends heavily on the maturity of the asynchronous training framework — idle compute often constrains outcomes more than the algorithm itself.
    • Run private evals beyond public leaderboards: treat public rankings as a reference and your own business evaluation set as the basis for decisions.

    The gap between open-source and closed-source frontier models is converging, and what ultimately determines whether you can put a model to good use is never a single metric. It is whether you understand the full trade-off system connecting parameters, attention, data, and the training framework.