Large-Scale Sparse Mixture-of-Experts Models: Architecture and Training Techniques

    This article examines how large-scale Mixture-of-Experts models separate total parameter capacity from per-token computation. It explains expert routing, training-data stages, Multi-Head Latent Attention, separated matrix optimization, multi-token prediction, sparse attention, capability evaluation, and the engineering challenges of deploying models with hundreds of billions of parameters. The focus is on how architecture, optimization, data strategy, and distributed systems work together to control cost while expanding model capacity.

    1. Separating Total Capacity from Compute Cost

    The growth of large language models is no longer defined simply by using every parameter during every inference step. Mixture-of-Experts (MoE) models use routing mechanisms to assign different inputs to different expert networks. This allows the model to maintain a very large total parameter capacity while activating only a portion of those parameters for a particular computation.

    According to the source technical information, the model uses an MoE architecture with approximately 744 billion total parameters and activates about 40 billion parameters per inference step. It contains 256 expert modules and has 80 layers.

    These concepts should be distinguished:

    • Total parameter count represents the model’s overall capacity to store knowledge, patterns, and task capabilities.
    • Activated parameter count represents the parameters that actually participate in processing a specific token.
    • Number of experts determines how many relatively independent capability specializations the model can learn.
    • Number of layers determines how many transformations information passes through and also affects training difficulty and inference latency.

    The key value of this design is that it decouples model capacity from per-step computation. If a model contains 744 billion parameters but activates only about 40 billion for each token, its per-token compute is not equivalent to that of a dense 744-billion-parameter model.

    However, sparse activation does not mean that the overall system is inexpensive. All expert parameters still need to be stored. Distributed training and inference also introduce additional costs, including expert dispatch, cross-device communication, and load balancing.

    Compared with the previous generation, the earlier model had approximately 355 billion total parameters and activated about 32 billion parameters. The newer version increases total capacity to 744 billion parameters and activated capacity to 40 billion. Total capacity therefore grows much faster than activated capacity. This suggests that the design priority is not simply to make every computation step heavier, but to use more experts to support broader knowledge coverage and more diverse task specialization.

    2. Expanding Training Data: Staging Matters as Much as Scale

    The source material states that the model was trained on approximately 28.5 trillion tokens. About 27 trillion tokens were used for pretraining, while approximately 1.5 trillion tokens were used for mid-training. The previous-generation model used roughly 23 trillion tokens.

    Pretraining generally establishes language understanding, knowledge representation, code modeling, and basic reasoning capabilities. Broader data coverage exposes the model to more language structures, programming patterns, and domain information.

    However, simply increasing the number of tokens does not guarantee a proportional improvement in capability. Data quality, duplication rates, contamination control, mixture balancing, and training order are equally important.

    Mid-training can be viewed as a capability-shaping stage. It often places greater emphasis on high-quality corpora, complex reasoning examples, coding tasks, long-context samples, and data with specific formats. Its purpose is not to teach language from the beginning, but to improve the model’s ability to handle complex tasks on top of its existing foundation.

    The 28.5-trillion-token figure is therefore only a scale indicator. More important questions include how the data is divided into stages, how the stages are transitioned, how different data categories are balanced, and whether the training process avoids excessive overfitting to a particular task type.

    3. MoE Routing: Why Only a Small Number of Parameters Are Activated

    A typical MoE model consists of shared components, multiple expert networks, and a router. When a token enters a layer, the router evaluates its representation and selects several experts. The selected experts perform feed-forward computation, after which their outputs are combined and passed to the next layer.

    This mechanism provides three major benefits.

    First, different experts can learn different types of patterns. Some may become better at natural-language expression, while others may specialize in code, mathematics, or long-text structure. In practice, this specialization is not necessarily assigned manually. It can emerge gradually through competition during routing and training.

    Second, the amount of computation per token can be controlled. As long as each token is sent to only a small number of experts, the activated parameter count does not increase linearly with the total parameter count.

    Third, the model’s overall capacity can continue to grow. Increasing the number of experts expands the model’s representation space without requiring every token to use all experts.

    Sparse routing also introduces risks. If too many tokens are sent to a small number of experts, the workload becomes unbalanced. Some devices may become bottlenecks while others remain underutilized. If routing is spread too broadly, communication costs may increase.

    As a result, expert load balancing, routing stability, and communication scheduling are all critical engineering concerns in large-scale training.

    4. Multi-Head Latent Attention: Reducing Long-Context Cache Pressure

    The model uses Multi-Head Latent Attention (MLA). Its basic approach is to avoid storing complete, high-dimensional keys and values for every attention head. Instead, relevant information is first compressed into a more compact latent representation and then recovered or mapped back into the attention computation space when needed.

    During autoregressive generation, the keys and values of historical tokens are typically stored in a cache so that later tokens can reuse them. As the context becomes longer, this cache consumes more GPU memory. By compressing the cached representation, MLA can reduce storage pressure during long-context inference and lower the memory bandwidth required to read the cache.

    This approach also creates new optimization challenges. The compressed representation must satisfy two requirements at the same time:

    1. It must be compact enough to reduce cache costs.
    2. It must preserve the fine-grained information required by multi-head attention.

    If the optimizer’s update strategy is not well matched to the projection matrices, the resulting model may perform worse than a structurally simpler grouped-query attention design.

    5. Separated Matrix Optimization for Attention Training

    The technical material describes an optimization issue encountered when a matrix-orthogonalization-based method was used with Multi-Head Latent Attention. The resulting model performed worse than a simpler grouped-query attention approach. The training team subsequently proposed a separated matrix optimization strategy.

    The key change was to stop treating the entire projection matrix as one object for orthogonalization. Instead, the matrix was split according to attention heads, and each part was processed separately.

    Orthogonalization can be understood as applying mathematical constraints that make weight directions more balanced and reduce redundancy or interference between different representation directions.

    Splitting the matrix by attention head gives each head a more independent optimization space. This may provide several benefits:

    • Reducing interference between updates to different attention heads;
    • Helping each head learn different types of representations more stably;
    • Improving the match between compressed representations and the multi-head structure;
    • Reducing the sensitivity of specialized attention architectures to the optimizer.

    This does not mean that matrix orthogonalization is always superior to conventional optimization methods. Its effectiveness depends on the model architecture, parameter partitioning strategy, learning-rate policy, and training stage.

    For large-scale models, the optimizer and network architecture often need to be designed together rather than evaluated independently.

    6. Multi-Token Prediction: Beyond One-Step-Ahead Modeling

    Traditional autoregressive language models generally predict the next token based on the preceding context. Multi-token prediction instead requires the model, during training, to predict information at several future positions simultaneously. This encourages the model to learn more than the immediate next-token relationship and to capture longer local dependencies.

    Potential benefits include:

    • Improving the model’s overall understanding of short-term semantic structure;
    • Improving the prediction of continuous structures in code, such as brackets, indentation, and function calls;
    • Helping the model establish representations for subsequent text earlier;
    • Providing more information for candidate generation and verification during inference.

    Multi-token prediction does not mean that inference will automatically generate several completely independent tokens at once. It is primarily a training-objective design. Whether it can produce higher throughput in deployment depends on the decoding strategy, verification mechanism, and hardware execution model.

    It should therefore not be interpreted simply as a guaranteed improvement in generation speed.

    7. Sparse Attention for Controlling Long-Context Complexity

    Standard full attention requires the current token to consider a broad range of historical content. As sequence length increases, both computation and memory-access pressure become more significant.

    Sparse attention reduces this burden by allowing the model to focus on selected positions rather than processing every historical token equally. Common approaches include local windows, block-based connections, globally important positions, and content-dependent dynamic selection.

    The goal is to preserve important dependencies while reducing unnecessary attention computation.

    The main challenge is that discarded information may contain the exact clue needed to produce the correct answer. Systems therefore usually need a mechanism to compensate for limited local visibility. Examples include retaining a small number of global channels or gradually aggregating long-range information across layers.

    8. How to Interpret Model Capability Evaluations

    The technical material indicates that the model achieved strong results on certain public intelligence evaluations and arena-style tests, placing it among the leading open models in those evaluations.

    These results should be interpreted carefully.

    First, no single benchmark can fully represent a model’s capabilities. Different evaluations emphasize knowledge question answering, code generation, mathematical reasoning, long-context understanding, instruction following, or safety.

    Second, arena results are often closer to user preference than to actual performance across every enterprise task.

    Finally, deployment cost, response latency, context length, stability, and maintainability also determine a model’s practical engineering value.

    For developers and enterprise users, a more reliable evaluation process is to build an internal task set that includes representative, boundary, and abnormal inputs from real workloads. The evaluation should track multiple metrics, including:

    • Accuracy;
    • Hallucination rate;
    • Code pass rate;
    • Average latency;
    • Peak GPU memory usage;
    • Cost per request.

    9. Four Engineering Issues for Production Deployment

    9.1 GPU Memory and Communication

    When the total parameter count is extremely large, distributed storage and parallel execution are usually required. Moving tokens between experts can become a bottleneck, so the communication topology and batching strategy must be designed together with the routing mechanism.

    9.2 Load Balancing

    Uneven expert utilization can overload some devices and reduce overall throughput. Routing constraints, capacity limits, and dynamic scheduling may be required to reduce hot experts and distribute work more evenly.

    9.3 Long-Context Stability

    Cache compression and sparse attention can reduce resource consumption, but they may also cause the model to lose long-range information. Before deployment, systems should be tested carefully on cross-section references, long code files, and multi-turn conversations.

    9.4 Evaluation and Monitoring

    After deployment, monitoring should not focus only on average response time. Teams should also track latency across different request lengths, expert utilization, GPU memory consumption, error rates, and changes in output quality.

    Conclusion

    The technical direction represented by these large-scale MoE models reflects an important trend in modern language-model development: increase total capacity to expand knowledge and task coverage, then use sparse activation, cache compression, structured optimization, and sparse attention to control actual computation costs.

    The real value of a model is determined not by parameter count alone. It depends on whether the architecture, training data, optimization methods, systems engineering, and application-level evaluation form a complete and effective loop.