Large-Scale Sparse Mixture-of-Experts Models: Architecture and Training Techniques

    This article explains how large sparse Mixture-of-Experts models expand total capacity while limiting the parameters activated for each token. It covers expert routing, load balancing, communication overhead, compressed attention caches, structure-aware optimization, multi-token prediction, sparse attention, and practical evaluation criteria for capability, efficiency, stability, and deployment.

    1. Model Design Through the Lens of Parameter Scale

    In recent years, competition among large language models has shifted from simply increasing parameter counts to achieving stronger capabilities under limited computing resources. Sparse Mixture-of-Experts (MoE) models have become an important approach in this transition.

    According to publicly available technical information, one large open-source model uses a Mixture-of-Experts architecture with approximately 744 billion total parameters and about 40 billion activated parameters per inference step. The model contains 256 expert modules and 80 network layers. Two concepts should be distinguished:

    • Total parameters: The combined parameters of all experts, shared modules, attention layers, vocabulary-related components, and other model components.
    • Activated parameters: The parameters actually selected and used by the routing mechanism when processing a given input token.

    This means the model can provide nearly trillion-scale knowledge capacity without sending every token through the entire network. Compared with dense models, sparse models can expand capacity while keeping the computation required for each inference step under control.

    Compared with the previous generation, the newer model increased total parameters from approximately 355 billion to 744 billion, activated parameters from about 32 billion to 40 billion, and training data from roughly 23 trillion to 28.5 trillion tokens. About 27 trillion tokens were used for base pretraining, while approximately 1.5 trillion tokens were used for mid-training.

    These changes show that model upgrades do not depend solely on parameter expansion. They advance in three directions at the same time:

    1. Increasing model capacity to improve knowledge coverage and complex-task modeling;
    2. Moderately increasing the activated parameter count to strengthen per-token representation;
    3. Expanding high-quality training data to improve language understanding, code generation, and reasoning stability.

    Parameter count alone does not determine model quality. Data quality, training objectives, optimizers, routing stability, context length, post-training methods, and inference strategies also have significant effects. Large models should therefore be evaluated based on capability, computational cost, and deployment complexity together.

    2. Understanding Sparse Mixture-of-Experts Architecture

    A Mixture-of-Experts model generally consists of shared components, multiple expert networks, and a router. An input token is first evaluated by the router and then assigned to a small number of experts for processing.

    During training, different experts may gradually develop different capability preferences. For example:

    • Some experts may become stronger at natural-language generation;
    • Some may specialize in code and structured text;
    • Some may perform better on mathematical reasoning;
    • Some may be more suitable for long-form text or specific language contexts.

    These functions are not manually hard-coded. Expert specialization usually emerges through the interaction between routing decisions and gradient updates.

    A simplified routing process looks like this:

    1. Input token
    2. │
    3. ▼
    4. Router computes expert scores
    5. │
    6. ▼
    7. Select a small number of top-scoring experts
    8. │
    9. ▼
    10. Experts process the token independently
    11. │
    12. ▼
    13. Combine outputs using routing weights
    14. │
    15. ▼
    16. Output hidden state

    The central benefit of sparsity is that total model capacity can continue to grow without causing the computation for each token to increase at the same rate. However, this architecture introduces several engineering challenges.

    1. Uneven Routing Load

    If a large number of tokens are routed to only a few experts, some experts become overloaded while others remain underused. This reduces hardware utilization and may destabilize training.

    Training therefore usually includes load-balancing mechanisms to keep the number of tokens assigned to each expert within a reasonable range. The routing system must also handle capacity limits, token dropping, cross-device communication, and changes in batch composition.

    2. Communication Costs from Expert Parallelism

    In distributed training, different experts may be placed on different computing devices. After routing, tokens often need to be redistributed across devices, and the results must later be collected. This process can create substantial communication overhead.

    The real efficiency of a sparse model therefore depends not only on theoretical computation, but also on:

    • Network bandwidth between devices;
    • Whether communication can overlap with computation;
    • How evenly experts are distributed;
    • Batch size and sequence length;
    • Whether the parallelism strategy matches the hardware topology.

    3. Attention Mechanisms: Reducing Cache Pressure for Long Contexts

    The model uses a form of multi-latent attention. Its main idea is to compress key and value representations so that inference does not need to store a complete key-value cache for every attention head.

    During autoregressive generation, the model must cache the keys and values of previous tokens. As the context becomes longer, this cache can consume large amounts of GPU memory and may become a primary inference bottleneck. Compressing the cache can provide several benefits:

    • Lower memory usage per sequence;
    • Higher concurrency for long-context workloads;
    • Reduced cache read and write bandwidth pressure;
    • Support for larger batches or longer inputs on the same hardware.

    However, cache compression also increases training and implementation complexity. The compressed representation must preserve enough information for different attention heads. Otherwise, the model may lose accuracy when modeling local details, long-range dependencies, and complex relationships.

    4. Structure-Aware Optimizer Improvements

    During training, the team found that directly applying a particular matrix optimization method to the complete projection matrices in multi-latent attention performed worse than using a simpler grouped-attention design. This suggests that the mathematical assumptions of an optimizer are closely coupled with the structure of the network.

    To address this issue, the model introduced an optimization strategy that splits the parameters by attention head. A conventional approach might treat the entire projection matrix as one object for orthogonalization or updating. The split approach processes the submatrix associated with each attention head separately.

    The two approaches can be summarized as follows:

    1. Global optimization:
    2. One large projection matrix → Apply a unified matrix transformation
    3. Split optimization:
    4. Large projection matrix → Split by attention head → Apply transformations separately

    This change is useful because attention heads generally operate in relatively independent representation subspaces. Processing them separately can reduce interference between subspaces and make the optimization process better aligned with the network structure.

    Orthogonalization should not be understood as simply making parameters “better.” It is more accurately a way to constrain the geometric structure of the parameters. Its effectiveness depends on where it is applied, the learning rate, parameter initialization, training stage, and how it is combined with other optimization strategies.

    5. How Multi-Token Prediction Improves Training Efficiency

    Traditional language models typically predict the next token based on the preceding context. Multi-token prediction extends this objective by asking the model to predict several future positions at the same time, in addition to the primary next-token objective.

    This design has two main benefits.

    First, it provides richer training signals. A training example supervises multiple future positions rather than only one, which may help the model learn dependencies across a longer range.

    Second, multi-token prediction may support faster continuous generation during inference. If future predictions can be verified and accepted, the system may reduce the waiting time associated with generating tokens one by one.

    However, multi-token prediction also introduces risks:

    • Uncertainty generally increases as the prediction distance grows;
    • Different prediction heads may produce conflicting results;
    • Poorly designed objectives may affect convergence on the primary task;
    • The inference system needs additional verification and fallback mechanisms.

    It is therefore not simply a matter of adding several output heads. The training objective, loss weights, and inference process must be designed together.

    6. Sparse Attention: Controlling Complexity for Long Text

    The computational complexity of standard self-attention usually grows quadratically with sequence length. When inputs expand from several thousand tokens to tens of thousands or more, the attention matrix creates significant computation and memory pressure.

    Sparse attention addresses this by avoiding the calculation of every possible token-to-token relationship. Instead, it may use local windows, block-based connections, important-position selection, or dynamic retrieval to compute only a subset of attention links.

    The goal is not merely to remove information, but to focus computation on positions that are more likely to be useful, such as:

    • Local context near the current paragraph;
    • Document titles, key entities, and structural markers;
    • Historical passages related to the current question;
    • Important tokens selected through routing or scoring mechanisms.

    Sparse attention can reduce computation for long-context tasks, but it also creates a risk of missing relevant information. Practical systems therefore need to balance the sparse attention range, global information pathways, and important-token retention strategies.

    7. How to Evaluate Models of This Type Objectively

    For extremely large sparse models, evaluation should not focus only on total parameters or a single benchmark score. A more useful evaluation framework should cover at least the following dimensions.

    1. Capability

    Relevant tasks include general question answering, complex reasoning, code generation, information extraction, long-context understanding, and multilingual processing. Different tasks respond differently to architectural choices, so broad evaluation is more informative than a single benchmark.

    2. Efficiency

    Important metrics include:

    • Per-token generation latency;
    • Time to first token;
    • Output throughput;
    • GPU memory usage;
    • Concurrency;
    • Performance degradation under long-context workloads.

    3. Stability

    Testing should cover different input lengths, batch sizes, and hardware configurations. The evaluation should determine whether outputs remain stable, whether routing becomes significantly skewed, and whether excessive token dropping or expert overload occurs.

    4. Deployment

    Once total parameters reach several hundred billion, full deployment requires substantial storage, memory, and communication resources even when the number of activated parameters per token is relatively low. Organizations must consider model partitioning, quantization, parallel inference, cache management, and failure recovery together.

    8. Engineering Lessons

    The main value of this class of models is not simply that they contain more parameters. They demonstrate the broader evolution of large-model systems engineering.

    1. Model capacity and computational cost can be decoupled
      Sparse expert architectures allow models to provide greater knowledge capacity while controlling the number of parameters activated for each token.

    2. Network architecture and optimization algorithms must be designed together
      When the attention structure changes, the way optimizers process matrices may also need to change.

    3. Long-context optimization cannot rely on a single technique
      Cache compression, sparse attention, positional encoding design, and parallelism strategies need to work together.

    4. Training and inference efficiency are equally important
      Multi-token prediction, expert routing, and cache optimization affect different stages of the system, but together they determine overall cost.

    5. Evaluation should extend from model metrics to system metrics
      A genuinely useful model must not only be capable, but also deployable, stable in operation, and able to maintain reasonable throughput on its target hardware.

    Overall, large-scale sparse Mixture-of-Experts models represent a technical path centered on expanding model capacity, controlling actual computation, and improving long-context processing. Their challenges have expanded beyond individual network design to include the coordination of routing, communication, optimization, caching, and inference systems.

    For developers and organizations, understanding these underlying mechanisms supports more accurate decisions about model selection, compute planning, and production deployment.