A Deep Dive into Large-Scale Sparse Mixture-of-Experts Models

    This technical analysis examines how large-scale sparse Mixture-of-Experts models separate total parameters from inference cost, and how data quality, low-cache attention, multi-token prediction, sparse attention, asynchronous agent reinforcement learning, and custom infrastructure work together. It also outlines practical criteria for evaluating model capability, efficiency, stability, and production readiness.

    1. Model Scale: Parameter Count Does Not Equal Inference Cost

    One of the most notable aspects of the report is its use of a large-scale Mixture-of-Experts architecture. The model has approximately 744 billion total parameters, but only about 40 billion parameters are activated during a single inference pass. Internally, it contains hundreds of expert modules distributed across dozens of layers.

    To understand these figures, it is important to distinguish between total parameters and active parameters.

    Total parameters represent the size of all learnable weights in the model. They directly affect model capacity, storage requirements, and training resource demands. Active parameters refer to the parameters that actually participate when an input passes through the network. This number is more closely related to inference-time compute.

    A Mixture-of-Experts model generally consists of shared components and multiple experts. For each input, a router calculates expert scores based on the hidden state and selects only a small number of experts to process the current token. This allows the model to maintain a large overall capacity without executing every parameter for every inference request.

    The process can be simplified as follows:

    1. Input token
    2. ↓
    3. Shared representation layer
    4. ↓
    5. Router calculates expert scores
    6. ↓
    7. Select a small number of experts
    8. ↓
    9. Experts compute in parallel
    10. ↓
    11. Weighted output aggregation

    This design provides two direct benefits:

    1. The model can maintain a larger knowledge capacity.
    2. Per-inference compute does not increase in direct proportion to the total parameter count.

    However, it also increases system complexity. Expert workloads may become unbalanced: some experts can be called too frequently while others remain underused. If routing is unstable, it can lead to training divergence, communication congestion, and lower hardware utilization.

    As a result, adding more experts is not always better. The key factors are routing stability, expert load balancing, and communication efficiency.

    2. As Data Scales, Training Focus Shifts to Data Quality

    According to the report, the model was trained on approximately 28.5 trillion tokens. Most of the data was used for foundational pretraining, with an additional portion used for mid-training.

    As the volume of training data continues to grow, data quantity is no longer the only decisive factor. Data quality, duplication, knowledge coverage, temporal distribution, and task composition all affect the final capabilities of a model.

    Large-scale model training typically requires multiple stages of data processing.

    1. Data Cleaning

    This stage removes corrupted text, meaningless templates, duplicate content, malformed formats, and obviously machine-generated text. The goal is to prevent low-quality data from consuming the training budget.

    2. Deduplication

    Repeated data can cause a model to over memorise certain expressions and reduce overall data diversity. Common approaches include document-level deduplication, paragraph-level deduplication, and approximate semantic deduplication.

    3. Quality Scoring

    Samples can be ranked according to language fluency, information density, structural completeness, and subject-matter depth. High-quality knowledge text can improve factual capabilities, while code, mathematics, and logic data can strengthen reasoning.

    4. Stage-Based Data Mixtures

    The pretraining stage generally prioritizes broad knowledge coverage. During mid-training, the proportion of specialized data, long-form text, code, and complex reasoning examples can be increased.

    Data expansion must therefore be accompanied by stronger data governance. Simply adding more tokens may produce diminishing returns and can even introduce additional noise.

    3. Architectural Change One: Co-Designing Low-Rank Attention and the Optimizer

    The model uses a type of multi-latent attention mechanism designed to reduce cache overhead. Its core idea is to compress keys and values into smaller latent representations instead of storing complete key-value states. This reduces memory consumption during long-context inference.

    In conventional attention, the key-value cache continues to grow as the sequence becomes longer. For long-context tasks, it often becomes one of the primary sources of memory pressure during inference. Low-rank compression can reduce the cache dimension, allowing longer inputs to run on the same hardware.

    However, architecture cannot be considered separately from the optimizer. The report notes that when a certain optimizer was applied directly to complete projection matrices, the low-cache attention structure performed poorly and was even less effective than a more conventional grouped-attention design.

    To address this issue, the research team introduced an optimization method similar to block-wise orthogonalization. Instead of treating an entire projection matrix as a single unit, the method optimizes attention heads or local structures separately. The goal is to preserve a more stable geometric structure across attention heads and prevent certain directions from being excessively amplified or compressed during training.

    This example demonstrates the strong coupling between model architecture, optimization algorithms, and parameter-update methods. A structure that is theoretically more memory-efficient may not translate into better practical capability without a compatible optimization strategy.

    Large-model development must therefore consider more than network architecture. It must also address:

    • Parameter initialization;
    • Gradient clipping;
    • Matrix-update rules in the optimizer;
    • Learning rates for different modules;
    • Numerical stability throughout training.

    4. Architectural Change Two: Multi-Token Prediction

    Traditional language models usually predict the next token based on the preceding context. Multi-token prediction attempts to predict several future positions during a single forward pass.

    This approach has two potential benefits. First, it provides richer training supervision. The model must learn not only immediate local continuation but also longer-range structure. Second, in some inference scenarios, multi-token prediction can support candidate generation and parallel verification, potentially improving generation efficiency.

    However, multi-token prediction is not simply a matter of concatenating multiple labels. The model must handle dependencies between prediction positions and prevent different objectives from interfering with one another. During training, the loss weights assigned to different positions also require careful design. Otherwise, the model may over-optimize nearby targets while neglecting more distant predictions.

    From a capability perspective, multi-token prediction may improve:

    • Continuous generation of code structures;
    • Organization of long sentences and paragraphs;
    • Intermediate-state modeling in multi-step reasoning;
    • Prediction of future text structure.

    Its benefits, however, depend on data quality, context length, and the design of the training objective. More prediction targets do not automatically produce a stronger model.

    5. Architectural Change Three: Sparse Attention for Lower Long-Context Cost

    The computational complexity of full attention generally grows quadratically with sequence length. As the context expands from several thousand tokens to tens of thousands or more, compute and memory pressure can increase rapidly.

    Sparse attention addresses this by allowing each token to access only a subset of important positions instead of the entire sequence. Common strategies include local windows, periodic connections, global tokens, block sparsity, and content-dependent sparsity.

    The report focuses on dynamic sparse attention, which attempts to select the most relevant context positions according to the input. Compared with a fixed window, this approach is more flexible. Compared with full attention, it can significantly reduce unnecessary computation.

    The main challenge of sparse attention is not simply reducing computation. It is ensuring that the information being ignored is not the information that determines the final answer. A practical system must balance efficiency with information coverage. It typically needs to:

    1. Maintain dense attention for local semantic relationships;
    2. Preserve global routes for titles, definitions, conclusions, and other important positions;
    3. Create jump connections for cross-paragraph references;
    4. Train the model to operate under sparse access patterns;
    5. Evaluate long-text information retrieval and recall.

    The report mentions that approximately 20 billion tokens of relevant training data achieved results close to those obtained with several hundred billion tokens. This finding is better understood as an observation about training efficiency rather than a universal rule. It suggests that, for specific tasks and with an appropriate architecture, structured training data can be more effective than blindly increasing data volume.

    6. A Fully Asynchronous Reinforcement Learning Framework for Agents

    In addition to pretraining and architecture, the report highlights reinforcement learning for AI agents.

    Traditional reinforcement learning workflows are often synchronous: the model generates trajectories, the environment returns results, the trainer collects a batch of data, and the model is updated before the next round begins. This approach is straightforward to implement, but complex tasks can be slowed down by the longest-running samples.

    Asynchronous training separates the system into independent components:

    1. Policy service: generates actions or answers
    2. Environment service: executes tasks and returns results
    3. Evaluation service: scores and filters trajectories
    4. Training service: computes gradients and updates parameters
    5. Data service: manages caching, sampling, and version control

    These components can run in parallel. The generator does not need to wait for the trainer to complete an entire update cycle, and environments can continue executing tasks. This improves hardware utilization and reduces waiting caused by different task durations.

    Asynchronous operation also introduces new challenges.

    Parameter Lag

    The model version used to generate a sample may no longer be the latest version. The trainer must record policy versions and control sample lifetimes so that excessively stale data does not affect updates.

    Data Distribution Shift

    As the model changes, earlier data may differ substantially from the current policy distribution. Importance correction, sample filtering, or prioritized sampling may be required to reduce this shift.

    Training Stability

    When multiple training jobs update concurrently, gradient statistics may become inconsistent. The system needs clear mechanisms for parameter synchronization, checkpointing, and failure recovery.

    Resource Scheduling

    Inference, environment execution, and training often have different hardware requirements. Resource isolation and elastic scheduling directly affect overall throughput.

    The core of asynchronous reinforcement learning is not merely running multiple processes at the same time. It is building a traceable, reversible, and evaluable data loop.

    7. The Value of Custom Training Infrastructure

    Once models reach tens or hundreds of billions of parameters, general-purpose training frameworks may not cover all operational requirements. Research teams often need dedicated infrastructure for communication, parallelism, data management, and fault recovery.

    A complete training platform must address at least the following:

    • Combining data, tensor, and pipeline parallelism;
    • Communication and load balancing between experts;
    • Memory management for long-sequence training;
    • Sample version control for asynchronous tasks;
    • Rapid recovery after training interruptions;
    • Real-time monitoring of gradients, loss, and throughput;
    • Automatic isolation of abnormal samples and failed nodes.

    In large-scale distributed training, hardware failures, network instability, and communication timeouts are normal operating conditions. System design should not assume that every node will remain healthy. Instead, it should provide checkpointing, retries, task migration, and partial recovery.

    8. How to Evaluate Models of This Class

    Large models should not be evaluated solely by a single leaderboard or their total parameter count. A more useful evaluation framework should cover four dimensions.

    Capability

    This includes knowledge question answering, mathematical reasoning, code generation, long-context understanding, and complex instruction following.

    Efficiency

    Relevant metrics include inference latency, throughput, memory usage, context-extension cost, and compute required per task.

    Stability

    Evaluation should consider consistency across repeated runs, completion rates for long tasks, robustness to abnormal inputs, and reliability in tool use.

    Usability

    Practical factors include deployment complexity, resource requirements, documentation quality, model licensing, and the maturity of the surrounding ecosystem.

    Strong performance on public benchmarks does not mean that a model is the best choice for every production environment. Organizations must also consider task requirements, data security, deployment costs, and operational capabilities.

    The report highlights several broader trends.

    First, model scaling is shifting from a competition over parameter counts to a competition over effective computation. Total parameters remain important, but activation ratios, communication efficiency, and useful computation per token deserve greater attention.

    Second, architectural innovation must be paired with suitable training methods. Sparse experts, low-cache attention, and multi-token prediction can produce stable gains only when the optimizer, data mixture, and training infrastructure are designed together.

    Third, the key to long-context processing is not simply expanding the context window indefinitely. It is improving the efficiency of information access. Sparse attention, retrieval mechanisms, and hierarchical memory may together form the foundation of future long-text systems.

    Fourth, agent capabilities increasingly depend on reinforcement learning and interaction with environments. Expanding pretraining data alone cannot fully solve multi-step planning, tool use, and task execution.

    Fifth, infrastructure has become part of model capability. Efficient data pipelines, asynchronous training frameworks, fault-recovery mechanisms, and observable systems can directly affect development cycles and final model quality.

    Overall, the report presents more than an isolated model technique. It describes an integrated approach spanning data, architecture, optimization, reinforcement learning, and systems engineering. For developers, the most valuable lesson is not any single parameter value. It is how to build an end-to-end large-model development system around compute efficiency, training stability, and real-world task performance.