Understanding Ultra-Scale Sparse Mixture-of-Experts Models
This article examines how ultra-scale sparse Mixture-of-Experts models combine massive parameter capacity with lower per-token computation. It explains the difference between total and active parameters, the impact of training data and architecture changes—including latent attention, multi-token prediction, and sparse attention—and the engineering trade-offs involved in training, inference, evaluation, and deployment.
In recent years, competition among large language models has shifted from simply increasing parameter counts toward jointly optimizing model capacity, computational efficiency, long-context performance, and training stability. A new generation of ultra-scale open models uses sparse Mixture-of-Experts (MoE) architectures to maintain substantial model capacity while activating only a small number of experts for each input, thereby reducing the computational cost of individual inference requests.
This article does not focus on a specific product. Instead, it uses a representative set of designs commonly described in public technical reports to explain why these models can achieve strong capabilities with relatively few active parameters, and how their architectural changes affect training, inference, and deployment.
1. Model Scale: Total Parameters and Active Parameters Are Not the Same
The representative model has approximately 744 billion total parameters and activates around 40 billion parameters for each inference step. It contains 256 experts and 80 layers. Two concepts must be distinguished:
- Total parameters: The combined parameters of all experts, attention modules, embedding layers, and output layers in the model.
- Active parameters: The parameters actually involved in processing a specific token.
In a traditional dense model, most parameters participate in every forward pass. In an MoE architecture, the input first passes through a routing module, which selects a small number of experts for computation. This allows the model to maintain a large knowledge capacity while controlling the computation required for each input.
An analogy is a large team of specialists. Every request does not need the entire team; instead, the system selects the members best suited to the task. A coding question may be routed toward program-analysis experts, a mathematics problem toward reasoning experts, and a long-context task toward experts better suited to contextual modeling.
However, a low active-parameter count does not necessarily mean low deployment cost. The complete model still needs to store all parameters. Distributed inference must also account for expert sharding, cross-device communication, routing balance, and memory capacity. Sparse models optimize the computation path, but they do not eliminate storage and systems-engineering costs.
2. Training Data and Model Scale Must Grow Together
Compared with the previous-generation design, the representative model increases total parameters from approximately 355 billion to 744 billion. Active parameters increase from around 32 billion to 40 billion, while the training corpus grows from about 23 trillion tokens to 28.5 trillion tokens. Of the latter, approximately 27 trillion tokens are used for base pretraining and 1.5 trillion tokens for mid-training.
This change reflects an important trend: scaling a model requires more than adding parameters. Data quality, training stages, and optimization strategies must also improve in parallel.
Base pretraining is primarily used to learn language patterns, knowledge distributions, and general-purpose behaviors. Mid-training is commonly used to improve specific capabilities, including:
- Complex reasoning and multi-step task performance;
- Code, mathematics, and structured-text processing;
- Information retrieval and association across long contexts;
- Output stability and instruction following.
Increasing the amount of data does not guarantee proportional gains. Duplicated content, low-quality web pages, formatting contamination, and factual errors can all reduce training efficiency. Data deduplication, quality filtering, corpus composition, and stage-specific sampling can therefore be as important as parameter count.
3. Architectural Change One: Latent Attention and Head-Wise Optimization
The representative model uses a latent attention mechanism. Its core idea is to compress the representations associated with keys and values, reducing the size of the cache compared with conventional attention.
During long-context inference, the model must retain the keys and values associated with previous tokens. This memory is commonly referred to as the KV cache. As the context grows, the KV cache consumes more memory. Latent attention reduces the cache dimension by sharing or compressing parts of these representations, providing several potential benefits:
- Lower memory pressure during long-context inference;
- Reduced cache read and write bandwidth;
- More concurrent requests per accelerator;
- Less data transferred between devices.
Changes to the attention structure can also affect training optimization. According to the technical report, applying a certain type of orthogonalization optimizer directly to complete projection matrices may perform worse than using a simpler grouped-query attention structure. One possible reason is that applying a uniform transformation to the full matrix can obscure functional differences between attention heads.
The report therefore proposes a head-wise optimization method. Instead of treating the entire projection matrix as a single unit for orthogonalization, the method processes each attention head separately. Intuitively, this allows different heads to preserve clearer subspace structures and reduces interference during optimization.
This design illustrates that optimizers and network architectures cannot be treated as independent components. An optimization method that works well for a dense architecture may need adjustments to its granularity and scope when applied to latent attention or MoE structures.
4. Architectural Change Two: Multi-Token Prediction
Traditional language models generally predict the next token based on the preceding context. Multi-token prediction attempts to predict several subsequent positions during a single training process.
This approach may provide two types of value. First, it offers richer supervision during training. The model learns not only what comes next, but also how a short sequence evolves over multiple positions. Second, in some inference strategies, the model can first generate several candidate tokens and then have the main model verify them, potentially reducing the waiting time associated with token-by-token generation.
Multi-token prediction is not simply a matter of producing multiple tokens in one step. It requires additional prediction heads, loss-function design, and coordination among training objectives. It must also handle the different difficulty levels associated with different prediction distances. The farther into the future the model predicts, the greater the uncertainty generally becomes. Training must therefore balance near-term and long-term targets instead of allowing the model to focus excessively on the easier next-token objective.
From an engineering perspective, the benefits of multi-token prediction depend mainly on three factors:
- The acceptance rate of candidate sequences;
- The additional computation required for verification;
- Whether the inference framework supports efficient batch verification and fallback.
If candidates are frequently rejected, the additional prediction heads may not provide meaningful acceleration. If the acceptance rate is high, the method may reduce the number of decoding rounds and improve responsiveness in interactive applications.
5. Architectural Change Three: Sparse Attention
As context length increases, the computational and memory costs of full attention rise rapidly. Sparse attention addresses this by allowing each position to attend only to a subset of relevant positions instead of the entire context.
One common approach uses a local window, prioritizing nearby content. Another adds a small number of global positions to transmit information across sections. A system may also select important regions dynamically based on content relevance.
The central challenge is reducing computation without losing critical information. If the sparsity pattern is too aggressive, the model may fail to connect definitions, conditions, and conclusions that are far apart. This can lead to omissions in long-document question answering.
Practical designs therefore need to balance:
- Local information modeling;
- The ability to capture cross-section dependencies;
- Attention complexity;
- Hardware parallelism and execution efficiency;
- Stability across different tasks.
For structured long-form content such as code repositories, legal documents, and technical manuals, sparse patterns should also account for relationships among chapters, functions, citations, and headings. A fixed position-based pattern may not adapt well to the semantic structure of real-world documents.
6. How to Interpret Evaluation Results
Strong results on general intelligence benchmarks, text and dialogue evaluations, and coding assessments indicate that this class of model can perform well in knowledge-based question answering, complex instruction understanding, program generation, and multi-turn interaction.
However, benchmark scores should not be treated as a direct measure of production readiness. When selecting a model, organizations should also consider:
- Time to first token and end-to-end generation latency;
- Accuracy degradation at long context lengths;
- Throughput under high concurrency;
- Output stability and format adherence;
- Risks involving hallucinations, unauthorized behavior, and sensitive content;
- Memory and communication resources required for deployment;
- Capability loss after quantization;
- Compatibility with business data and tool-calling workflows.
For MoE models in particular, average throughput can differ substantially from tail latency under extreme workloads. If routing becomes imbalanced, certain devices may become bottlenecks. This can cause requests to queue even when the overall computational load is relatively low.
7. Implications for Engineering Deployment
First, total parameter count should not be the only metric considered. Total parameters determine model capacity, while active parameters more closely reflect the computation required for a single input. Actual deployment cost also depends on caching, communication, and concurrency strategies.
Second, model architecture must be designed together with the hardware system. Expert parallelism, tensor parallelism, pipeline parallelism, and data parallelism need to be combined appropriately. Otherwise, communication overhead can offset the benefits of sparse computation.
Third, long-context capability requires end-to-end validation. Supporting longer inputs at the attention-architecture level does not guarantee that the model can correctly use all available information. Evaluation should include cross-section retrieval, position changes, distracting information, and multi-document combinations.
Fourth, optimizers, training objectives, and network structures require joint tuning. Latent attention, multi-token prediction, and sparse experts all change gradient distributions. Simply copying conventional training configurations is unlikely to produce optimal results.
Overall, the development of ultra-scale sparse models is not about stacking parameters without limit. It is about finding a better balance among model capacity, active computation, cache usage, training stability, and system efficiency. For developers and enterprises, the most valuable insight is not any single metric, but an understanding of how these architectural choices collectively shape model capability, deployment cost, and user experience.
