MoE Foundation Models: Technical Evolution and Engineering Practices
This article examines how modern open-source foundation models are moving beyond brute-force parameter scaling through Mixture-of-Experts architecture and conditional computation. It explains the roles of MLA, Muon, Multi-Token Prediction, and DSA sparse attention, and analyzes how model capacity, activated parameters, training data, and long-context optimization work together. It also summarizes benchmark results and draws practical implications for RAG, AI agents, private deployment, and cost-efficient inference.
From Brute-Force Scaling to Selective Activation
As large language models have evolved, simply increasing parameter counts is no longer the only path to better performance. The industry is increasingly aligned around a broader principle: the true density of intelligence depends on how effectively architectural design can unlock greater model capacity within a limited compute budget.
A new generation of open-source foundation models illustrates this shift with a notable configuration: 744 billion total parameters, while only 40 billion parameters are activated during each inference pass. This large gap between total and active parameters captures the central appeal of the Mixture of Experts, or MoE, architecture.
MoE is fundamentally a form of conditional computation. It divides a large neural network into multiple sub-networks, known as experts. These experts share a similar structure but gradually specialize in different capabilities. A lightweight routing network, or Router, analyzes the features of each input token and dynamically selects the most appropriate group of experts for the forward pass.
This design separates model knowledge capacity from computational cost. A model can contain a very large number of parameters to represent a broader range of world knowledge, while each inference request activates only a small subset. The result is a model that can maintain relatively low inference cost and latency while offering greater capacity than a dense model with a comparable total parameter count.
Compared with the previous-generation model, which had 355 billion total parameters and activated 32 billion parameters, the new generation roughly doubles total capacity and increases active parameters to 40 billion. The pretraining corpus also grows from 23 trillion tokens to 28.5 trillion tokens.
These changes are not simply linear increases. They reflect a new balance among data scale, model capacity, and computational efficiency. In general, total parameters determine the upper bound of a model’s knowledge capacity, active parameters influence the depth of computation performed for each inference request, and training data determines how thoroughly the model learns patterns in the world.
The simultaneous growth across all three dimensions suggests that the new model is not only broader in what it knows, but also deeper in how it reasons.
A Three-Part Architectural Upgrade: MLA, Muon, and Multi-Token Prediction
If MoE provides the model’s structural framework, attention mechanisms and optimization algorithms provide much of its practical efficiency. Three changes are particularly important in the technical design: Multi-head Latent Attention (MLA), the Muon optimizer, and Multi-Token Prediction (MTP).
1. Reducing Attention Overhead with MLA
Traditional Multi-Head Attention (MHA) requires the Key-Value, or KV, representation for every token to be stored in GPU memory during inference. As sequence length increases, the KV cache can grow rapidly and become a major bottleneck for long-context processing. In extremely long-context scenarios, the KV cache may consume 30% to 50% of available GPU memory.
MLA addresses this problem through low-rank compression. It compresses the Key and Value representations into a shared latent vector. During inference, the model stores the compressed latent representation instead of the original Key-Value matrices.
This reduces KV-cache overhead by more than 90% in the reported design. The lower memory requirement makes extremely long contexts more practical on consumer-grade GPUs. It can also improve inference throughput by reducing memory traffic and data movement.
From the perspective of architectural evolution, MLA follows a design direction similar to that used by other leading proprietary models, including a V3 model from an overseas cloud provider. This convergence suggests that low-rank compression has broad value for long-context inference.
2. Optimizer Design: The Geometric Intuition Behind Muon
Optimizer selection directly affects training convergence and final model quality. AdamW, one of the most widely used optimizers, maintains momentum and squared-gradient statistics for each parameter to adapt the learning rate. This leads to memory overhead equivalent to roughly twice the model’s parameter storage.
The technical design described here uses the Muon optimizer. Unlike AdamW, which adapts updates on a parameter-by-parameter basis along coordinate axes, Muon is based on the matrix polar decomposition. It constrains update directions within an orthogonal space.
The underlying intuition is that, for matrix parameters, direction can be more important than scale. By updating along orthogonal directions, Muon can accelerate convergence for matrix parameters with poor conditioning without increasing the optimizer’s memory footprint.
In practical terms, Muon aims to maintain convergence stability comparable to AdamW while reducing demands on memory bandwidth. This can save valuable hardware resources when training very large MoE models. The choice also reflects growing interest in the geometric properties of optimization algorithms.
3. Learning to Predict Further Ahead with MTP
A conventional autoregressive language model predicts only the next token at each position during training. This single-step training objective can be inefficient and may contribute to error accumulation during long-form generation.
Multi-Token Prediction requires the model to predict several future tokens from each position. Although this appears to be a straightforward extension of the training task, it provides two important benefits.
First, it encourages the model to capture longer-range dependencies. The model must learn sentence structure and semantic relationships at a higher level instead of focusing only on local word combinations. Second, MTP provides denser training signals, increasing the amount of useful learning extracted from each training example.
A typical implementation introduces multiple independent prediction heads. Each head predicts tokens at a different future offset. During inference, only the main prediction head is retained, so the additional training objective does not impose a corresponding generation-speed cost.
This mechanism has been shown to improve performance on tasks that require long-range planning, including code generation and mathematical reasoning.
Tackling Extremely Long Contexts with DSA Sparse Attention
As context windows expand from 128K tokens toward 1 million tokens and beyond, the quadratic complexity of full attention, O(n²), becomes increasingly difficult to afford. To process long sequences within a limited compute budget, the technical design introduces DSA, or Dense-Sparse Attention.
DSA follows a divide-and-conquer strategy by splitting attention heads into two groups: dense attention heads and sparse attention heads.
- Dense attention heads: These perform standard attention over the global context. They capture long-range dependencies and maintain semantic coherence. Although they are computationally expensive, they preserve the model’s ability to integrate information across the entire sequence.
- Sparse attention heads: These use fixed or learned sparse patterns, such as local-window or stripe-based attention. They focus only on selected regions of the sequence, substantially reducing computation and making large token volumes more manageable.
By combining the two types of attention heads, DSA aims to approach the representational quality of full attention while delivering a near-linear increase in computational cost.
The key insight is that not every attention head needs to examine every token. Some heads may specialize in local syntactic relationships, while others can selectively skip through the sequence to capture long-range references. This division of labor improves the model’s overall ability to process extremely long contexts.
Data Strategy and Benchmark Performance
The new-generation model continues a strategy that emphasizes both scale and quality. Its pretraining corpus grows from 23 trillion to 28.5 trillion tokens, including 27 trillion tokens used during pretraining and 1.5 trillion tokens used during an intermediate training phase.
The data mixture is not simply expanded through indiscriminate accumulation. Greater emphasis is placed on high-information-density sources, including code, mathematics, and scientific literature. These high-quality datasets provide important training signals for logical reasoning and structured problem solving.
According to the Artificial Analysis Intelligence Index v4.0, the model receives an overall score of 50 and ranks first among open-source models. Its overall capabilities are reported to be approaching those of leading proprietary commercial models.
On the text and code arenas of LMArena, formerly known as Chatbot Arena, the model also achieves the top open-source ranking. Its overall performance is described as being in the same tier as leading international proprietary models.
These results suggest that open-source models are reaching a practical level where developers can consider them as alternatives to closed systems, particularly when control, customization, data privacy, and deployment flexibility are important.
Engineering Implications: From Technical Leadership to Production Use
The technical design offers three key lessons for developers and engineering teams.
Balancing cost and performance
The use of 40 billion active parameters demonstrates that a carefully designed MoE model can deliver an experience close to that of leading proprietary models while keeping the cost of individual inference requests within a more manageable range.
When building Retrieval-Augmented Generation, or RAG, systems and AI agents, enterprise teams do not necessarily need to select the largest available model. Instead, they should evaluate whether the active parameter count matches their task requirements, latency targets, and deployment budget.
Long context is a systems problem
The combination of MLA and DSA shows that long-context capability depends on more than the nominal context-window length. The model must also retrieve and use the right information accurately within a long sequence.
When designing knowledge-base applications, developers should consider combining RAG with foundation models optimized for long contexts. Multi-level caching and indexing strategies are also important for controlling memory use and improving retrieval efficiency.
The growing maturity of the open-source ecosystem
The model provides further evidence that open-source foundation models are becoming viable for production use. For organizations with strict data privacy and compliance requirements, private deployment based on an open-source foundation model can offer a path that balances data control and operational cost.
In this approach, teams may use general-purpose GPU infrastructure and model deployment toolchains, such as those provided through Baidu AI Cloud, to optimize and operate their own deployments. Availability and service coverage should be evaluated for the target region and compliance environment.
The release of a technical report is only the starting point. The next major challenge for AI engineering is to convert advanced architectural capabilities into measurable productivity improvements in real business scenarios.
