Large-Scale Sparse Mixture-of-Experts Models: Architecture, Training, and Engineering Insights
As open-source AI competition shifts from raw parameter counts to useful work per unit of compute, large-scale sparse Mixture-of-Experts models are becoming an important design direction. This article examines how sparse activation, low-memory attention, multi-token prediction, dynamic attention patterns, and asynchronous reinforcement learning can improve reasoning, code generation, and agent performance while highlighting the infrastructure and evaluation challenges that determine real-world efficiency.
In recent years, competition among open-source large language models has gradually shifted from “how many parameters does a model have?” to “how much useful work can it complete per unit of compute?” A recently published technical report describes a relatively complete approach: use a massive Mixture-of-Experts (MoE) model to provide broad knowledge capacity, control inference costs through sparse activation, and combine new attention mechanisms, multi-token prediction, and asynchronous reinforcement learning to improve performance on complex reasoning, code generation, and AI agent tasks.
This article does not promote a specific product. Instead, it examines the design principles, training methods, and engineering value of this class of models from a general technical perspective.
1. From Parameter Scale to Effective Compute Efficiency
The model described in the report uses a Mixture-of-Experts architecture with approximately 744 billion total parameters, while activating about 40 billion parameters during a single inference pass. It contains 256 expert modules and 80 layers, and is trained on roughly 28.5 trillion tokens. Most of the data is used for pretraining, with the remainder allocated to intermediate training and capability reinforcement.
The central idea is to decouple model capacity from computation.
A traditional dense model uses all of its parameters at every layer. As the parameter count increases, matrix computation, memory consumption, and communication pressure during inference generally increase as well. An MoE model, by contrast, selects only a small number of experts for each input. A coding-related query may be routed primarily to experts that have learned programming patterns, while a mathematical reasoning task may activate another group of experts.
This design provides two major benefits:
- Higher total capacity: More experts can store more specialized knowledge and task patterns.
- Lower per-request compute: Each input activates only a subset of the model instead of running every parameter.
However, sparse activation does not automatically eliminate system-level costs. Data dispatch between experts, cross-device communication, load balancing, and routing stability all affect actual throughput. The key to an effective MoE system is therefore not simply the number of experts, but also the routing strategy, parallelism design, and communication scheduling.
2. Three Major Directions in Architecture Evolution
The most notable architectural changes described in the report focus on three areas:
- Low-dimensional latent attention for long contexts;
- Multi-token prediction;
- Dynamic sparse attention.
These designs address three different problems: memory consumption, inference speed, and the computational complexity of long-context processing.
2.1 Low-Dimensional Latent Attention and Optimizer Adaptation
In standard self-attention, keys and values are stored in a cache so that they can be reused when the model generates subsequent tokens. As the context grows, the memory required for the key-value cache also increases. In large-scale inference, the cache can become a more serious bottleneck than the model weights themselves.
The basic idea behind low-dimensional latent attention is to compress keys and values into a lower-dimensional latent space and reconstruct or process the relevant information when needed. This reduces the size of the key-value cache and is particularly useful for long-context and high-concurrency generation.
Changes to the attention structure can nevertheless affect training stability. The report notes that when using an optimizer based on certain matrix-geometric transformations, applying the method directly to a complete projection matrix may not produce ideal results. It may even underperform a structurally simpler grouped-query attention design.
To address this issue, the researchers proposed an optimization method that processes attention heads separately. Its main principles are:
- Treat each attention head independently instead of processing the entire projection matrix as one unit;
- Apply orthogonalization or normalization separately to different heads;
- Preserve the relatively independent geometric structure of each head;
- Reduce interference between attention heads.
This illustrates why model architecture and optimizer design cannot be considered independently. An optimization strategy that works well for conventional attention may require significant adaptation when applied to a low-cache attention structure. In large-model training, the optimizer, parameter layout, normalization method, and parallelism strategy effectively form a single integrated system.
2.2 Multi-Token Prediction: Producing More Information per Forward Pass
Traditional autoregressive models usually predict one token at a time. Generating a sequence of length N requires N rounds of sequential prediction. Even if each round is fast, the sequential dependency limits overall generation speed.
Multi-token prediction attempts to have the model predict several future positions after a shared backbone computation. The primary prediction head may generate the next token, while additional heads estimate several subsequent tokens. During training, future ground-truth tokens can be used as supervision to help the model learn longer local dependencies.
This approach is valuable for two main reasons:
- It increases the amount of information produced by each forward pass;
- It provides better candidate sequences for speculative decoding.
During inference, the system can first generate multiple candidate tokens quickly and then verify them with the main model. If most of the candidate sequence is correct, the system can reduce the number of full decoding rounds and improve throughput.
Multi-token prediction also introduces new training challenges. Prediction uncertainty generally increases for positions farther in the future, and different prediction heads may compete for gradient updates. Training therefore requires carefully selected loss weights and mechanisms that prevent the model from over-relying on a single prediction position.
From an engineering perspective, multi-token prediction does not make generation fully parallel. Language generation remains subject to causal dependencies. It can, however, reduce the number of sequential steps through candidate generation and batched verification.
2.3 Dynamic Sparse Attention: Reducing Long-Context Computation
The computational complexity of standard full attention generally grows quadratically with sequence length. When a context expands from several thousand tokens to hundreds of thousands or more, allowing every position to interact with every other position becomes extremely expensive.
Dynamic sparse attention allows each query position to focus only on a subset of important positions instead of accessing the entire context. The sparse pattern can change according to the content. For example, the model may:
- Focus on nearby local regions;
- Attend to structured positions such as document titles and section summaries;
- Retrieve passages related to the current question;
- Preserve a small number of global aggregation positions;
- Selectively connect definitions, calls, and dependencies in source code.
An important conclusion reported in the study is that, with a relatively limited number of training tokens, sparse attention can achieve results close to those of much larger-scale full-attention training. This suggests that model efficiency does not have to come solely from adding more data and compute. More effective information-access patterns can also improve capability.
Sparse attention is not simply a matter of “looking at less content.” If important information is filtered out incorrectly, the model may omit facts, fail at cross-paragraph reasoning, or lose long-range dependencies. Sparse patterns therefore need to be robust and may need to work with global summaries, retrieval mechanisms, or strategies that preserve critical positions.
3. An Asynchronous Reinforcement Learning Framework for AI Agents
Beyond the base architecture, another important direction in the report is asynchronous reinforcement learning for agent tasks.
A conventional reinforcement learning workflow typically includes the following steps:
- The model generates a trajectory;
- The environment executes the model’s actions;
- The result is evaluated and rewards are calculated;
- Model parameters are updated;
- The updated model generates new trajectories.
When environment execution times vary significantly, the entire process can be slowed by the slowest task. A simple problem may finish in a few seconds, while a complex coding task may require multiple tool calls and a much longer execution time. In a fully synchronous batch process, every sample must wait for the slowest task to complete.
An asynchronous framework separates sampling, execution, evaluation, and training into relatively independent pipelines:
- Sampling processes continuously generate task trajectories;
- Environment processes execute tool calls in parallel;
- Evaluation processes calculate rewards and quality signals;
- Training processes continuously consume available samples and update the model;
- A parameter service distributes model versions to different workers.
This design can improve hardware utilization and reduce idle time, especially for agent tasks involving frequent tool calls and unpredictable execution latency.
Asynchronous training also introduces the problem of stale samples. A trajectory may be generated by an older model version but used for training after several parameter updates have already occurred. Using it without adjustment can introduce policy bias. Common approaches include:
- Recording the model version that generated each sample;
- Reweighting samples according to policy differences;
- Limiting the maximum acceptable version delay;
- Dropping or down-weighting excessively old trajectories;
- Combining online samples with high-quality offline data.
The challenge in asynchronous reinforcement learning is therefore not merely parallelizing tasks. The system must balance throughput, sample freshness, and training stability.
4. Why Custom Training Infrastructure Matters
When a model reaches tens of billions of activated parameters or an even larger scale, general-purpose training frameworks may not satisfy all operational requirements. The training infrastructure must handle:
- Data parallelism, tensor parallelism, and pipeline parallelism;
- Expert parallelism and cross-device routing;
- Data exchange over high-speed interconnects;
- Dynamic batching;
- Splitting and managing long-sequence samples;
- Resource isolation for training, sampling, and evaluation;
- Checkpointing and fault recovery;
- Monitoring of training metrics and abnormal states.
MoE models face a particular challenge: uneven expert utilization can leave some devices overloaded while others remain underused. Systems typically need capacity limits, routing constraints, or load-balancing losses to prevent individual experts from becoming hotspots.
Agent reinforcement learning introduces additional requirements, including environment scheduling, tool-execution timeouts, trajectory storage, and reward propagation. A mature infrastructure platform must provide capabilities similar to those of a distributed operating system, managing tasks, data, devices, versions, and failures.
In enterprise environments, these capabilities can be integrated with object storage, message queues, container platforms, logging services, and monitoring and alerting systems to create a training platform that is observable, recoverable, and scalable. General infrastructure categories provided by cloud platforms such as Baidu AI Cloud can also be used to support large-scale training and inference, although the specific architecture must be designed according to the model size, network conditions, and task requirements.
5. How to Interpret Evaluation Results Correctly
Large language model evaluation should not rely on a single overall score. Different benchmarks measure different capabilities:
- General evaluations focus on knowledge, reasoning, and instruction following;
- Dialogue evaluations focus on preference and quality in real interactions;
- Code evaluations focus on program generation, repair, and explanation;
- Long-context evaluations focus on retrieving information across documents;
- Agent evaluations focus on planning, tool use, and task completion rates.
Strong performance on a general benchmark does not necessarily mean that a model will perform equally well in every real-world application. When selecting a model for enterprise use, organizations should build their own test sets covering at least the following dimensions:
- Business terminology and internal knowledge;
- Consistency across multi-turn conversations;
- Stability of structured outputs;
- Code correctness and executability;
- Citation accuracy in long documents;
- Tool-call success rates;
- Latency, throughput, and memory consumption;
- Safety refusals and sensitive-information handling.
The value of an open-source model is also not limited to access to its weights. Deployability, fine-tuning support, and observability are equally important. Real engineering value must be assessed across model capability, hardware cost, operational complexity, and data security.
6. Key Takeaways for Developers
This technical direction offers three main lessons.
First, future model optimization will place greater emphasis on effective compute. Total parameter count remains important, but activated parameters, cache consumption, communication efficiency, and task completion per unit cost are equally significant.
Second, model architecture and the training system must be designed together. Attention mechanisms, optimizers, parallelism strategies, and inference scheduling are tightly coupled. Optimizing one component in isolation does not necessarily improve the system as a whole.
Third, agent capabilities require a dedicated training paradigm. Simply expanding pretraining data is unlikely to fully solve tool use, task planning, and long-horizon execution. Asynchronous sampling, environment feedback, and reinforcement learning infrastructure will become increasingly important.
Overall, the development of large-scale sparse models is not simply about increasing parameter counts. It is about combining architectural sparsity, attention compression, faster prediction, and redesigned training pipelines so that limited computing resources produce stronger performance on practical tasks. This approach offers useful reference points for model researchers, infrastructure engineers, and enterprise teams planning to deploy large language models.
