Sparse MoE Large Models Explained: Architecture, Efficiency Optimizations, and Engineering Practice
This article examines a next-generation open-source sparse Mixture-of-Experts (MoE) model and the engineering practices behind it. It explains why sparse MoE architectures draw attention, what a configuration of roughly 740 billion total parameters with about 40 billion activated parameters means in practice, and how techniques such as low-rank latent attention, per-head orthogonalization, multi-token prediction, and sparse attention improve long-context efficiency. It also covers asynchronous agentic reinforcement learning for the post-training stage, the training infrastructure required to run these systems at scale, and a multi-dimensional framework for judging whether an open-source model approaches leading closed-source models. Developers and enterprise teams will find practical guidance on turning these techniques into deployable engineering capabilities.
From “Parameter Scale” to “Effective Intelligence”: Why Sparse MoE Draws Attention
In the evolution of large model technology, the industry has long focused on two questions. First, can a model sustain a higher level of intelligence on complex reasoning, code generation, long-text understanding, and agentic tasks? Second, can a model control training and inference costs while scaling up its parameters? The sparse Mixture-of-Experts (MoE) architecture is a representative technical approach built to answer both questions.
The core idea of MoE is to give a model a larger parameter capacity while activating only a subset of those parameters in each forward pass. As a result, the model’s overall expressive power can grow with its total parameter count, while the compute required for a single inference does not need to grow proportionally. For enterprise applications, this design matters for two reasons. On one hand, a larger parameter scale helps improve generalization on complex tasks. On the other hand, the sparse activation mechanism keeps inference costs within a relatively controllable range.
According to publicly disclosed technical reports, a certain next-generation open-source model adopts a large-scale sparse MoE architecture: roughly 740 billion total parameters, about 40 billion activated parameters, 256 experts, and 80 layers. Compared with the previous generation of the same architectural lineage, total parameters expanded from roughly 355 billion to about 740 billion, activated parameters increased from about 32 billion to about 40 billion, and the training data scale grew from about 23T tokens to about 28.5T tokens — with roughly 27T tokens in the pretraining stage and about 1.5T tokens in the mid-training stage.
This shift reflects a clear trend in current large model R&D: simply expanding parameter scale is no longer enough. What truly matters is the coordinated optimization of parameter scale, activation strategy, data quality, training methods, and inference efficiency. A model is no longer simply “better when bigger”; it must find a better balance among intelligence, resource consumption, and engineering usability.
Model Scale and Efficiency: What 740 Billion Total and 40 Billion Activated Parameters Mean
From an engineering perspective, 740 billion total parameters with 40 billion activated parameters represents a typical “large capacity, low activation” design. Its advantages show up in three main areas.
1. Higher model capacity without fully translating into inference cost
In a traditional dense model, every parameter participates in computation at every layer during inference. The more parameters, the greater the compute and memory pressure. A sparse MoE model instead uses a gating network to select only some experts for each computation, leaving most parameters inactive during a single inference. For high-concurrency online services, this pattern helps reduce per-request latency and compute cost.
2. Stronger expressive power on complex tasks through expert specialization
Each expert in an MoE model can be viewed as a relatively independent feature transformation unit. Different experts can learn different language phenomena, code patterns, mathematical structures, or domain knowledge. In tasks such as code generation, tool calling, and multi-turn dialogue, the model dynamically selects an appropriate combination of experts based on the input context, giving it more expressive power than a single-path network.
A simplified MoE forward pass can be expressed as:
def moe_forward(x, router, experts, top_k=2):# 1. Compute routing scores for each expertscores = router(x)# 2. Select the top-k experts with the highest scoresselected = top_k(scores, k=top_k)# 3. Merge the selected experts' outputs by weightoutput = 0for expert_id, weight in selected:output += weight * experts[expert_id](x)# 4. Residual connection and normalizationreturn x + output
This pseudocode only illustrates the core flow. In real engineering, MoE also involves load balancing, expert parallelism, communication overhead, routing stability, and training recoverability.
3. Controllable inference cost, but more complex training and scheduling
The advantages of sparse MoE do not automatically turn into low cost. On the contrary, it places higher demands on the system:
- With many experts, a sound parallelism and partitioning strategy is required;
- Routing can be imbalanced, causing some experts to be overloaded while others are underutilized;
- In distributed training, expert parameters spread across different devices, increasing communication complexity;
- Inference serving must dynamically adjust batching and expert-caching strategies according to request distribution.
The value of an MoE model therefore depends not only on the model architecture itself, but also on whether the underlying training platform, inference scheduling, resource orchestration, and monitoring capabilities are mature enough.
Low-Rank Latent Attention and Per-Head Orthogonalization: Key Optimizations for Long-Context Efficiency
In long-context scenarios, the attention mechanism is usually the main source of memory and compute overhead. Standard multi-head attention needs to cache the keys and values corresponding to historical tokens, and as sequence length grows, the key-value cache expands rapidly. For tasks such as long-document understanding, codebase analysis, and multi-turn agentic interaction, the key-value cache significantly affects inference throughput and memory usage.
To mitigate this, the next-generation model adopts a low-rank latent attention approach. The basic idea is to compress keys and values into a low-dimensional latent space for caching, reducing memory usage and memory-bandwidth pressure during inference.
1. Basic flow of low-rank latent attention
It can be represented by the following pseudocode:
def low_rank_attention(x):# Compress the input into the latent spacekv_latent = project_kv_down(x)# Cache only the compressed latent representation during inferencekv_cache = kv_latent# Compute the query-side latent representationq_latent = project_q_down(x)# Reconstruct keys and values from the latent spacek, v = project_kv_up(kv_cache)# Compute the attention outputattn_score = q_latent @ k.transpose(-1, -2)attn_score = attn_score / sqrt(d_k)attn_weight = softmax(attn_score, dim=-1)return attn_weight @ v
The advantage of this design is that long-context inference no longer needs to store full-precision key-value tensors, but instead stores a compressed representation. For scenarios that must handle large volumes of historical messages, long documents, or long code contexts, low-rank latent attention can significantly reduce resource usage.
2. Training stability: low-rank attention is not a simple add-on
However, the technical report also highlights a key issue: under certain training configurations, combining low-rank latent attention with a more complex orthogonalization training strategy can be less stable than a simpler grouped-query attention baseline.
The reason is not hard to understand. The attention mechanism is essentially a highly coupled matrix computation process. The numerical distributions of queries, keys, values, and projection matrices directly affect training gradients. If the optimizer applies uniform orthogonalization to an entire projection matrix, it may alter the parameter distribution across different attention heads, causing some heads to learn at a different rate than others. When the model is large and has many layers, this imbalance can amplify into training fluctuations.
3. Per-head orthogonalization: keeping each attention head independently stable
To address this, the technical report proposes a per-head approach: instead of uniformly orthogonalizing an entire projection matrix, orthogonalize each attention head separately.
The illustrative logic is as follows:
def per_head_orthogonalize(weight, num_heads, in_dim, out_dim):# Reshape the projection matrix into a multi-head structureweight = reshape(weight, [num_heads, in_dim, out_dim])for head in range(num_heads):# Orthogonalize each head independentlyweight[head] = orthogonalize(weight[head])# Restore the original matrix shapereturn reshape(weight, [in_dim, out_dim])
The significance of this approach is that it:
- Preserves the memory and long-text inference advantages of low-rank latent attention;
- Reduces parameter coupling across attention heads;
- Improves gradient stability in large-scale training;
- Makes the model easier to converge under complex attention structures.
This also reflects an important idea in current large model engineering: rather than simply pursuing more complex structures, seek a more reasonable balance between structural complexity and training stability.
Multi-Token Prediction: From “Predicting the Next Word” to “Predicting Multiple Future States”
Traditional autoregressive language models typically use next-token prediction as the training objective. Given the preceding context, the model outputs a probability distribution over the next token and is optimized via cross-entropy loss. This approach is simple and direct, and has been thoroughly validated in large-scale pretraining.
Multi-token prediction goes further: a model can predict not only the next token but also multiple future tokens at once. This offers value in at least three ways.
1. Higher training signal density
Each input sample can provide multiple prediction targets. The model must not only understand the current context but also form a reasonable expectation of what comes next over a period of time. This is especially important for tasks such as code generation, long-text continuation, and logical reasoning.
A simplified multi-token prediction training objective can be expressed as:
def multi_token_loss(hidden_state, future_tokens):losses = []for step, target in enumerate(future_tokens):logits = predict_head[step](hidden_state)loss = cross_entropy(logits, target)losses.append(loss)return sum(losses)
2. Stronger modeling of contextual structure
Predicting multiple future tokens requires the model not only to capture local language patterns but also to understand longer semantic structures. For example, in code generation the model must anticipate function signatures, variable naming, return types, and subsequent logic; in long-text generation, it must maintain topical consistency and paragraph structure.
3. Potential for inference acceleration
During inference, multi-token prediction can also be combined with draft models, speculative decoding, or local parallel generation. The model can first propose several candidate tokens, which the main model then verifies or corrects. This can reduce the number of sequential generation steps to some extent and improve generation efficiency.
Of course, multi-token prediction is not without challenges. Training requires carefully designed loss weights, and inference must handle the consistency of candidate tokens. If the prediction of future tokens deviates too much from the main model’s generation distribution, it may introduce additional verification costs instead.
Sparse Attention: Balancing Quality and Cost in Long-Context Tasks
Beyond low-rank latent attention, sparse attention is another important direction in this technical report. Standard full attention must compute pairwise relationships among all tokens, and its computational complexity grows quadratically with sequence length. When context length expands from thousands to hundreds of thousands of tokens or more, full attention brings significant compute and memory pressure.
The goal of sparse attention is to reduce unnecessary attention computation while preserving key information as much as possible. Common approaches include local window attention, block-sparse attention, and content-based dynamic sparse attention.
1. The core trade-off of sparse attention
The key to sparse attention is not simply “computing less,” but retaining important interactions as accurately as possible. If the sparse strategy is poorly designed, the model may lose critical long-range information, degrading long-text understanding. Sparse attention therefore typically needs to address the following questions:
- Which tokens or blocks should be attended to?
- What sparsity ratio balances efficiency and quality?
- Can the sparse pattern be learned?
- How should it be combined with low-rank latent attention?
- Does it cause the attention distribution to become overly concentrated on long-context tasks?
An illustrative sparse attention flow is as follows:
def sparse_attention(q, k, v, block_size):# Select several blocks based on the importance of queries and keysselected_blocks = select_blocks(q, k, block_size)scores = []values = []for block in selected_blocks:scores.append(q @ block.keys().transpose(-1, -2))values.append(block.values())# Compute attention over the selected blocksattn_score = concat(scores, dim=-1)attn_score = attn_score / sqrt(d_k)attn_weight = softmax(attn_score, dim=-1)return attn_weight @ concat(values, dim=-2)
2. Efficiency observations in the report
The technical report notes that in certain controlled experiments, the sparse attention scheme came close to the quality of a denser scheme trained on roughly 943.7 billion tokens, using only about 20 billion tokens of training resources. This observation shows that, under the right tasks and the right sparse strategy, a model does not necessarily need full dense computation to achieve comparable quality.
Results like this are highly relevant to enterprise adoption. Scenarios such as long-document question answering, retrieval-augmented generation over codebases, historical dialogue memory, and multi-step agentic tasks often require handling extremely long contexts. If sparse attention can reduce computation while preserving task quality, it can significantly improve service cost and response latency.
Asynchronous Agentic Reinforcement Learning: Solving the “Environment Bottleneck” in Agent Training
If sparse MoE, low-rank attention, multi-token prediction, and sparse attention mainly address model architecture and pretraining efficiency, then asynchronous agentic reinforcement learning mainly addresses the post-training stage.
As large models evolve from “text generation tools” into “agents that can call tools and execute multi-step tasks,” training methods are changing too. Agentic tasks usually involve more than outputting a piece of text; they include:
- Understanding the user’s goal;
- Calling tools such as search, code execution, and database queries;
- Observing the results returned by tools;
- Adjusting the next action based on those results;
- Completing the final goal across multiple rounds of interaction.
Such tasks are naturally suited to reinforcement learning, but they also bring clear challenges: environment interaction is usually much slower than the model’s forward computation. Under a fully synchronous training approach, sampling, environment execution, reward computation, and parameter updates must proceed sequentially, leaving large amounts of compute idle while waiting.
1. The bottleneck of synchronous reinforcement learning
In synchronous mode, the flow is typically:
model generates action -> environment executes -> reward obtained -> model updates -> next round of generation
If environment execution requires calling external tools, running code, or accessing complex systems, each step may introduce substantial latency. Much of the compute in the training cluster may sit waiting for the environment to return results, resulting in low resource utilization.
2. The basic idea of fully asynchronous training
Asynchronous training decouples sampling, environment interaction, reward computation, and policy updates. Different roles can run in parallel:
- The sampling service continuously generates action trajectories;
- The environment service executes tool calls or code runs in parallel;
- The reward service scores trajectories;
- The training service pulls data from the experience pool to update the model;
- The version management service records the policy version behind each trajectory.
Simplified pseudocode:
def async_agent_rl(policy, env_pool, reward_model, experience_pool):# The sampling thread continuously produces trajectorieswhile True:env = env_pool.get()trajectory = env.rollout(policy)reward = reward_model.score(trajectory)experience_pool.add(trajectory, reward)# The training thread continuously consumes experienceswhile True:batch = experience_pool.sample()loss = reinforcement_learning_loss(policy, batch)optimizer.step(loss)
3. The core challenge of asynchronous training: stale data and distribution shift
The biggest challenge of asynchronous training is that trajectories used for training may be produced by an earlier version of the model. This causes the experience data to be inconsistent with the current policy distribution, which can lead to unstable training.
Common solutions include:
- Recording the policy version number when each trajectory is generated;
- Applying importance sampling to old-policy data;
- Limiting the effective window of experience data;
- Using confidence intervals or KL constraints to control the magnitude of policy change;
- Filtering out low-quality trajectories;
- Normalizing rewards to reduce the impact of environment noise.
A simplified version-aware sampling logic can be expressed as:
def sample_for_training(experience_pool, current_version, max_age):samples = []for item in experience_pool:if current_version - item.version > max_age:continueweight = importance_weight(item.version, current_version)if weight > threshold:samples.append((item, weight))return normalize(samples)
4. Benefits of the asynchronous architecture
The value of a fully asynchronous training framework is mainly reflected in:
- Higher training throughput;
- Less idle compute caused by waiting on the environment;
- Support for parallel sampling across multiple environments;
- Adaptability to long-tail latency scenarios such as tool calls and code execution;
- Faster learning from agentic interactions.
These capabilities are especially critical for building models that execute real tasks. Enterprise-grade agents require not only accurate answers, but also stable tool calling, exception handling, adherence to safety boundaries, and continuous policy improvement across multiple rounds of interaction.
Supporting Training Infrastructure: From Algorithm Design to Engineering Implementation
Large-scale sparse MoE models and asynchronous reinforcement learning training place high demands on infrastructure. Algorithm design alone is not enough; stable distributed training, data processing, task scheduling, and observability are also required.
1. Distributed training and expert parallelism
Sparse MoE models must solve the expert load-balancing problem during training. If some experts are frequently activated while others remain idle for long periods, resources are wasted and training efficiency drops. Common approaches include:
- Routing load-balancing loss;
- Expert capacity limits;
- Dynamic expert replication;
- Sharded parallelism and communication optimization;
- Gradient aggregation and parameter synchronization strategies.
On the inference side, caching and scheduling must be based on expert access frequency. Hot experts can be kept near high-speed storage or compute units, while cold experts are loaded on demand per request.
2. Reinforcement learning training infrastructure
Asynchronous agentic reinforcement learning places even higher demands on infrastructure. A relatively complete training system usually includes the following modules:
- Trajectory storage: saving actions, observations, rewards, and policy versions;
- Experience pool: supporting efficient sampling, filtering, and lifecycle management;
- Policy version management: recording data produced by different model versions;
- Reward computation service: supporting rule-based rewards, model-based rewards, or human feedback;
- Environment sandbox: safely executing code, tool calls, or external interactions;
- Monitoring and alerting: observing training loss, reward distribution, throughput, failure rate, and resource utilization.
Its data flow can be represented as follows:
Object storage: store datasets, checkpoints, trajectories and logs|vTraining scheduler: allocate compute resources, manage training tasks|vSampling service: model generates actions|vEnvironment sandbox: execute tool calls, code runs or business interactions|vReward service: compute trajectory scores|vExperience pool: store trajectories with version information|vTraining service: update the model and produce new checkpoints
3. A general implementation approach on Baidu AI Cloud
For enterprise teams, this kind of large-scale model training and inference engineering can typically be built on general-purpose cloud capabilities such as those offered by Baidu AI Cloud. For example:
- Using a container platform to host distributed training, inference services, and asynchronous sampling tasks;
- Using object storage to store training data, model checkpoints, trajectory samples, and evaluation logs;
- Using a message queue to connect the sampling service, environment service, reward service, and training service for asynchronous decoupling;
- Using a logging service to record training errors, environment failures, request latency, and policy update traces;
- Using monitoring and alerting to observe GPU utilization, memory usage, throughput, queue backlog, failure rate, and end-to-end latency.
This combination of general capabilities is not tied to any specific model; it applies to many engineering scenarios such as large-scale model training, agentic reinforcement learning, and inference service deployment. Enterprises can gradually build offline training, online inference, and continuous evaluation systems according to their own business scale.
Evaluation Perspective: How to Judge Whether an Open-Source Model Approaches Leading Closed-Source Models
When selecting a model, parameter scale alone is not enough. A more reasonable approach is to establish a multi-dimensional evaluation system.
1. General capability evaluation
General capability evaluation usually focuses on a model’s performance in language understanding, logical reasoning, knowledge question answering, and mathematical computation. Public composite intelligence indices can serve as a reference, but enterprises should focus more on task sets highly relevant to their own business.
For example:
- Customer service scenarios: multi-turn dialogue understanding, intent recognition, compliant responses;
- Code scenarios: code completion, defect localization, unit test generation;
- Data analysis scenarios: SQL generation, metric interpretation, report generation;
- Agentic scenarios: tool calling, task planning, exception recovery.
2. Long-text and code capabilities
Long-text capability requires examining a model’s performance on tasks such as long-document summarization, cross-paragraph reasoning, contract review, and codebase understanding. Code capability requires attention to code correctness, executability, security, and understanding of engineering context.
3. Cost and latency
Enterprises must consider inference cost during deployment. Evaluation metrics include:
- Average per-request latency;
- Peak-period throughput;
- Memory usage for long-context requests;
- Batching efficiency;
- Tool-calling success rate;
- Exception retry cost;
- Manual review cost.
4. Safety and controllability
One advantage of open-source models is that they are auditable, can be privately deployed, and can be customized. However, enterprises still need to establish safety evaluation mechanisms, including:
- Sensitive content filtering;
- Access control;
- Tool-calling whitelists;
- Code execution sandboxes;
- Log retention;
- Manual review processes.
Conclusion: Sparsification, Asynchronization, and Engineering Are Becoming the Main Lines of Large Model Competition
Overall, the direction shown by this next-generation sparse MoE model technical report represents an important trend in current large model engineering: models no longer pursue parameter-scale expansion alone, but place greater emphasis on sparse activation, long-context efficiency, multi-token prediction, asynchronous reinforcement learning, and underlying training infrastructure.
For developers and enterprise users, understanding the value of these techniques is not only about reading model reports, but more about turning them into deployable engineering capabilities. Whether it is large-scale sparse training, low-rank attention optimization, or agentic reinforcement learning, all ultimately rely on stable distributed resources, efficient data pipelines, observable monitoring systems, and flexible cloud deployment.
In this context, enterprises can gradually build a complete pipeline — from data processing, model training, and evaluation validation to online inference — tailored to their own business scenarios. With the general capabilities of cloud platforms such as Baidu AI Cloud, teams can focus more on model strategy, business logic, and quality optimization, thereby improving the delivery efficiency and long-term maintainability of large model applications.
