Sparse MoE Large Models in the Trillion-Parameter Era: Architecture Design and Training Practices

    This article examines how a 744B-parameter open-source Mixture-of-Experts model activates only 40B parameters per inference step while achieving leading results among open models. It explains the economics of total versus active parameters, the roles of low-rank latent attention, multi-token prediction, sparse attention, and optimizer improvements, and provides practical guidance for deployment, evaluation, quantization, and fine-tuning.

    1. Why This Generation of Open Foundation Models Matters

    Competition in the large-model industry has moved beyond whether a model exists to how capable it is. One notable recent development is an open-source MoE model with 744B total parameters that activates only 40B parameters during inference. It has achieved leading results among open models across several authoritative third-party evaluations, placing its overall capabilities in the same tier as top proprietary models.

    For developers and enterprises, this signals two important changes:

    1. The capability gap between open and proprietary models is narrowing rapidly. Enterprises with data-compliance and private-deployment requirements now have more self-controlled options that are capable enough—and in some cases highly effective—for production use.
    2. Architectural innovation matters more than simply increasing parameter counts. This model is not merely a larger version of its predecessor. It introduces systematic changes to attention mechanisms, optimizer strategies, and training objectives. These design choices provide useful references for teams training or fine-tuning large models.

    The following sections analyze the model from four perspectives: its core specifications, three major architectural changes, training strategies, and engineering deployment considerations.

    2. Model Fundamentals: The Parameter Economics of MoE

    2.1 Separating Total Parameters from Active Parameters

    The model follows the general Mixture-of-Experts (MoE) approach. The central idea is to replace the feed-forward network (FFN) in a conventional Transformer with multiple expert sub-networks. A router dynamically selects a small number of experts based on the features of each input token.

    This design produces two important effects:

    • Model capacity is determined by total parameters. More experts allow the model to store more domain knowledge and behavioral patterns.
    • Inference cost is determined primarily by active parameters. Each forward pass uses only a small number of computational paths, significantly reducing compute and memory requirements compared with a dense model of the same total size.

    In its stated configuration, the model contains 256 experts and 80 layers, with 744B total parameters and 40B active parameters per inference step. The active-parameter ratio is approximately 5.4%.

    By comparison, the previous-generation model had 355B total parameters and 32B active parameters. Total parameters increased by more than two times, while active parameters grew by only 25%. This illustrates the typical MoE pattern: model capacity can grow faster than computational cost.

    A simplified implementation of an MoE layer can be expressed as follows:

    1. def moe_forward(x, experts, router, top_k=2):
    2. # x: [batch, seq_len, hidden_dim]
    3. gate_scores = router(x) # Compute the matching score for each expert
    4. topk_ids = gate_scores.topk(top_k).indices # Select only the top-k experts
    5. output = 0
    6. for i in range(top_k):
    7. expert = experts[topk_ids[..., i]]
    8. weight = gate_scores.gather(-1, topk_ids[..., i:i+1])
    9. output += weight * expert(x) # Combine expert outputs with routing weights
    10. return output

    2.2 Scaling the Pretraining Dataset

    An increase in parameter count must be accompanied by a corresponding increase in training data; otherwise, some of the model’s capacity may remain underutilized.

    The model’s pretraining corpus grew from 23T tokens in the previous generation to 28.5T tokens. Approximately 27T tokens were used during the main pretraining stage, with another 1.5T tokens used during the intermediate-training stage.

    This intermediate stage is particularly notable. It commonly covers annealing, increased weighting of high-quality data, long-context training, and reinforcement of reasoning capabilities. Major model teams have increasingly adopted this refinement stage, and its influence on the final capability ceiling may be comparable to that of the main pretraining phase.

    2.3 Evaluation Results

    Across several authoritative third-party evaluations, the model reportedly achieved a score of 50 on a comprehensive intelligence index, placing it among the leading open models. It also ranked first among open models in both text and code tracks of a community-based arena. Overall, its performance was reported to be in the same tier as leading proprietary models.

    These rankings should still be interpreted carefully. Any benchmark represents only one aspect of model capability. Enterprises should validate candidate models against private evaluation sets that reflect their own applications and risk requirements.

    3. Three Major Architectural Changes: From Bigger to Better

    The key difference between this generation and its predecessor lies in three systematic architectural changes.

    3.1 Low-Rank Latent Attention and Optimizer Improvements

    MLA (Multi-Latent Attention) projects keys and values into a lower-dimensional latent space and caches only compressed representations during inference.

    Traditional multi-head attention must cache complete key-value matrices. During long-context inference, both memory usage and memory-access overhead grow linearly with sequence length. MLA uses low-rank compression to reduce the size of the KV cache by an order of magnitude, significantly improving throughput and time to first token in long-context scenarios.

    This approach has already been validated by several leading models and is becoming a common design choice for long-context systems.

    At the optimizer level, the team adopted a method based on Muon-like second-order or adaptive update concepts, with a split adaptation for distributed training—referred to as “Split.” At very large training scales, this approach reportedly provides better convergence efficiency and stability than conventional AdamW-style methods.

    The engineering lesson is clear: once a model crosses a certain scale threshold, optimizer selection is no longer a default implementation detail. It becomes a decision that directly affects training cost and final model quality.

    3.2 Multi-Token Prediction

    In conventional autoregressive training, each position predicts only the next token. This produces relatively sparse supervision and limits the efficiency of gradient utilization.

    Multi-token prediction enables the model to predict several subsequent tokens at each position. It offers two main benefits:

    • Denser training signals: A single forward pass produces multiple sources of supervision, improving data efficiency.
    • A foundation for speculative decoding: The model naturally develops the ability to draft multiple tokens in one step. During inference, these drafts can be combined with a verification mechanism to accelerate generation.

    3.3 Sparse Attention

    MoE reduces computation in the FFN layers, but attention remains a dense O(n²) operation for long sequences.

    Dynamic sparse attention allows each token to attend only to a small number of selected key tokens, reducing the attention cost for long contexts. It complements MLA rather than replacing it:

    • MLA reduces the size of the KV cache.
    • Sparse attention reduces the amount of attention computation.

    Together, these techniques support lower-cost inference over very long contexts.

    4. Practical Guidance for Developers

    For enterprise teams planning private deployment or further development on an open foundation model, the following considerations are important:

    1. Choose an appropriate deployment architecture. Even with only 40B active parameters, a 744B MoE model has a large total weight footprint and requires a multi-GPU cluster. Teams should consider deploying it through a mature large-model inference framework together with cloud containers and GPU resources. Cloud platforms such as Baidu AI Cloud, which provide elastic GPU capacity and model-serving capabilities, can help reduce operational overhead.

    2. Apply quantization and inference optimization. The main memory bottlenecks for an MoE model are expert weights and the KV cache. Quantization, native MLA-based KV reuse, and continuous batching can be combined to improve throughput.

    3. Evaluate before selecting a model. Public leaderboards should not be the sole basis for an enterprise model decision. Build an evaluation set that reflects the target business scenarios, and validate the model across general capabilities, coding, instruction following, and long-context performance.

    4. Use a cautious fine-tuning strategy. Fine-tuning an MoE model requires attention to router stability. In many cases, it is advisable to freeze router-related parameters or update them with a low learning rate to avoid imbalanced expert utilization.

    5. Conclusion

    The progress of this generation of open foundation models is essentially a three-part improvement across parameter scale, architectural efficiency, and data quality.

    MoE increases model capacity. MLA and sparse attention reduce inference costs. Multi-token prediction and optimizer improvements increase training efficiency. For the first time, open models are approaching leading proprietary systems in overall capability, signaling a broader transition toward more accessible large-model technology.

    For enterprises, this is a reasonable time to evaluate private deployment and begin building more autonomous AI capabilities.