MoE Architecture Evolution and Training Optimization in Chinese LLMs

    This article examines the core technical directions behind current Chinese large language models: sparse MoE scaling, attention efficiency with MLA and reduced attention heads, and pretraining improvements such as high-quality data pipelines and the Muon optimizer. It also discusses trade-offs in routing, communication, KV cache memory, and inference latency, offering developers and enterprises a technical reference for architecture evaluation and model selection.

    Large language model (LLM) technology is advancing rapidly, and several high-performance models are now available. At the underlying architecture level, the differences among current mainstream models are not as large as they may appear. To a significant extent, they share similar design philosophies and technical foundations. Nevertheless, specific architecture configurations and training strategies reflect distinct technical considerations. This article sets aside specific brands and products to analyze the core strengths and evolution trends of current large models from a technical perspective, providing developers and enterprise users with a reference for technology selection and architecture evaluation.

    1. Evolution of Mixture-of-Experts Architecture and Sparse Activation

    Mixture-of-Experts (MoE) architectures at the trillion-parameter scale have become a mainstream choice for improving model performance and efficiency. In one recently released model, for example, the total parameter count reaches 1.04T, while only about 32B parameters are activated during inference. This sparse activation design delivers significant performance gains: the model maintains a large knowledge capacity while effectively controlling inference compute overhead.

    Analysis based on scaling laws indicates that continued increases in sparsity can bring meaningful performance improvements. To achieve this, the research team increased the number of experts to 384, compared with 256 experts in an earlier base model, further refining the division of labor among experts. The model hidden dimension is set to 7168, while the expert hidden dimension is 2048. More experts allow the model to route knowledge from different domains to more appropriate expert networks. For example, in tasks such as code generation, mathematical reasoning, or natural language understanding, the routing mechanism can dynamically assign tokens to the experts best suited to handle them. This substantially expands model capacity without significantly increasing the compute required for a single inference pass. However, it also places higher demands on routing algorithm robustness and cross-node communication overhead, requiring deep engineering optimization in distributed training and inference frameworks.

    2. Balancing Attention Mechanisms and Inference Efficiency

    For attention mechanisms, current mainstream large models commonly adopt innovations such as Multi-head Latent Attention (MLA) to alleviate the high KV cache memory usage of traditional multi-head attention (MHA) when processing long texts.

    MLA uses low-rank compression to project Key and Value into a low-dimensional latent space and then dynamically reconstructs them during inference, greatly reducing KV cache memory requirements. This enables the model to handle longer contexts and improves long-text understanding and generation.

    To further reduce compute overhead during inference, some models make targeted architectural adjustments. For example, the number of attention heads is reduced to 64, whereas earlier models typically used 128. This head-reduction strategy effectively lowers the time complexity of attention computation without significantly affecting model expressiveness. In autoregressive generation, attention computation is one of the main sources of latency. Reducing the number of attention heads can directly improve decoding speed, giving large models higher throughput and lower response latency in real deployments. This is particularly important for large-scale concurrent requests in enterprise application scenarios.

    3. Training Optimization: From Optimizers to Data Quality

    During pretraining, data quality and optimization algorithms are key determinants of a model’s final capabilities.

    The first challenge is at the data level. As the availability of high-quality human data becomes increasingly constrained, token efficiency is becoming a critical factor in scaling large language models. This means models need to extract as much knowledge as possible from limited high-quality data. Therefore, one model used up to 15.5T high-quality tokens during pretraining and applied fine-grained control over data cleaning, deduplication, and mixing. In practice, this involves complex data processing pipelines, including heuristic rule-based filtering, deduplication algorithms such as MinHash, and optimization of data mixing ratios across different subject domains. These steps help the model converge to a better state during pretraining and avoid overfitting or knowledge imbalance.

    Second, in optimizer selection, the industry has seen innovations beyond traditional AdamW. One team adopted the Muon optimizer in training its base model. Unlike Adam-series optimizers, which rely on momentum and adaptive learning rates, Muon introduces operations such as matrix orthogonalization to orthogonalize gradient updates. This helps ensure the orthogonality of weight update matrices, providing a more stable gradient descent direction in the early stages of training and helping avoid gradient explosion or vanishing. The Muon optimizer accelerates model convergence and may help the model reach a lower loss value with the same amount of data. This optimizer innovation offers new ideas for the training efficiency and stability of large models and provides a feasible alternative for future large-model pretraining.

    4. Conclusion and Outlook

    In summary, the core advantages of current large models are not primarily reflected in disruptive architectural innovation, but rather in fine-grained adjustments to MoE architectures, efficiency optimization of attention mechanisms, and continuous refinement of training strategies. By increasing the number of experts to improve sparsity, optimizing the number of attention heads to reduce inference overhead, and introducing new optimizers to improve training efficiency, these technical practices together drive overall improvements in model capability.

    For developers and enterprise users, understanding these underlying technical details helps make more rational judgments during model selection. For example, if a business scenario is extremely sensitive to inference latency, models optimized through attention-head reduction may offer advantages. If the scenario requires processing very long documents, models with efficient latent attention mechanisms may be a better choice. At the same time, these technical trends provide valuable references for developing and fine-tuning applications based on large models. In the future, as data dividends fade, finding new breakthroughs in architecture innovation and algorithm optimization will remain a shared challenge for the entire large-model industry.