The Evolution of Chinese Foundation Models: Core Technologies and a Deep Dive into MoE Architecture
This article examines the technical foundations behind leading Chinese large language models, focusing on long-context processing, fast inference, Mixture-of-Experts (MoE) architecture, optimizer innovation, training stability, synthetic data, and cloud-native deployment. It explains how expert sparsity, attention optimization, Muon, QK-Clip, and distributed infrastructure work together to improve parameter efficiency, token efficiency, and enterprise usability.
As large language model development accelerates in China, different models are developing distinct strengths for specific application scenarios. Some excel at long-context processing and deep reading, making them suitable for legal, research, and analytical work. Others focus on engineering integration and fast inference for complex enterprise tasks and real-time responses. Some emphasize lightweight deployment and multimodal capabilities.
Despite these differences at the application layer, their underlying technical approaches often converge. Many advanced models rely heavily on Mixture-of-Experts architectures and related innovations. This article analyzes the core technical logic behind these models from three perspectives: model architecture, optimizer evolution, and cloud engineering practice.
1. Scenario-Based Capabilities and Their Technical Foundations
Before examining the underlying architecture, it is useful to clarify how large language models demonstrate different capabilities across business scenarios. These capabilities are supported by specific network structures and training strategies rather than existing in isolation.
Long-Context Processing and Deep Reading
When processing lengthy financial reports, legal case files, or large codebases, a model needs strong context compression and retrieval capabilities. Some current models support context windows of one million tokens or more.
The key to this capability lies in optimizing the attention mechanism. In traditional multi-head attention, computational complexity grows quadratically as sequence length increases. To address this limitation, the industry has introduced latent attention mechanisms. By compressing the key-value cache into a latent space, these mechanisms substantially reduce memory usage during long-sequence inference. As a result, models can process extremely long documents within limited GPU memory while maintaining accurate information extraction and deep reading performance.
Engineering Capability and Fast Inference
Enterprise workloads often require very low response latency without sacrificing generation quality. This type of fast user experience depends not only on hardware performance but also on architectural engineering.
Sparse-activation models such as MoE play an important role here. During inference, each token activates only a small number of expert networks. This allows the model to maintain a very large total parameter count while reducing the actual computation to a level comparable to that of a conventional dense model. As a result, the model can continue to respond smoothly under highly concurrent, real-time workloads.
2. The Evolution of MoE Architecture and the Exploration of Sparsity
Technical reports on leading trillion-parameter models show a high degree of architectural similarity. Many use Transformer-like Mixture-of-Experts designs. The central idea is to use a routing mechanism to assign each input token to the expert subnetworks best suited to process it.
Balancing Sparsity and the Number of Experts
Scaling-law analysis suggests that, with a fixed total parameter budget, increasing model sparsity—by adding more experts while reducing the number of parameters activated for each token—can produce significant performance gains.
For example, one frontier trillion-parameter model has approximately 1.04 trillion total parameters, while activating only about 32 billion parameters during each forward pass. To further improve performance, its development team increased the number of experts to 384, a substantial increase over the 256 experts used by earlier models of a similar type.
A larger expert pool enables more fine-grained knowledge specialization. The model can therefore match incoming tokens more precisely with the expert capacity most relevant to their domain or characteristics.
Aggressively Reducing Inference Overhead
While increasing the number of experts, developers must also control inference costs. One approach is to significantly reduce the size of the attention module. For example, the number of attention heads can be reduced from the conventional 128 to 64.
This asymmetric design principle—expanding capacity in the expert layers while reducing computation in the attention layers—is an important strategy for improving model performance under constrained computing resources.
A representative MoE configuration can be summarized as follows:
Total parameters: ~1.04TActivated parameters: ~32BHidden dimension: 7168Expert hidden dimension: 2048Total number of experts: 384Number of attention heads: 64 (typically 128 before optimization)
3. Optimizer Evolution and Token Efficiency in Pretraining
If the MoE architecture forms the skeleton of a large model, pretraining provides the process through which useful capabilities are developed. As high-quality human-generated text becomes increasingly scarce, token efficiency—the amount of information a model can extract from each token—has become a major bottleneck in scaling.
Adopting New Optimizers
Large-scale pretraining has traditionally relied on optimizers such as AdamW. However, at the trillion-parameter scale, optimizer memory consumption and update efficiency become increasingly challenging.
Some advanced models have begun experimenting with new optimizers such as Muon. Muon orthogonalizes gradient matrices to enable more stable weight updates. This can help the model extract more useful features from the same amount of training data. The optimizer is designed to improve token efficiency and can help mitigate the limitations caused by the shortage of high-quality data.
Training Stability and QK-Clip
After introducing a new optimizer and increasing model scale, training can become vulnerable to instability, including sudden loss spikes.
To address this issue, development teams have introduced QK-Clip. During training, this technique dynamically clips the dot-product results between queries and keys in the attention mechanism. This helps prevent the softmax function from entering a region where gradients vanish, improving the stability of large-scale distributed training.
Expanded Use of Synthetic Data
To extract more value from existing high-quality tokens, advanced models increasingly incorporate synthetic data generation into pretraining workflows.
A stronger existing model can generate high-quality instruction data with a targeted distribution. This data can then be used for pretraining or fine-tuning a new model. The approach helps compensate for the limited supply of real-world data and can improve performance on complex logical reasoning tasks.
4. Engineering and Cloud Deployment Under Architectural Convergence
From a technical perspective, the differences between leading Chinese foundation models at the base-architecture level are becoming narrower. They can increasingly be viewed as different branches of evolution within a shared technical lineage.
The more meaningful sources of differentiation often lie in engineering implementation and cloud deployment capabilities.
For enterprise developers, applying these models requires infrastructure that can support large parameter counts and complex routing mechanisms. In practical deployments, organizations typically rely on mature cloud-native platforms. For example, container platforms and high-performance computing instances provided by Baidu AI Cloud can be used to build distributed inference clusters for MoE-based models.
Cloud object storage and message-queue services can support the continuous delivery of large-scale training data. Comprehensive logging, monitoring, and alerting systems can also help developers track expert-routing hit rates, peak GPU memory usage, and inference latency in real time. These metrics provide a basis for tuning batch sizes and parallelization strategies.
Conclusion
The strengths of Chinese foundation models in long-context processing, lightweight deployment, and engineering integration are fundamentally the result of continued innovation in MoE architecture, attention optimization, and pretraining algorithms.
Increasing the number of experts, adopting the Muon optimizer, and applying QK-Clip are all steps toward improving token efficiency and computational efficiency. For developers and enterprise users, understanding these underlying technical principles makes it easier to select an appropriate model for a specific business scenario. It also establishes a solid foundation for future private deployment, cloud-native implementation, and secondary development of large models.
