Chinese LLM Architecture Comparison: MoE, MLA, Training, and Evolution

    This article analyzes the shared technical foundation and differentiated design choices of Chinese large language models, covering sparse MoE Transformer architectures, MLA attention, the tradeoff between expert count and attention heads, optimizer and token-efficiency efforts, and post-training strategies. It also offers product-neutral criteria for developers evaluating task fit, deployment cost, licensing, and engineering integration.

    In recent years, Chinese large language models have displayed a distinct pattern: strong technical convergence with innovation dispersed across specific areas. The flagship models released by several leading teams are highly similar at the overall architecture level, generally adopting a Mixture-of-Experts (MoE) Transformer route. Their differentiation appears in details such as expert scale, attention-head design, and optimizer choice. This article does not benchmark or recommend any specific product. Instead, it examines the technical commonalities and innovation logic of this model generation from a general engineering perspective.

    1. Why Mainstream Architectures Share a Common Origin

    Nearly all current Chinese flagship model foundations follow the same technical paradigm: an MoE Transformer. This paradigm has been widely validated for scalability. Under an equivalent compute budget, a sparsely activated model can hold a larger total parameter count while keeping inference cost manageable. The relative similarity is therefore not accidental; it is a rational convergence guided by scaling laws.

    The meaningful differences appear at three levels: architecture hyperparameter tradeoffs, training infrastructure and optimizers, and data engineering and post-training strategies.

    2. Architecture: The Tradeoff Between Expert Count and Attention Heads

    2.1 Widespread Adoption of MLA Attention

    Multi-head Latent Attention (MLA) projects keys and values into a low-dimensional latent space, substantially compressing KV cache memory usage. This brings clear benefits for long-context inference and batch deployment. Most Chinese MoE models now use this mechanism. Hidden dimensions are commonly around 7,000, while expert hidden dimensions are around 2,000, forming a de facto industry standard.

    2.2 Opposite Adjustments to Expert Count and Attention Heads

    An interesting architecture trend is emerging. As scaling-law analyses indicate that greater sparsity can deliver meaningful performance gains, some teams have expanded the number of experts from 256 to 384 or more. At the same time, to offset the resulting increase in inference overhead, they have reduced the number of attention heads from 128 to 64.

    The reasoning behind this tradeoff deserves attention:

    • More experts provide greater model capacity and finer-grained knowledge partitioning. Each activation can draw on more flexible expert combinations, which may help with complex tasks.
    • Fewer attention heads reduce the cost of attention computation and KV cache usage, helping keep overall inference cost balanced.
    • Together, the two choices reflect a design philosophy of adding capacity while subtracting computation.

    For developers, the ratio of activated parameters to total parameters—the sparsity level—is worth monitoring during model selection. It reflects actual inference cost more directly than total parameter count alone.

    3. Training: Optimizers and Token Efficiency

    3.1 Exploration of New Optimizers

    Adam-family optimizers face memory overhead and convergence-efficiency bottlenecks at trillion-parameter scale. Some teams have begun adopting optimizers that update weights using spectral methods, such as approaches that exploit the orthogonality of weight matrices to accelerate convergence. These methods show potential for training stability and convergence speed. This points to an important innovation direction at the training-infrastructure layer: optimizer improvements can often extract more effective compute from the same compute budget.

    3.2 Token Efficiency Becomes a New Bottleneck

    The supply of high-quality human-generated corpus is approaching its growth ceiling, and the industry consensus is shifting from piling up more data to improving efficiency:

    • Data mixing and cleaning: Multi-stage deduplication, quality scoring, and domain-mixing strategies help each available token deliver more value.
    • Curriculum-style training: Progressively ordering data by difficulty can improve learning efficiency.
    • Synthetic data and annealing strategies: Introducing high-quality synthetic corpora late in training can target reasoning, coding, and other capabilities.

    A typical trillion-parameter MoE model is pretrained on the order of 10T to 20T tokens. That number is already close to the limits of publicly available high-quality corpora, and further scaling must rely on innovation in data engineering.

    4. Post-Training and Capability Differentiation

    Once pretraining architectures converge, differences in model behavior mainly come from post-training:

    1. Strengthening agent capabilities: Fine-tuning and reinforcement learning on large-scale tool-call trajectory data can equip models with multi-step planning, function calling, and self-correction. This is a key investment area for many teams.
    2. Long context and reasoning chains: Extending chain-of-thought, using process reward models, and similar methods improve mathematical and logical performance.
    3. Alignment strategy differences: The quality and scale of human preference data, as well as the mix of RLHF and RLAIF, directly shape real-world interaction quality.

    5. Selection Considerations for Developers

    Faced with a model ecosystem where core technologies are highly similar, developers can evaluate options along the following dimensions:

    • Task fit: For agent-style tasks, focus on tool-calling benchmarks; for long-document tasks, focus on effective context length and recall.
    • Deployment cost: Sparsity and KV cache compression directly affect per-token inference cost.
    • Open-source ecosystem and licensing: Degree of weight openness, context-length limits, and commercial-use terms.
    • Engineering fit: How readily the model can be deployed efficiently through inference frameworks such as vLLM and SGLang (referring to the general category of open-source inference engines) on cloud container platforms and GPU resource pools, as well as the convenience of integration with cloud infrastructure such as object storage and monitoring/alerting.

    Conclusion

    Chinese large language models are moving from architecture following to targeted innovation. Attention mechanisms, expert scale, optimizers, and data engineering are the four main battlegrounds for differentiation. For technical teams, rather than chasing parameter-count milestones, it is more useful to understand foundational design logic such as sparse MoE, MLA, and token efficiency. These are the key measures for evaluating a model’s capability boundaries and cost model.