Technical Convergence in Chinese LLMs: Design Commonalities Through the MoE Architecture

    This article analyzes why many leading Chinese large language models share a common technical foundation built on Mixture-of-Experts Transformers. It explains the role of MoE and MLA in training and inference efficiency, examines differentiated trade-offs in expert count, attention heads, optimizers, and synthetic data, and shows how technical reports can reveal hidden architectural inheritance. Developers will learn how to interpret architecture choices and evaluate model costs more rationally.

    Technical Convergence in Chinese LLMs: Design Commonalities Through the MoE Architecture

    1. The Conclusion First: High Convergence in Technical Routes

    From a pure model-architecture perspective, the mainstream models released by several leading Chinese LLM companies are highly similar at the underlying design level. Most can be traced back to the same technical paradigm: large-scale Transformers based on Mixture-of-Experts (MoE), whose design blueprint is closely related to the efficient sparse architectures widely discussed in the open-source community. This is not a coincidence but an inevitable result of technical evolution. Once an open-source architecture has been proven to achieve a good balance among training efficiency, inference cost, and final performance, incremental optimization on top of it is the most rational engineering choice for later entrants.

    Therefore, the right way to answer “What is the core advantage of each model?” is not to look for a secret technique unique to one company. Instead, it is to understand which detailed trade-offs each company makes on the same architectural foundation to form its own characteristics.

    2. The Common Foundation: Why MoE Became Mainstream

    2.1 Basic Principles of MoE

    In a traditional dense model, every token must pass through all parameters of the model for computation. MoE architecture replaces the feed-forward network layer with multiple “experts,” and a router dynamically selects a small number of experts for each token to participate in computation. This brings a key property:

    • The total parameter count can be very large (trillion-level), providing massive knowledge capacity.
    • The activated parameter count remains relatively small (tens-of-billions level), controlling the compute cost of a single inference pass.

    A typical configuration is: total parameters around the 1T level, about 32B activated per inference, hidden dimension around 7168, and expert hidden dimension around 2048. This set of numbers has become a “standard recipe” for the current generation of Chinese open-source flagship models.

    2.2 Evolution of Attention: MLA

    In addition to MoE, these models commonly adopt Multi-head Latent Attention (MLA) as the attention implementation. MLA performs low-rank compression on Key and Value, greatly reducing the GPU memory footprint of the KV Cache. This is critical for long-context inference and cost control in large-scale deployment. It can be said that the combination of MoE + MLA is the current “public foundation” of Chinese large models.

    3. Differentiated Trade-off 1: Balancing Sparsity and Attention Heads

    Take the latest generation trillion-parameter model from one company as an example. Its technical report candidly explains its inheritance from the common foundation and makes targeted adjustments in two dimensions:

    1. The number of experts increased from 256 to 384. According to Scaling Law analysis, continued improvements in sparsity can bring significant performance gains. More experts mean finer-grained knowledge specialization, and each expert can become more “specialized.”
    2. The number of attention heads decreased from 128 to 64. This is a reverse move: fewer attention heads reduce computation and memory overhead during inference. The team evidently judged that in long-text scenarios, the benefit of lower attention overhead outweighs the modeling-capacity loss caused by fewer heads.
    1. Example of architectural trade-offs (illustrative):
    2. Total parameters: 1T (same order of magnitude)
    3. Activated parameters: 32B (same order of magnitude)
    4. Experts: 384 ↑ (higher sparsity)
    5. Attention heads: 64 ↓ (lower inference overhead)

    This “one increase and one decrease” reflects a typical engineering mindset: add capacity where it matters, and subtract cost where inference overhead matters.

    4. Differentiated Trade-off 2: Optimizer and Training Stability Innovations

    At the pretraining level, the most recognizable difference among companies lies in optimizer selection. Traditional large-model training generally uses Adam-based optimizers, but some recent models have shifted to Muon-style optimizers. These use orthogonalized updates for matrix parameters and perform better in token efficiency. This means the same amount of data can yield a stronger model. As high-quality human data becomes increasingly scarce, token efficiency is becoming a key factor in model scaling.

    However, new optimizers also introduce training instability, especially the risk of training collapse caused by overly large logits in attention layers. A common industry solution is to introduce targeted constraint techniques, such as dynamically clipping the update magnitude of specific parameters, to preserve the optimizer’s efficiency advantage while keeping stability within bounds.

    In addition, synthetic data generation is now widely introduced into pretraining pipelines. By rewriting, expanding, and restating existing high-quality tokens from multiple perspectives, it amplifies the value of limited data without introducing low-quality external data.

    5. How to Infer “Implicit Inheritance”: Comparison Clues in Technical Reports

    One methodological detail is worth noting: not every vendor explicitly states the architectural source in its technical report. However, we can infer it from the comparison experiment settings in the report. If a model’s evaluation baselines frequently include a certain public architecture, and multiple hyperparameters—such as hidden dimension, expert granularity, and learning-rate schedule—closely match it, then the probability of an inheritance relationship is very high. This method of “reverse-engineering the technical lineage from ablation experiments” is a practical technique for reading large-model technical reports.

    6. Implications for Developers

    1. Do not be misled by surface differences among models. For models built on the same foundation, capability differences come more from data recipes, post-training (SFT/RLHF), and engineering tuning than from the pretraining architecture itself.
    2. On the deployment side, focus on activated parameters and attention implementation. In MoE models, total parameters affect memory, activated parameters affect computation, and MLA affects KV Cache. These three determine your inference cost model.
    3. Technical reports are first-hand material. Learning to read architecture parameter comparison tables, Scaling Law analyses, and ablation experiments is more useful for judging a model’s true level than reading marketing materials.

    In summary, leading Chinese large models are “siblings from the same school” at the architecture level. The real core competitiveness differences settle in training strategies, data engineering, and post-training alignment—areas that are not easily made public. Understanding this is essential for making rational technical choices.