Architectural Convergence in Chinese LLMs: MoE, MLA, and Efficient Training

    Leading Chinese large language models are converging on a shared design: Mixture of Experts (MoE) Transformers, Multi-head Latent Attention (MLA) to compress the KV cache, and pretraining on more than 10T tokens. This article explains why these choices have become common, how expert routing and MLA work, and what the trend means for training methods, inference cost, and developers choosing a model.

    Architectural Convergence in Chinese LLMs: MoE, MLA, and Efficient Training

    1. The Bottom Line: Architectural Differences Are Narrowing

    From a purely technical standpoint, one clear trend stands out among several leading Chinese large language models: their core architectures are becoming very similar. Most use a Transformer with a Mixture of Experts (MoE) structure. They use latent attention to compress the KV cache, keep activated parameters at about 3% of total parameters, and train on pretraining corpora that typically exceed 10T tokens.

    This convergence is not accidental. It reflects engineering practice and Scaling Law research pointing in the same direction. Under real constraints, including limited compute and sensitivity to inference cost, certain designs have repeatedly proven to be locally optimal. Over time they have become industry consensus. Understanding these shared designs is more useful than focusing on small differences in parameter settings. This article covers architecture, training, and inference in turn.

    2. MoE: Trading Sparse Activation for Greater Capacity

    2.1 Why MoE Has Become the Default

    Dense models face a basic limit: model capacity and per-token inference cost are tightly linked. Doubling the parameters roughly doubles the compute. MoE separates these two factors through expert routing. Each forward pass activates only a small number of experts. Total parameters can reach the trillion scale (1T or more), while the compute per inference step corresponds to roughly 30B activated parameters.

    A typical configuration looks like this:

    • Total parameters: around 1T;
    • Activated parameters: around 30B to 40B;
    • Model hidden dimension: around 7168;
    • Expert hidden dimension: around 2048;
    • Experts per layer: 256 to 384, with about 8 experts selected for each token.

    2.2 Why the Number of Experts Keeps Growing

    Research on sparsity and Scaling Laws shows that, at a fixed total compute budget, increasing sparsity by adding more experts while keeping the number of activated experts constant produces consistent performance gains. This explains why newer models have raised the expert count from 256 to 384, pushing sparsity higher.

    More experts also create an engineering problem: load balancing. If routing is uneven, a few popular experts become throughput bottlenecks. Common responses in the industry include:

    • Auxiliary-loss-free bias adjustment: adding a learnable bias to the routing logits and adjusting it during training. This avoids the interference that traditional load-balancing losses can cause on the main training objective;
    • Sequence-level balance constraints: applying balancing pressure across a sequence instead of individual tokens. This is coarser but more stable;
    • Node-limited routing: restricting the devices an expert selection can span during inference, which reduces cross-machine communication overhead.

    3. MLA: Reducing Attention Cost

    Multi-head Latent Attention (MLA) is an attention scheme that many Chinese models now use. Standard multi-head attention caches a full Key and Value vector for each head. MLA instead compresses the hidden state into a low-dimensional latent vector. During inference, only this latent vector needs to be cached. The per-head K and V vectors are then reconstructed through projection matrices.

    Compared with standard MHA and GQA, MLA has three main advantages:

    1. Much smaller KV cache: in long-context settings, memory use can drop by roughly an order of magnitude, which raises the number of concurrent requests a single GPU can serve;
    2. Minimal performance loss: low-rank decomposition preserves expressive capacity, and measured results are roughly on par with MHA;
    3. Natural fit with MoE: MLA optimizes the attention path and MoE optimizes the feed-forward path, so their gains can stack.

    One detail is worth noting: the number of attention heads. One model reduced the head count from 128 to 64. The reasoning is that attention compute during inference scales with the number of heads, and under MLA’s low-rank structure, fewer heads have limited effect on expressive power. This is a typical engineering trade-off. Cutting capacity where returns are diminishing buys real reductions in inference cost.

    4. Pretraining: Upgrades in Data Efficiency and Optimizers

    4.1 Token Efficiency Becomes the New Competitive Focus

    Leading models now pretrain on corpora in the range of 15T tokens. However, the growth of high-quality human-generated data has slowed, and the so-called data wall is becoming a practical problem. The industry is shifting from simply adding more tokens to getting more learning out of each token. Key methods include:

    • More careful data cleaning and deduplication, removing content with low information density;
    • Curriculum-style data mixing, which raises the share of math, code, and reasoning data in the later stages of pretraining;
    • Compliant use of synthetic data, such as generated reasoning traces, to supplement scarce domains.

    4.2 Practice with New Optimizers

    Traditional AdamW carries significant memory overhead at the trillion-parameter scale, since it maintains first and second moment estimates. Some recent models use second-order optimizers based on matrix orthogonalization, such as Muon-style methods. These optimizers have the following properties:

    • They approximately orthogonalize the gradients of two-dimensional weight matrices, making update directions more regular;
    • Compared with Adam, they achieve higher token efficiency, so loss decreases faster for the same amount of data;
    • They require less memory for momentum states, which helps with very large model training.

    These optimizers are usually used together with Adam. Embeddings, routers, and other non-matrix parameters still use Adam. This layered approach is a pragmatic way to optimize.

    4.3 Engineering for Training Stability

    Interruptions are very costly when training trillion-parameter MoE models. Common stability measures include clipping router logits numerically, monitoring gradient norms for experts, and saving fine-grained fault-tolerant checkpoints, typically every hour, so training can roll back quickly. Teams also combine pipeline, expert, and data parallelism. In cloud training environments, teams often pair a container platform with monitoring and alerting to build self-healing jobs. When hardware fails, these jobs restart automatically and resume from the last saved progress.

    5. What This Means for Developers

    If architecture is converging, it is no longer the deciding factor when choosing a model. The real differences show up in the following areas:

    1. Post-training quality: the craft of instruction tuning and reinforcement learning determines actual performance in conversation, coding, and reasoning;
    2. Context and cost: KV cache efficiency directly affects the price of serving long-context workloads;
    3. Ecosystem and engineering support: inference framework compatibility, quantization support, and API stability determine how well a model works in production.

    Teams that want to deploy privately should start from an open-source MoE checkpoint rather than build a foundation model from scratch. Cloud object storage can hold the model weights, a container platform can host the inference service, and logging and monitoring services can keep production stable. Together these form a complete pipeline from deployment to observability.

    6. Summary

    Architectural convergence among Chinese large language models reflects a practical response to shared constraints on compute, cost, and the data wall. MoE sparsity, MLA attention compression, and high-token-efficiency training form the current core of mainstream design. Understanding this pattern matters more than memorizing the specific parameters of any one model, because the next generation of models will most likely continue to iterate within this framework.