How China's Large Open-Source Models Compete Within a Shared MoE Design
China's large-parameter open-source models largely share a Transformer-based sparse mixture-of-experts design. This article explains that common architecture, then compares how teams differ in hyperparameters, training methods, and capability focus, and offers a practical way to choose a model.
1. The conclusion first: differentiation inside a shared architecture
From a purely technical perspective, China’s mainstream large-parameter open-source models look highly similar at the architecture level. Nearly all of them follow the same paradigm: a Transformer-based sparse mixture-of-experts (MoE) model. A research team first demonstrated that this approach was practical at large scale in an open model. It then quickly became an industry default—essentially different derived implementations of the same base architecture.
That does not mean the models are identical. The real differences show up in three places:
- Architecture trade-offs: expert count, active parameters, attention-head count, and other hyperparameters.
- Training innovations: optimizers, stability techniques, and synthetic-data strategies.
- Capability emphasis: how much post-training is invested in reasoning, code, and agent tool use.
The sections below unpack those layers.
2. Why MoE became the default
Before comparing models, it helps to recall the idea behind MoE: sparse activation. Total parameter count can be very large, but each token activates only a small fraction of those parameters during the forward pass.
Total parameters = attention-layer parameters + expert-layer parameters (N experts)Active parameters per token = attention-layer parameters + a small number of routed expert parameters
A common industry configuration puts total parameters at the trillion scale (about 1T) while activating only around 30B parameters per token. The benefits are:
- Higher training efficiency: compute scales with active parameters, not total parameters.
- Lower inference cost: with expert parallelism, expert offloading, and similar optimizations, a large-parameter model can run even on a single accelerator.
- A higher capacity ceiling: the number of experts sets an upper bound on knowledge capacity. Scaling-law analysis also suggests that, under a fixed compute budget, increasing sparsity can keep improving performance.
Key components include:
- MLA (Multi-head Latent Attention): low-rank compression of the KV cache sharply reduces memory use for long-context inference. It is now a standard attention design in large sparse MoE models.
- Fine-grained experts plus shared experts: experts are split more finely, while a small set of shared experts handles general knowledge, which makes routing more flexible.
- Auxiliary-loss-free load balancing: a traditional load-balancing auxiliary loss can interfere with the main objective. Newer methods adjust expert load with dynamic bias so balance and performance can be maintained together.
3. Hyperparameter choices and how teams trade them off
Even under the same paradigm, teams make different engineering judgments about hyperparameters. Those choices are themselves a form of differentiation.
Expert count and sparsity. Some teams raise the expert count from a common 256 to 384. The rationale is that higher sparsity can keep delivering gains: the model can expand knowledge capacity without meaningfully increasing per-token compute. The cost is engineering complexity. More experts make routing and parallelism harder and place stricter demands on training infrastructure.
Attention-head count. Some teams cut the number of attention heads in half, for example from 128 to 64. Fewer heads directly reduce attention compute at inference time, while the hidden dimension stays the same (such as 7168) and the expert hidden size remains on the order of 2000. This “fewer heads, same width” design keeps representational capacity while shrinking the most expensive part of inference.
Tiered model sizes. Leading model families commonly ship a flagship and a lighter variant. The flagship may have hundreds of billions of total parameters and tens of billions active; the lighter model roughly halves total parameters and scales active parameters down as well. The lighter version lets teams with limited hardware deploy and fine-tune on a single multi-GPU server, which lowers the barrier to using the open ecosystem.
4. Training techniques: where the real split appears
If the architectures are close relatives, training is where the differences become clearer.
4.1 Optimizer choice
Large-model training has long relied on AdamW-style optimizers, whose state is memory-heavy because each parameter keeps first- and second-order moments. Some teams have tried Muon-style optimizers, which update weights with spectral methods, at trillion-parameter MoE scale. The main advantages are:
- More efficient parameter updates and clearly better token efficiency during pretraining.
- Lower optimizer-state memory, which matters at very large parameter counts.
Muon-style optimizers can be unstable on extremely large models. A typical failure is that query-key norms in attention layers keep growing until training collapses. A common countermeasure is a QK-Clip-style mechanism: monitor query and key norms in each layer, and when they exceed a threshold, scale and clip the corresponding projection weights. That preserves convergence speed while keeping long training runs stable. It is a typical pairing of a new optimizer with a stability patch.
4.2 Data strategy: token efficiency as the core metric
High-quality human data is increasingly hard to obtain. That is an industry consensus. Compute can be bought; data cannot be expanded without limit. Token efficiency—how much capability is extracted per unit of data—is therefore becoming a key scaling factor. Common practices include:
- Synthetic data: using a strong model to rewrite, expand, and add reasoning chains to existing text so each token carries more information.
- Curriculum-style mixing: changing the share of code, math, multilingual text, and other data across training stages.
- Long-context extension: a late pretraining stage that lengthens sequences, often with RoPE scaling and related techniques.
4.3 Post-training and capability alignment
Pretraining sets the ceiling. Post-training decides where capability actually lands. Leading open models have recently spent heavily on post-training:
- Scaled reinforcement learning: large RL runs driven by verifiable rewards in math and code, which substantially improve reasoning.
- Dedicated agent training: multi-turn tool use, code execution, web browsing, and other interactive data so the model learns to plan and correct itself.
- General agent-data synthesis: automatically generating tens of thousands of multi-turn trajectories across tools and environments.
This is why models with similar architectures still show different strengths in evaluations: some lean toward code generation, some toward mathematical reasoning, and some toward agent tool orchestration.
5. A method for inferring technical lineage
Not every team states its architectural source in a technical report. A reasonable inference can still be made by comparison:
- Extract the published hyperparameters: hidden size, expert count, active parameters, attention-head count, and MoE topology.
- Compare them item by item with known public configurations of the same paradigm.
- Look at how training is described—whether it mentions auxiliary-loss-free balancing, multi-token prediction (MTP), and similar techniques.
- If the parameters match closely and the paper actively compares itself in detail with other models from the same family of designs, the architecture most likely comes from that same technical paradigm.
This kind of “parameter fingerprint” comparison is a useful way to map the technical lineage of open models.
6. How to choose: guidance for developers and enterprises
For a real deployment decision, drop the single story of “which model is strongest” and match the model to the job:
- Agents and tool use: look at multi-turn interaction and function-call format compliance, and at how much of the context window is actually usable.
- Reasoning and math: look at scores on verifiable tasks, and at whether a thinking mode can be switched on and budgeted.
- Private deployment: prefer families that offer a lighter variant, and estimate inference cost from active parameters. For MoE, plan on expert parallelism or offloading.
- Fine-tuning cost: full-parameter fine-tuning of a sparse MoE has a higher memory floor than a dense model. With limited resources, a compromise is to fine-tune only the shared experts and attention layers.
In the cloud, mature inference platforms can host open models, object storage can hold weights and data, and monitoring and alerting can keep the service stable. Those general cloud capabilities form the engineering base for LLM applications.
7. Summary
Competition among China’s large-parameter open-source models has moved past the question of whether to use MoE. It is now fine-grained engineering inside one architectural paradigm. The four main fronts are sparsity and expert count, adoption of new optimizers, pushing token efficiency, and specialized agent training. For developers, understanding these details is more valuable over time than chasing leaderboard rank. It is what determines sound decisions on deployment, fine-tuning, and cost.
