Architectural Convergence in Mainstream Chinese Large Language Models: A Technical Analysis
Technical reports from several leading Chinese large language model teams describe architectures that look strikingly alike, including sparse Mixture of Experts (MoE), Multi-head Latent Attention (MLA), and similar parameter settings. This article breaks down the shared design choices behind this trend, examines where training and post-training methods still differ, and discusses what the convergence means for model selection, evaluation, and deployment costs.
Introduction: A Notable Technical Phenomenon
Over the past two years, several leading Chinese large model teams have released their own technical reports. When the architecture sections of these reports are read side by side, a clear pattern emerges. Although the models differ in style and focus, their underlying architectures are highly consistent, with similar hidden dimensions, similar attention mechanisms, and similar sparse expert structures. Some researchers have half-jokingly called them “half-brothers from the same father.”
This pattern is not accidental. It reflects a common rule in engineering practice: designs that have been proven to work tend to be reused widely. This article breaks down the shared design choices behind this trend and discusses what they mean for model selection, technical evaluation, and the industry ecosystem.
1. The Core of Architectural Convergence: Sparse MoE as the Mainstream Consensus
1.1 Why MoE Has Become a Default Choice
Mixture of Experts (MoE) is built on a simple idea. A model has a very large total number of parameters, but each forward pass activates only a small subset of them. In current mainstream designs, total parameters can reach the trillion (T) scale, while the number of parameters activated for a single inference is in the tens of billions (B).
This design brings two direct benefits:
- Training side: With the same compute budget, the model can have greater capacity and absorb more knowledge.
- Inference side: The compute cost of a single request depends on activated parameters rather than total parameters, so inference costs stay under control.
Scaling law analysis suggests that, under a fixed compute budget, increasing sparsity (adding more experts and lowering the fraction of experts activated per token) yields steady performance gains. This explains why newer models commonly increase the number of experts from 256 to 384 or more.
1.2 Typical Architecture Parameters Compared
Using data disclosed in several recent public technical reports as an example, the convergence in parameters is clear:
| Architecture Dimension | Typical Value |
|---|---|
| Model hidden dimension | 7168 |
| Expert hidden dimension | 2048 |
| Total parameter scale | Approximately 1T |
| Activated parameter scale | Approximately 32B |
| Number of experts | 256 to 384 |
| Number of attention heads | 64 to 128 |
When a model’s hidden dimension, expert dimension, and activation ratio all closely match a well-validated open-source baseline, its architectural lineage can usually be inferred. Inferring technical lineage from parameters is a common method the technical community uses to analyze where models come from.
2. A Unified Choice for Attention: MLA
Multi-head Latent Attention (MLA) is the most representative component of this round of architectural convergence. In traditional multi-head attention (MHA), inference requires caching the complete Key and Value matrices for every token. The KV cache grows linearly with sequence length, which makes GPU memory a bottleneck for long-context inference.
MLA applies low-rank compression to Keys and Values. During inference, only a low-dimensional latent vector needs to be cached. When attention is computed, that vector is projected back into the full attention space. This can cut the memory footprint of the KV cache by about an order of magnitude, with almost no loss of model capability.
Teams still make their own adjustments on top of MLA. For example, one team reduced the number of attention heads from 128 to 64 to lower inference compute, then shifted the saved compute budget to more experts. Trade-offs like this, which reallocate compute under constraints, are the core craft of architecture design.
3. Training Optimization: Where the Real Differences Lie
If architecture has converged, training methodology is where teams begin to separate.
3.1 Optimizer Innovation
Large model training has commonly used the AdamW optimizer, but AdamW consumes a lot of memory and makes limited use of second-order information. Some teams have recently introduced new optimizers based on matrix orthogonalization, such as Muon. These optimizers apply an orthogonality constraint to the update direction of matrix parameters, which aims to improve parameter update efficiency and token utilization.
New optimizers, however, tend to become unstable at large scale. A typical symptom is exploding attention logits. The engineering response is to introduce techniques similar to QK-Clip, which apply norm clipping to the Query and Key projections and keep attention scores within a safe range. The lesson is that any optimizer improvement must be paired with corresponding stability safeguards, because at the trillion-parameter scale, a training crash is extremely costly.
3.2 Token Efficiency Becomes a New Competitive Focus
As access to high-quality human-written data becomes more limited, learning more from a fixed amount of data matters more than simply adding more data. Mainstream approaches include:
- Token-efficient optimization algorithms: Each parameter update extracts more information.
- Synthetic data generation: Models produce high-quality training samples, extracting more value from existing data. For example, existing knowledge can be rewritten, expanded, or reconstructed from multiple perspectives.
- Curriculum-style data mixing: The data distribution changes across pre-training stages, so the model learns general abilities first and specialized abilities later.
3.3 Post-Training and Reasoning Enhancement
As pre-training converges, post-training has become the key stage where each team builds its own user experience. It includes supervised fine-tuning, human preference alignment, and reinforcement learning for reasoning and agent tasks. Technical reports disclose the least about this stage, yet it may have the greatest impact on what end users actually experience.
4. Practical Implications of Architectural Convergence for Developers and Enterprises
4.1 Model Selection Should Shift From Architecture to Capability and Service
Because mainstream models share highly similar underlying architectures, asking which one has the more advanced architecture is no longer a meaningful selection question. Developers and enterprises should focus on:
- Measured performance on specific tasks: Benchmark with real business data rather than relying on public leaderboards.
- Inference cost and latency: Activated parameter count, KV cache efficiency, and quantization support directly determine deployment cost.
- Context length and tool-calling ability: These are core constraints for agent-style applications.
- Service stability and ecosystem support: Consider API availability, rate-limiting policies, and how easily the service integrates with cloud logging, monitoring and alerting, object storage, and other general infrastructure.
4.2 Lower Barriers to Self-Development and Fine-Tuning
Architectural convergence also means the technology stack is becoming standardized. Mature MoE training frameworks, validated MLA implementations, and publicly available training recipes allow capable teams to fine-tune open-source baselines for their own domains instead of training from scratch. For most enterprises, adapting a general-purpose model to a specific domain offers a far better return on investment than building a foundation model in-house.
5. Looking Ahead: Where Differentiation Will Emerge After Convergence
Architectural convergence is a phase of relative stability rather than a permanent state. Differentiation may come from several directions:
- Further evolution of attention mechanisms: Hybrid designs that combine linear attention, sparse attention, and MLA could handle million-token contexts.
- More aggressive sparsification: Expert counts may keep growing, paired with more refined routing algorithms.
- New training paradigms: Closed loops of synthetic data and self-improving training may reshape competition around
