LLM Architecture Tuning and Training Strategies: A Technical View of Chinese Large Models
This article examines why leading large models often converge on similar architectures while still differing in engineering decisions. It covers MoE capacity and sparsity tradeoffs, attention design for long context, optimizer stability, data efficiency, enterprise model selection, and the shift toward system-level capabilities. Readers gain a reusable technical framework for evaluating models beyond parameter counts.
Introduction
When reading recent technical reports on mainstream large models, one interesting pattern emerges: many models are often in the same range for parameter count, architecture choice, and training data scale. Many people therefore conclude that they are “homogeneous.” From an engineering perspective, however, this convergence looks more like a technical consensus: the combination of Transformer and Mixture-of-Experts (MoE) architectures has been validated at scale and has become a base that is cost-controllable and performance-predictable. The real differences lie in finer engineering decisions: the ratio of expert count to activated parameters, attention head allocation, the impact of optimizers on memory and stability, and how data corpora are organized.
This article does not evaluate specific products. It breaks down these differences from a technical perspective to help developers and enterprise users build a reusable set of evaluation criteria.
1. Shared Origins, Heterogeneous Designs: Why Leading Models Converge
1. Building a base model from scratch is extremely costly
Training an ultra-large base model is not just a matter of compute investment. Data collection and cleaning, training framework adaptation, parallel strategy tuning, stability assurance during long training cycles, and reproducibility verification all require substantial engineering experience. As a result, more teams choose to continue building on mature base models or open frameworks, investing limited resources in data, alignment, inference optimization, and other areas where differentiation is easier to achieve.
2. Evaluation systems drive technical convergence
When the industry widely adopts similar public evaluation sets, model optimization objectives during pretraining gradually converge. Developers tend to choose validated mainstream architectures rather than explore a completely different paradigm from scratch. This further pushes architecture design toward standardization.
3. Shared origins do not mean homogeneity
Even when models share the same skeleton, different parameter ratios produce completely different capability structures. For example:
- More experts increase model capacity, but routing and communication become more complex.
- Fewer attention heads reduce inference latency, but multi-subspace representational capability may be limited.
- Adjusting hidden dimensions directly affects memory usage and FFN computation.
- Different optimizers and data mixes lead to different performance in long-text, mathematical reasoning, and code capabilities.
These micro-level differences are amplified at trillion-parameter scale, ultimately creating a situation where models “look similar but actually make different tradeoffs.”
2. MoE Architecture: Balancing Capacity, Sparsity, and Inference Cost
The core idea of MoE is not to assign the entire model to all tokens, but to use a gating network to select a small number of experts for each token. In this way, the total parameter count can be very large, but each token only passes through a subset of parameters, and inference computation mainly depends on activated parameters rather than total parameters.
The following pseudocode shows the basic forward process of MoE:
def sparse_moe_forward(x, experts, router, k=4):scores = router(x) # Scores of each token for each experttop_k_scores, top_k_indices = topk(scores, k)weights = softmax(top_k_scores)output = 0for i, expert in enumerate(experts):mask = (top_k_indices == i)output += mask * expert(x) * weights[..., i]return output
In practical design, several key tradeoffs deserve attention.
1. More experts is not always better
Increasing the number of experts can enhance the model’s ability to split knowledge at a finer granularity, but it also brings load imbalance, routing conflicts, higher memory usage, and cross-device communication overhead. Leading models usually set the number of experts to several hundred and continue to increase sparsity, but the specific value needs repeated experimentation based on compute scale, parallel strategy, and inference performance.
2. Ratio of total parameters to activated parameters
A model with 1T-level total parameters and 30B-level activated parameters only activates about 3% of parameters during inference, but the total parameters still occupy a large amount of storage. During deployment, it is not enough to look only at “fewer activated parameters”; you also need to check whether memory can hold the model, whether weight loading time is acceptable, and whether reasonable quantization and compression are supported.
3. Routing load balancing
Gating networks can easily concentrate high-frequency tokens on a small number of experts, causing some experts to be overloaded while others sit idle. During training, auxiliary losses are usually added to encourage experts to receive tokens evenly, or dynamic expert parallelism is used to balance load as much as possible.
4. Shared experts and independent experts
Some models keep a small number of shared experts in MoE so that every token can pass through them, preserving general capabilities, while using many independent experts to expand long-tail knowledge. This design can be understood as a combination of “general capability + domain capability,” balancing stability and capacity.
3. Attention Mechanisms: Capacity and Efficiency Tradeoffs in Long-Context Scenarios
The upper bound of large model capability is largely determined by attention mechanisms. Traditional multi-head attention needs to store K/V caches for each token, and memory overhead rises rapidly as context length increases. To solve this problem, current mainstream approaches mainly fall into two categories.
1. Reducing the number of attention heads
Fewer heads directly reduce attention computation, but may weaken the model’s ability to obtain information from different subspaces. To compensate, the dimension of each head is usually increased, or the hidden layer structure is adjusted so that a single head can take on richer semantic roles. In some designs, fewer attention heads are combined with more experts to achieve a balance that “trades expert capacity for attention overhead.”
2. Attention mechanisms with low-rank projection
This approach first projects high-dimensional Key/Value into a low-rank space before participating in attention computation, thereby reducing KV cache usage. It is especially valuable in long-context scenarios, but it is sensitive to the choice of low-rank dimension and requires balancing representational capability and compression rate. If compression is excessive, the model may lose detailed information; if compression is insufficient, memory savings are limited.
In specific models, we often see combinations of “more experts + fewer attention heads.” This is essentially a system-level cost control strategy, not simple parameter stacking.
4. Optimizers and Training Stability: Hidden Factors That Determine Training Success
During pretraining, optimizer choice profoundly affects model convergence quality, training stability, and final compute cost.
Common adaptive moment estimation optimizers need to store a large number of optimizer states, resulting in high memory usage. In large-scale training, this often needs to be optimized through sharding, low-precision storage, and other methods. However, some teams have begun exploring lightweight momentum-based optimizers, which use less state and theoretically improve token utilization efficiency.
That said, lightweight optimizers are prone to oscillation in ultra-large-scale training, especially in the Query/Key scores of attention modules. Once the scores become too large, the probability distribution after softmax becomes sharp, leading to gradient instability. Common countermeasures include:
- Constraining or clipping the range of Query and Key dot-product results to limit extreme logits.
- Using gradient clipping, loss scaling, and learning rate warmup and decay together.
- Monitoring loss curves, activation variance, and gradient norms in real time during training, with alerts or rollback when anomalies occur.
Although these stability measures do not directly improve model scores on evaluation sets, they are the “invisible infrastructure” that enables a model to complete ultra-large training tasks stably.
5. Data Efficiency: From “More Tokens” to “Better Tokens”
As high-quality human corpora are gradually “used up,” simply increasing training data volume is no longer sustainable. Many teams have begun to focus on token efficiency, that is, making each token contribute more effective information to the model.
Specific methods include:
- Synthetic data: using existing models to generate supplementary corpora, then filtering high-quality samples through rules or automatic evaluation for training or post-training.
- Data rewriting and augmentation: rewriting, denoising, and structuring original corpora to increase information density.
- Curriculum learning: arranging training order by difficulty or domain so that the model first learns general capabilities and then strengthens specialized domains.
- Instruction replay and mixing ratios: mixing different task data during post-training to preserve general capabilities without forgetting.
However, synthetic data is not risk-free. Repeatedly consuming a model’s own outputs may lead to reduced diversity, amplified errors, and even model collapse. Therefore, it is necessary to strictly control the proportion of synthetic data, maintain a balance between real and synthetic corpora, and perform quality filtering and diversity validation.
6. Enterprise Selection: Do Not Use Total Parameter Count as the Only Metric
For developers and enterprise users facing multiple models with similar technical routes, evaluation dimensions should be more practical.
1. Consider both activated and total parameters
Activated parameter count determines the approximate latency of online inference, while total parameter count affects storage and loading costs. The two need to be evaluated together with hardware specifications; it is not enough to look only at promotional claims of “total parameters exceeding one trillion.”
2. Match context length with real usage scenarios
If an application needs to process long documents, code repositories, or complex logs, pay attention to how the attention mechanism supports long context and how memory usage grows. Longer context is not always better; the key is whether it can be used stably at controllable cost.
3. Fine-tuning and secondary development costs
Whether the model supports efficient fine-tuning, quantization, and low-rank adaptation; whether it is compatible with existing business processes; and whether it has a complete evaluation toolchain all affect the actual deployment cycle.
4. Inference deployment ecosystem
Whether model services can be easily integrated with object storage, message queues, container platforms, logging services, monitoring and alerting, and other infrastructure, and whether they provide elastic scaling, rate limiting, auditing, and security isolation, are also important considerations for enterprise selection.
5. Result controllability
Enterprise business often requires stable and traceable outputs. Prompt engineering, knowledge base retrieval, content safety review, and other methods need to be combined to ensure generation quality.
Technical selection should not rely only on market hype. It should be based on small-scale validation using your own training data, deployment environment, and business metrics before expanding investment.
7. Future Trends: The Center of Competition Is Shifting from the Model Itself to System Capabilities
As base architectures gradually converge, model capability gaps will increasingly depend on data engineering, post-training, inference engines, and productization capabilities. Expected directions include:
- Edge and small models: while maintaining capability, use distillation, quantization, and sparsification to deploy models on more devices.
- Multimodal fusion: unify modeling of text, images, audio, and video so that models have stronger cross-modal understanding.
- Reinforcement learning and tool use: models no longer only generate text, but learn to call external tools, execute complex tasks, and perform self-verification.
- Continual learning and data flywheels: continuously collect feedback data after deployment so that models can improve while protecting user privacy.
For developers, this means paying attention not only to the model itself, but also to data pipelines, evaluation systems, deployment and operations, and other “capabilities beyond the model.” True technical advantage is not a product name brought by a single model release, but an engineering system spanning the entire chain of data, training, deployment, and feedback.
Conclusion
The technical routes of current mainstream Chinese large models are indeed converging, but this does not mean there is no differentiation. Differentiation appears in architecture parameter tradeoffs, optimizer choices, data efficiency exploration, and engineering stability. Understanding these details helps developers see the real gaps beyond “parameter count” and enables enterprises to judge cost and benefit more rationally during model selection.
