Diverging Technical Routes of Chinese LLMs: An Analysis of Core Architectures

1. From Following to Diverging: A Technological Turning Point

Over the past few years, the development trajectory of large language models (LLMs) in China has experienced a significant shift. In the early stages, most teams relied on a universal architecture centered on decoder stacking, largely following the public roadmaps of leading overseas laboratories regarding training paradigms and alignment strategies. However, starting around 2023, a clear trend emerged: leading teams stopped converging on a single set of technical solutions. Instead, based on their own assessments of application scenarios, they chose distinctly different architectural directions.

This divergence is not accidental. It is driven by three combined forces: first, the differentiated demands from the application side regarding inference costs, response latency, and context length; second, the varying accumulation paths of engineering teams in distributed training and inference optimization; and third, data-level factors—specifically the structural characteristics, knowledge density distribution, and multimodal content richness of Chinese corpora—which forced targeted adjustments in model design.

The following sections analyze three representative technical routes: the sparse activation Mixture of Experts (MoE) architecture, lightweight design for long contexts, and the fusion approach of multimodality and knowledge enhancement. Each route reflects a different design philosophy and engineering trade-off.

2. Sparse Mixture of Experts Architecture: Trading Efficiency for Scale

2.1 Core Concept

The underlying logic of the Mixture of Experts (MoE) architecture can be summarized in one sentence: using a larger total parameter count for stronger expressive power, while using sparse activation to control the computational cost of a single inference.

Traditional dense models activate all parameters during every forward pass. When the parameter scale grows to hundreds of billions, inference costs increase linearly, placing immense pressure on server-side deployment. MoE addresses this by splitting the model into several “expert” subnetworks. Each input token is routed to only a few of these experts for computation.

2.2 Key Design of the Gating Mechanism

The core component of MoE is the Gating Network, which determines which experts each token should be assigned to. A typical gating computation process is as follows:

import torch
import torch.nn.functional as F
class TopKGating(torch.nn.Module):
    def __init__(self, num_experts, top_k=2):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k
        # Gating weight matrix
        self.gate = torch.nn.Linear(hidden_dim, num_experts)
    def forward(self, x):
        # Calculate routing scores for each expert
        logits = self.gate(x)  # [batch, seq_len, num_experts]
        # Temperature coefficient adjusts distribution sharpness
        logits = logits / temperature
        # Select top-k experts
        topk_logits, topk_indices = torch.topk(logits, self.top_k, dim=-1)
        # Softmax normalization for routing weights
        topk_weights = F.softmax(topk_logits, dim=-1)
        return topk_weights, topk_indices

A notable engineering detail here is that the adjustment of the temperature coefficient directly determines the concentration of expert activation. At higher temperatures, the softmax output trends toward uniformity, activating more experts. This maximizes model capacity utilization but increases computational overhead. At lower temperatures, the distribution becomes sharp, selecting only the top-ranked experts. This improves inference efficiency but may sacrifice some expressive power. In actual deployment, this coefficient often requires dynamic adjustment based on the task type.

2.3 Engineering Challenges and Trade-offs

The engineering challenges faced by the MoE architecture primarily focus on three areas:

• Load Balancing: If the gating network repeatedly favors the same group of experts, it leads to overload on some experts while others remain undertrained. A common practice is introducing an auxiliary loss function to penalize unbalanced expert utilization.
• Communication Overhead: In distributed training, experts are dispersed across different devices. Token routing implies cross-device communication. As the number of experts increases, all-to-all communication can become a bottleneck.
• Inference Memory: Although only a fraction of parameters is activated per inference, all expert weights must still be loaded into memory. This imposes high demands on memory capacity, requiring mitigation through techniques like expert offloading and quantization.

This route is suitable for scenarios that are sensitive to inference throughput but can accommodate higher memory usage in server-side deployments, such as batch API calls or offline content generation.

3. Lightweight Design for Long Contexts

3.1 Real-World Demand for Long Contexts

Long-context capability is not merely a “parameter race.” In practical applications, it directly impacts the feasibility of several core scenarios:

• Long Document Understanding: Complete analysis of legal contracts, technical manuals, and academic papers often requires context windows of tens of thousands of tokens.
• Multi-turn Dialogue Consistency: In complex tasks, the model needs to remember key constraints from dozens of previous dialogue turns.
• Repository-level Code Understanding: Cross-file code completion and refactoring suggestions require the model to process multiple source files simultaneously.

3.2 Combining Lightweight Design with Long Contexts

A common misconception is that long contexts inevitably require a massive model footprint. In reality, context length and model parameter count are not strictly positively correlated. Through the following technical means, relatively long context support can be achieved at a medium parameter scale:

Extrapolation and Interpolation of Positional Encoding. Rotary Position Embedding (RoPE) and its variants allow models to perform a certain degree of length extrapolation beyond the training length by adjusting the base frequency parameters. Linear interpolation and NTK-aware scaling are two common strategies:

def adjust_rope_frequency(base_freq, scale_factor):
    """
    Adjust RoPE base frequency to support longer contexts.
    base_freq: Original base frequency, typically 10000.
    scale_factor: Expansion ratio.
    """
    return base_freq * (scale_factor ** (dim / (dim - 2)))
# Example: Expanding context window from 4K to 32K
new_base = adjust_rope_frequency(10000, 8.0)

Sparsification of Attention Mechanisms. The computational complexity of standard self-attention is O(n²), causing overhead to rise sharply as sequence length grows. Solutions like sliding window attention, dilated attention, and content-based routing attention can reduce computation while preserving the ability to model long-range dependencies.

Quantization and Compression of KV Cache. During inference, the Key-Value (KV) cache consumes significant memory. Compressing the KV cache through methods like grouped quantization or low-rank decomposition can significantly reduce the memory pressure of long-context inference.

3.3 Applicable Scenarios

The lightweight long-context approach is suitable for scenarios that require processing longer inputs but have limited computational resources, and are sensitive to the latency of the first token, such as real-time document Q&A or interactive code assistants. Its advantage lies in a lower deployment barrier, capable of running on a single machine with multiple GPUs or even a single GPU.

4. Fusion of Multimodality and Knowledge Enhancement

4.1 Levels of Multimodal Capability

Multimodal capability is not a single dimension but consists of multiple levels:

• Perception Level: Basic capabilities like image captioning, video understanding, and speech transcription.
• Alignment Level: Tasks requiring semantic alignment across modalities, such as cross-modal retrieval, image-text matching, and visual question answering.
• Generation Level: Cross-modal generation tasks, like generating images from text or generating code from images.
• Reasoning Level: Complex cognitive tasks combining visual information for logical reasoning or mathematical problem-solving.

The architectural requirements vary significantly across these levels. Perception-level tasks typically rely on concatenating a pre-trained vision encoder with a language model, whereas reasoning-level tasks demand deeper cross-modal fusion.

4.2 Integrating Knowledge Graphs with Language Models

Introducing structured knowledge into language models is a critical approach to improving factual accuracy and reasoning reliability. Integration methods generally fall into three categories:

Input-side Fusion. Entities and relationships from the knowledge graph are serialized into text and fed into the model as additional context. This is simple to implement but constrained by the context window length.

Intermediate-layer Fusion. A knowledge retrieval module is introduced in the intermediate layers of the model, using attention mechanisms to let the model dynamically attend to relevant knowledge triplets. This requires designing additional training objectives to align knowledge representations with language representations.

Output-side Constraint. Knowledge constraints are introduced during the decoding phase. Through constrained decoding or post-processing verification, outputs are ensured to be consistent with the knowledge base. This is suited for scenarios demanding extremely high factual accuracy.

4.3 Key Issues in Engineering Practice

The fusion of multimodality and knowledge enhancement faces several typical issues during engineering implementation:

• Data Pipeline Complexity: The cost of cleaning, aligning, and annotating multimodal data is much higher than for pure text. Building and maintaining knowledge graphs also requires sustained investment.
• Inference Latency: Introducing vision encoders and knowledge retrieval modules increases the length of the inference chain. Optimization requires techniques like model distillation and cache warming.
• Evaluation Difficulties: Evaluation metrics for multimodal tasks are not yet standardized, and the effects of knowledge enhancement are difficult to measure with a single metric.

This route is suitable for scenarios with explicit demands for multimodal understanding and corresponding data accumulation, such as mixed image-text Q&A in intelligent customer service or problem analysis in the education sector.

5. A Framework for Technology Selection

When evaluating the three technical routes above, developers and enterprises can assess them based on the following dimensions:

Evaluation Dimension	Sparse MoE Architecture	Lightweight Long Context	Multimodal Knowledge Fusion
Inference Cost	Medium (High memory requirement)	Low	High
Long Text Capability	Depends on implementation	Strong	Medium
Multimodal Support	Requires extra extension	Limited	Strong
Deployment Complexity	High	Low	Medium
Suitable Scenarios	High-throughput server-side	Real-time interactive applications	Complex cognitive tasks

It is important to emphasize that these three routes are not mutually exclusive. In actual systems, dynamic routing can be used to direct tasks to different models or expert combinations. For example, within a unified service framework, simple Q&A can use the lightweight path, complex reasoning can use the MoE path, and mixed image-text tasks can use the multimodal path.

6. Conclusion

The technological divergence of LLMs is essentially the natural result of a deep understanding of application scenarios. No single architecture can optimally satisfy all demands—sparse activation pursues a balance between scale and efficiency, lightweight long contexts pursue a balance between deployment flexibility and context depth, and multimodal knowledge fusion pursues a balance between perception breadth and reasoning reliability.

For developers, understanding the design philosophy and engineering constraints behind these technical routes is more valuable than memorizing the parameter scale of a specific model. The core of selection lies not in “which model is the strongest,” but in “which model is best suited for the constraints of the current task.” As application scenarios become further refined, we can expect this technological divergence to continue deepening, which is precisely the fundamental reason this field retains its vitality.