From a Unified Paradigm to Divergent Roadmaps: The Evolution of Chinese LLMs

Since the Transformer architecture was introduced in 2017, “pretraining + scaling” became the unified paradigm for LLM development. In the years that followed, most domestic teams advanced along paths already validated by early movers: similar architectures, comparable training pipelines, and analogous data strategies. During this phase, the core competitive question was “can it be built at all.”

Around 2023, the question shifted—from “can it be built” to “why should it be built this way.” Hard constraints on compute budgets, differentiated requirements from downstream scenarios, and the share of inference cost in business models collectively pushed Chinese LLMs out of the convergence zone and into several clearly bounded technical roadmaps:

• Long-context and code capability route: centered on “read it all, remember it, write it correctly,” serving heavy-duty scenarios such as document analysis and code engineering;
• MoE sparse architecture route: centered on “large total parameters, small activated parameters,” seeking the optimum between model capacity and inference compute;
• Lightweight route: centered on “small models, low cost, edge-deployable,” covering privacy-sensitive and low-latency scenarios;
• Multimodal route: centered on “cross-modal understanding and generation,” extending toward real information forms such as images, documents, and video.

Roadmap divergence is not involution—it is division of labor. Understanding the principled boundaries and engineering costs of each route is a prerequisite for technology selection. The following sections break them down one by one.

Route 1: Long Context—Enabling Models to “Read It All and Remember It”

2.1 Where the Demand Comes From

Information in real business contexts rarely arrives in units of “a few hundred words”: a prospectus can run hundreds of pages, a code repository can contain hundreds of thousands of lines, and the accumulated context of a multi-turn session can far exceed the window limit of an ordinary model. When context length is insufficient, engineering practice can only rely on retrieval-augmented generation (RAG) to stitch together fragments; the goal of long-context models is to give the model a global view of “reading the whole document,” reducing information loss in the retrieval step.

2.2 Three Major Technical Bottlenecks

1. Attention computational complexity: The compute of standard self-attention scales quadratically with sequence length (O(n²)). Expanding context from 4K to 128K magnifies single-layer attention compute by roughly a thousand times; without systematic optimization, both latency and cost become unacceptable.
2. KV Cache memory explosion: Autoregressive inference requires caching the key-value states of every historical token. For a 40-layer model with 8 KV groups, 128 dimensions per head, and fp16 precision, each token’s cache is about 160KB; at a context of 128K, a single sequence’s cache is roughly 20GB—merely “remembering” can exhaust the memory of an inference card.
3. Positional encoding extrapolation: Models see a limited range of positions during training. Once inference exceeds that range, attention distributions degrade rapidly, manifesting as irrelevant answers or repetitive output.

2.3 Common Industry Solutions

• Efficient attention implementations: tiled computation, IO-aware memory access optimization, and sparse or linear attention variants to reduce the practical constant overhead under quadratic complexity;
• KV Cache compression: grouped-query attention (multiple query heads sharing one set of KV heads), cache quantization, and hybrid designs combining sliding-window and global attention;
• Positional encoding improvements: building on rotary position encoding, combined with position interpolation and extrapolation strategies to extend the usable context range;
• Data curriculum: progressively expanding training sequences from short to long, paired with high-quality long-text corpora such as books, code, and long documents, to avoid the situation where “the window is open but the capability lags behind.”

2.4 Code Capability: Naturally Coupled with Long Context

Code is the most typical intersection of long context and precision requirements. On one hand, code has strong structure and long-range dependencies (cross-file function calls, type definitions), naturally requiring repository-level context; on the other hand, code is verifiable—whether generated code compiles and passes unit tests is objective, making it an ideal training signal for reinforcement learning with execution feedback. As a result, models focused on code capability are usually also deep users of long-context techniques, with the two capabilities amplifying each other.

Route 2: MoE Sparse Architecture—Decoupling Capacity from Compute

3.1 Core Idea: “Each Expert Has Its Specialty”

In dense models, every token passes through all parameters, locking model capacity and inference compute into a fixed ratio. MoE (Mixture of Experts) replaces the feed-forward layer with multiple parallel “expert” subnetworks, and a gating network selects only the few most relevant experts for each token (Top-K sparse activation). In this way, total parameter count determines the upper bound of knowledge capacity, while the activated parameter count per inference determines compute overhead—the two are decoupled for the first time.

Illustrative code:

import torch
import torch.nn.functional as F
def topk_gating(logits: torch.Tensor, k: int = 2, temperature: float = 1.0):
    """
    logits: [batch*seq_len, num_experts] output of the gating network
    temperature: temperature coefficient controlling the sharpness of the routing distribution
    """
    # Higher temperature makes the softmax distribution flatter and routing more "diffuse";
    # lower temperature makes it sharper and more focused on the top-scoring few experts.
    probs = F.softmax(logits / temperature, dim=-1)
    topk_val, topk_idx = torch.topk(probs, k, dim=-1)
    # Renormalize the weights of the selected K experts so the weighted-sum coefficients add up to 1
    topk_val = topk_val / topk_val.sum(dim=-1, keepdim=True)
    return topk_idx, topk_val

3.2 Engineering Challenges: What Is Cheap Is Not Free

1. Load balancing: Without intervention, gating tends toward “winner takes all”—a few experts get routed frequently while most receive little training. The common industry practice is to introduce an auxiliary balancing loss that penalizes imbalance in load across experts.
2. Training stability: Routing decisions are inherently discrete, and tokens “switching” between experts causes training jitter; finer learning-rate scheduling, routing noise control, and early-training warmup are typically required.
3. System overhead: Experts are usually distributed across different devices (expert parallelism), and tokens must be dispatched across cards (all-to-all communication); communication and scheduling are key to whether the system can handle large-capacity models. Meanwhile, although few parameters are activated, all expert parameters must reside in memory, so MoE is more like a “memory-for-compute” trade.

3.3 When to Choose It

MoE suits scaled services with massive token throughput and sensitivity to per-token cost; for scenarios with low call volume where the cost of keeping the model resident dominates, a small dense model is more economical. Moreover, MoE’s memory requirements mean it demands more from hardware for private deployment.

Route 3: Lightweighting—Fitting Models into Smaller Devices

The lightweight route answers the question: when budget, latency, or privacy constraints rule out large models, how can the most critical capabilities be preserved with the smallest possible model? The main techniques include:

• Knowledge distillation: using a large model as teacher and transferring general capabilities to a small model via soft labels or output-distribution distillation;
• Quantization: post-training quantization (PTQ) and quantization-aware training (QAT) compress weights from fp16 to int8 or even lower bit widths, significantly reducing memory and bandwidth overhead;
• Structured pruning: pruning redundant layers, attention heads, or channels;
• Custom training for small models: not simple truncation, but redesigning data ratios and training objectives for small capacity, so that limited parameters prioritize high-frequency core capabilities.

The essence of lightweighting is repricing the trade-off curve: finding the point on the capability-resource plane where the business has “enough.” For edge scenarios such as phones, vehicles, and edge devices, lightweighting is not a fallback but the only feasible solution.

Route 4: Multimodal—From “Reading Text” to “Understanding the World”

Human input is inherently multimodal: documents contain charts, meetings involve screen sharing, and operations involve monitoring curves. Typical multimodal architectures fall into two categories:

• Bridging: an independent vision encoder extracts image features, a projection layer aligns them to the language model’s representation space, and the language model serves as the unified reasoning hub;
• Unified tokenization: images, audio, and other modalities are discretized into tokens that share the same Transformer backbone and training objectives as text.

Training typically proceeds in two stages: first, modality alignment pretraining (teaching the model the underlying mapping of “describing what it sees”), then cross-modal instruction fine-tuning (enabling the model to perform OCR, chart question answering, document understanding, and similar tasks per instructions). The engineering difficulties lie in the high cost of constructing high-quality alignment data, inconsistent cross-modal semantic granularity (whether a token corresponds to an entire image or an image patch), and the multiplied compute cost of multimodal inference.

Controllability and Alignment: Two Paths to the Same Destination

As model capabilities grow, “controllability” matters no less than “capability.” The industry has explored two complementary paths:

• Rule internalization: through reinforcement learning from human feedback and safety alignment training, behavioral boundaries are internalized as constraints in model parameters, so the model “knows what it must not do”;
• Data shaping: working at the training-data level—deep cleaning of high-quality local corpora, fusion of knowledge graphs with language models (using deterministic knowledge to constrain the fuzzy boundaries of probabilistic generation), and value-corpus ratios that match the local context.

The former builds a safety framework from inside the model; the latter shapes output behavior from the data source. One relies on annotators’ value-ranking feedback, the other on the injection of structured knowledge. The two paths are not mutually exclusive; mature practice usually stacks them: alignment training as the foundation, knowledge augmentation to fill blind spots.

A Selection Evaluation Framework: Five Dimensions

No roadmap is inherently superior—only better matched. A structured evaluation can use the following framework:

1. Task fit: Is the core task long-document analysis, code generation, edge inference, or image-text understanding? Define the task first, then choose the route;
2. Context and throughput requirements: Measure the input-length distribution and concurrency of real business workloads, not just the advertised maximum window;
3. Cost and latency budget: Convert per-token price, time to first token, and throughput into unit business cost;
4. Deployment form and compliance: API calls, private deployment, or edge embedding? Is data allowed to leave the premises?
5. Evolvability: Does it support fine-tuning? Are there mature toolchains and evaluation support?

At the implementation level, there are three practical suggestions:

• Build your own evaluation set: General leaderboards have limited relevance to specific business needs; a few hundred evaluation cases built from real business samples are more persuasive than any public leaderboard;
• Gray-scale comparison: Run new and old models in parallel gray-scale deployments and let online metrics (adoption rate, rework rate, manual correction time) speak;
• Full-link observability: Use logging services and monitoring alerts to track call quality and cost, making model performance a continuously operable metric rather than a one-time conclusion.

In practice, mainstream cloud platforms can support the comparison experiments above. For example, platforms such as Baidu AI Cloud generally provide a complete toolchain for model invocation, fine-tuning, evaluation, and deployment, along with infrastructure such as object storage, container platforms, and monitoring alerts, helping teams build quantifiable and reproducible evaluation environments at low cost.

Conclusion

The story of Chinese LLMs is shifting from “catching up to the same answer” to “answering different questions.” Long context addresses information completeness, MoE addresses scaling cost, lightweighting addresses deployment boundaries, multimodality addresses information form, and alignment and controllability address trust. For developers and enterprise users, what truly matters is not “which one is strongest” but “which route resonates with my scenario.” Understanding the boundaries of technical principles and building your own evaluation system is the most durable competitive advantage in the LLM era.