Chinese Large Language Model Technical Routes: Long Context, MoE, and Native Multimodal Explained
This article breaks down the technical routes behind Chinese large language models, moving beyond single-benchmark rankings to explain why different teams make different architectural tradeoffs and which scenarios those tradeoffs fit. It examines three main routes—long context paired with strong code capabilities, sparse activation (Mixture of Experts), and native multimodal—alongside lightweighting and deployment optimization as a cross-cutting engineering theme. For each route, readers get the underlying principles, the sources of advantage, and the boundaries of applicability. The article also includes a side-by-side comparison of the routes, a shared foundation section covering data, knowledge augmentation, and safety alignment, and a practical checklist for enterprise model selection.
Chinese large language models have iterated quickly over the past few years. After initially following mainstream architectures from abroad, they have gradually moved toward a range of distinct technical routes. When developers and enterprises compare models, the usual question is “which model is the strongest.” A more useful question is different: why do different teams make different tradeoffs, and which use cases do those tradeoffs serve?
This article groups the technical routes of Chinese large language models into three: those built around long context and code capability, those built around sparse activation (the Mixture-of-Experts architecture), and those built around native multimodality. Lightweighting is an engineering direction that runs across all of them. Below, we examine each route’s technical principles, the source of its advantages, and the boundaries of where it applies, and finish with a side-by-side comparison and guidance for enterprise adoption.
1. First, a clarification: model strengths come from tradeoffs, not a single metric
When people discuss model strengths, they easily fall into “parameter-count-only” or “leaderboard-only” thinking. In reality, the capability space of a general-purpose large model is multidimensional: foundational knowledge and reasoning, context length, instruction following and output controllability, inference cost per token, multimodal understanding, and how feasible deployment and on-premises hosting are.
These dimensions are in clear tension with one another, and several are hard engineering constraints:
- The longer the context window, the higher the cost of attention computation and the KV cache.
- The larger a dense model’s parameter count, the higher the per-token inference cost.
- Native support for modalities such as vision and audio requires additional encoders, projection modules, and cross-modal training data.
As a result, when a team leads on one dimension, it usually means making tradeoffs on others. The key to selection is not asking “who is the strongest,” but asking “which dimension matters most to my business, and what am I willing to pay for it.”
From a technology-stack perspective, any large model can be decomposed into five layers: data, architecture, alignment, engineering, and ecosystem. The routes discussed below differ mainly at the architecture and engineering layers, which in turn drive differences in data mixture and alignment strategy.
| Layer | Key contents | Impact on model performance |
|---|---|---|
| Data | Corpus scale, quality, and mixture (natural language, code, multilingual, interleaved text-and-image); cleaning and deduplication | Determines breadth of knowledge, coding ability, and language coverage |
| Architecture | Dense or sparse, attention mechanism, positional encoding, native multimodality or not | Determines context length, inference cost, and range of supported modalities |
| Alignment | Instruction tuning, preference optimization, safety and value constraints | Determines usability, stability, and controllability |
| Engineering | Distributed training, inference acceleration, quantization, and deployment optimization | Determines training feasibility, serving throughput, and unit cost |
| Ecosystem | Model size matrix, interface forms, tooling, and documentation | Determines enterprise integration cost and room for secondary development |
2. Route one: Long context and code capability — for the real engineering of “understanding long documents and modifying large codebases”
2.1 Where the difficulty of long context actually lies
Specs like “supports one million tokens” are easy to understand, but the engineering challenges behind them are often underestimated. They show up in four main areas:
- Attention complexity. The cost of standard self-attention grows quadratically with sequence length. Double the sequence and the computation roughly quadruples, so training and inference costs spiral out of control.
- Positional encoding extrapolation. Mainstream models commonly use rotary position embeddings (RoPE). When a model is trained on shorter sequences, its positional encoding represents very long positions poorly, and directly extrapolating to them leads to a clear drop in quality.
- KV cache growth. Autoregressive decoding must cache the keys and values of past tokens, so the longer the sequence, the larger the cache.
- The “lost in the middle” problem. Research shows that models tend to use information at the beginning and end of a long text more effectively than information in the middle. This means long context is not just about “whether you can fit it in,” but “whether you can find it and use it.”
2.2 Key techniques for long context
For the problems above, the industry has converged on a relatively mature set of solutions:
- Positional encoding extrapolation. By adjusting RoPE’s rotation frequencies or interpolating position indices (approaches such as NTK-aware interpolation and YaRN), positions never seen during training are mapped into a range the model is already familiar with. A small amount of continued training on long text then lets the model gradually adapt to longer sequences.
- Efficient attention. Sparse attention, a “window plus global” pattern that combines a sliding window with a few global tokens, and linear attention can all bring attention complexity down from quadratic to near-linear. On the implementation side, chunked prefill splits a long sequence into chunks and mixes it with short requests in scheduling, which can raise GPU utilization without a large increase in latency for long requests.
- Reducing KV cache overhead. This includes:
- Grouped-query attention (GQA): multiple query heads share a group of key/value heads, greatly shrinking the cache with only a small quality loss;
- Paged KV cache management: allocating cache blocks on demand, which significantly reduces memory fragmentation and reserved-but-unused capacity;
- KV cache quantization;
- Sequence/context parallelism: splitting a very long sequence across multiple accelerators to compute collaboratively.
- Data and training strategy. Continued training on long documents, code repositories, cross-document summaries, long dialogues, and similar data teaches the model to actually exploit distant information, not merely to “fit it in.”
The KV cache memory estimate is as follows (in GiB):
KV_Cache_GiB =2 × n_layers × n_kv_heads × head_dim × seq_len × batch × dtype_bytes/ 1024^3
Example: assume a model with 60 layers that uses GQA with 8 key/value heads, a head dimension of 128, and 16-bit weight precision (dtype_bytes=2). With a batch size of 16 and a context length of 128K tokens:
- The KV cache needs about 61 GiB of memory — nearly the full capacity of an 80 GB accelerator on its own;
- With 4-bit quantization, it drops to about 15 GiB;
- With standard multi-head attention (n_kv_heads equal to the 128 attention heads), the 16-bit cache balloons to about 976 GiB.
This shows that KV cache optimization is not a nice-to-have. It is a decisive factor in whether long-context serving can actually be deployed.
2.2 Code capability: it is more than “lots of code data”
Code capability comes from four layers:
- High-quality code corpora. Covering mainstream programming languages, open-source repositories, technical documentation, programming Q&A, and code review records.
- Training objectives suited to completion. Fill-in-the-Middle (FIM) training randomly removes a span of code from a sample and has the model predict the middle content while seeing both the preceding and following context. This closely matches the real scenario of “completing code at the current cursor position” in an IDE.
- Repository-level context. Real development requires understanding type definitions, interface contracts, and call relationships across files. This can be supported by long context, and by retrieval-augmented generation (RAG) and abstract syntax tree (AST)-based code retrieval, which assemble relevant code snippets into the prompt.
- Verifiable reinforcement learning. Code can naturally be verified by unit tests, compilation results, and static analysis, so reinforcement learning can use test pass rates as a reward signal. This kind of “verifiable reward” has become an efficient way to improve coding ability.
Interestingly, long context and code capability are naturally complementary: modifying a large project requires both cross-file understanding (long context) and the generation of compilable, testable code (coding ability). Combined, they let a model evolve from “completing a single line of code” into a coding assistant that can “understand a task, locate files, and propose a change plan.” Coding tools such as Comate from Baidu AI Cloud represent the engineering realization of exactly this direction.
3. Route two: Sparse activation (MoE) — decoupling “model capacity” from “compute per pass”
3.1 The core idea: specialists each do their own job
The core idea of the Mixture-of-Experts (MoE) architecture is to replace the feed-forward layers in a Transformer with several parallel “expert” subnetworks, and to let a gating network (the Router) decide which experts handle each token.
The schematic formula is as follows (k is the number of experts activated per token; N is the total number of experts):
Gate scores: g = softmax(W_gate · x)Routing: top_idx, top_w = TopK(g, k)Layer output: y = Σ_{i ∈ top_idx} top_w[i] · Expert_i(x)
Together with a softmax temperature coefficient, the routing distribution can be adjusted flexibly:
- With a higher temperature, the probability distribution is more even, more experts take part in the computation, and the result is smoother;
- With a lower temperature, the distribution is sharper, only the few top-ranked experts are selected, and “specialization” is emphasized.
One common misconception is worth clearing up: MoE’s sparsity comes from “each token passing through only k experts,” not from “the temperature parameter controlling how many experts are activated.” Temperature affects the smoothness of routing and the stability of training.
A minimal implementation written in the style of a mainstream deep learning framework follows, to make the relationship between gating, sparse computation, and load balancing easier to understand:
import torchimport torch.nn as nnclass TopKMoE(nn.Module):"""Minimal MoE layer example: illustrates gating routing, sparse activation, and a load-balancing loss"""def __init__(self, d_model: int, n_experts: int, k: int = 2, aux_coef: float = 0.01):super().__init__()self.n_experts, self.k, self.aux_coef = n_experts, k, aux_coef# Gating network: score each token across all expertsself.gate = nn.Linear(d_model, n_experts, bias=False)# Each expert is an independent feed-forward subnetworkself.experts = nn.ModuleList([nn.Sequential(nn.Linear(d_model, 4 * d_model),nn.GELU(),nn.Linear(4 * d_model, d_model),) for _ in range(n_experts)])def forward(self, x):# x: [batch, seq_len, d_model]logits = self.gate(x) # [B, L, E]probs = logits.softmax(dim=-1)top_w, top_idx = probs.topk(self.k, dim=-1) # each token activates only k expertsout = torch.zeros_like(x)# Illustrative version: compute expert by expert. A production implementation# usually groups tokens by expert first, then uses grouped matmuls to reduce# the number of small operators and improve parallelismfor e, expert in enumerate(self.experts):mask = (top_idx == e) # [B, L, k]# If the same token selects the same expert more than once, sum the weightsweight = (mask.float() * top_w).sum(dim=-1, keepdim=True) # [B, L, 1]out = out + weight * expert(x)# Load-balancing auxiliary loss: penalize "a few experts overloaded, most idle"# f: the fraction each expert was actually selected; p: the average gating probability per expertf = torch.zeros(self.n_experts, device=x.device)for e in range(self.n_experts):f[e] = (top_idx == e).float().mean()p = probs.mean(dim=(0, 1))aux_loss = self.n_experts * (f * p).sum()return out, self.aux_coef * aux_loss
3.2 The core benefits of MoE
The most fundamental value of MoE is that it decouples two metrics: model capacity is determined by the total number of parameters, while the compute for a single inference pass is determined by the number of activated parameters.
For example, a model with roughly 240 billion total parameters that activates only about one-third of its parameters per token has a per-token compute cost roughly comparable to a hundred-billion-scale dense model, yet its knowledge capacity approaches that of a much larger dense model. Under the same per-token compute budget, a sparse model can often deliver stronger capability at a better “quality-to-cost ratio,” which is especially suited to high-concurrency online serving. Around 2024, several vendors in China shifted to this architecture and adopted it as the mainstay route for their flagship models.
3.3 Key engineering challenges and how to address them
- Load balancing. When some experts are overused while the rest are nearly idle — “expert collapse” — capacity is wasted and training slows down. Common countermeasures include:
- An auxiliary load-balancing loss: the aux_loss in the formula above is the simplest version;
- A z-loss on the routing distribution: constraining the numerical magnitude of the gating logits to prevent them from growing without bound and destabilizing training;
- An expert capacity factor: capping how many tokens a single expert can receive, so that overflow tokens spill to the next layer or are dropped.
- Fine-grained and shared experts. Slicing experts more finely and using more of them can improve specialization and compositional expressiveness. Keeping a small number of shared experts that always participate in computation, meanwhile, carries general knowledge and reduces the redundancy of different experts relearning the same knowledge.
- Parallelism and communication. Experts are usually distributed across different devices (cards, machines), so after gating, tokens must be sent across devices — this is All-to-All communication. It can become a training bottleneck. Optimization directions include expert parallelism, overlapping computation with communication, and limiting the number of cross-node tokens per token.
- Training stability. Sparse routing introduces gradient noise, so a training-friendlier normalization (such as RMSNorm) is usually chosen, and the loss coefficients related to routing are tuned carefully.
- Inference deployment. MoE does not “save memory” — the total weight volume is still there, and memory usage and cross-card communication pressure remain. But its compute requirement is low, so a single card or just a few cards can host a very large model capacity. This is especially friendly to teams with limited budgets that still need strong capability.
3.4 Supporting cost-reduction engineering
Truly lowering the cost per token often relies not on the architecture alone but on several engineering techniques stacked together:
- Quantized inference: quantizing weights and activations to 8-bit or even 4-bit, reducing memory usage and memory-access overhead;
- Continuous batching: stitching tokens from different requests into a dynamic batch to raise GPU utilization;
- Speculative decoding: using a lightweight draft model (or the model’s own shallow-layer predictions) to quickly generate candidate tokens, which the target model then verifies in one parallel pass. When the acceptance rate is high, this can noticeably reduce decode latency;
- Prefix cache reuse: computing shared prefixes such as system prompts only once, and reusing the cache directly for repeated requests.
3.5 Suitable scenarios
- Conversational, writing, and customer-service applications serving massive numbers of users, where per-token cost is sensitive;
- Startup teams and small-to-medium businesses that need fairly strong general capability but have limited budgets and hardware;
- Inference and on-premises deployment, or scenarios that want to fine-tune a model for a specific domain at relatively low cost.
4. Route three: Multimodal — from “stitching text and images” to “native understanding”
The early mainstream approach was to attach a vision encoder in front of a language model and then map visual features into the language model’s space through a projection layer — essentially “stitched-together” multimodality. A newer trend in recent years is native multimodality: from day one of pretraining, text, images, audio, and video are jointly modeled as a unified token sequence. This creates deeper cross-modal semantic connections and more complete understanding of long documents, tables, charts, and interface screenshots.
Multimodal models face four main technical difficulties:
- The cost-effectiveness of visual tokens. A high-resolution image can produce hundreds or even thousands of visual tokens after passing through an encoder. The industry commonly uses two kinds of methods:
- Dynamic resolution: first generate a downscaled thumbnail of the whole image, then split it by aspect ratio into multiple small tiles encoded separately, balancing global context and fine detail;
- Compression and merging: reducing token count via token merging, pooling, or query-based compression modules.
- The scale and quality of interleaved text-and-image data. Large-scale, high-quality image-text alignment data is needed, as well as interleaved data from books, web pages, and documents. Otherwise the model tends to produce shallow, “describe-the-picture” matching with insufficient reasoning depth.
- Modal conflict and hallucination. A multimodal model may ignore the image content and answer purely from language priors, or describe objects that are not in the image. Mitigations include negative-sample training (pairing the image with irrelevant questions), preference alignment, prompt strategies that force the model to cite visual evidence before answering, and image-grounded validation of outputs before they are returned.
- The difficulty of multimodal evaluation. Beyond traditional visual question answering and document-understanding benchmarks, you also need multimodal test sets that reflect real business — for example, receipt/invoice recognition, chart reading, and defect detection on industrial images.
Teams that invest heavily in the multimodal route usually also have the following characteristics:
- Their model family forms a “size ladder,” covering different compute conditions from small on-device sizes to flagship sizes;
- They connect perception and generation: beyond understanding images, they can also generate images and synthesize speech;
- They target document and Agent scenarios: they can “read” interface screenshots and operate software to complete tasks, which is also an important foundation for general-purpose Agent deployment.
5. An engineering thread running through multiple routes: lightweighting and deployment optimization
Lightweighting is not an independent technical route. It is the engineering thread that determines whether a model can enter real business. Whether you choose the long-context, MoE, or multimodal route, you ultimately face the same question: how to serve inference reliably and cheaply on limited hardware. Common techniques include:
- Distillation: training a smaller model on the outputs of a large model (including soft label distributions) so that it approaches the teacher model on most common tasks;
- Pruning and quantization: trimming redundant structures and lowering the precision of weights and activations;
- A model size matrix: offering multiple sizes within the same family, from on-device to flagship, so the business side can route by task difficulty to models of different sizes — “small models as a fallback, large models for hard cases”;
- Cloud-edge collaboration: completing sensitive or high-frequency simple tasks locally, and sending complex tasks up to a large model in the cloud.
6. Side-by-side comparison of the routes
| Dimension | Long context and code capability | Sparse activation (MoE) | Multimodal |
|---|---|---|---|
| Source of advantage | Efficient attention, positional extrapolation, FIM, and verifiable reinforcement learning | Decoupling of capacity and compute; low per-token compute demand | Native cross-modal tokenization; visual token optimization |
| Core capability | Long-document understanding, code generation and completion, repository-level tasks | General Q&A, content generation, cost-sensitive tasks | Image and document understanding; mixed text-and-image reasoning |
| Main cost | Inference latency on long sequences; KV cache memory overhead | Complex training; expert-parallel communication overhead | Heavy visual token cost; high alignment difficulty |
| Typical scenarios | Contract and report analysis, intelligent coding, knowledge-base Q&A | High-concurrency dialogue, intelligent customer service, content production | Receipt and document recognition, visual quality inspection, interface Agents |
| Selection tip | Focus on measured performance in long-text retrieval and multi-hop reasoning | Focus on per-token cost and concurrent throughput | Focus on accuracy on real business images |
7. A shared foundation: data, knowledge augmentation, and safety alignment
7.1 Data quality sets the floor
Whichever route you take, the final performance is determined by data:
- Data cleaning: deduplication, detoxification, and removal of low-quality content;
- Mixture design: blending natural language, code, math, and multilingual data in set proportions;
- Curriculum-style training: progressing in stages such as “fundamentals → capability improvement → long-context/multimodal expansion.”
In areas such as Chinese semantic understanding, idioms and classical allusions, laws and regulations, and traditional culture, high-quality local corpora are an advantage that cannot be replaced.
7.2 Knowledge augmentation: retrieval and knowledge graphs each have their strengths
A model’s knowledge always lags behind updates in the real world, so it needs external knowledge:
- Retrieval-augmented generation (RAG) chunks unstructured documents, vectorizes them, and stores them in a vector database; at generation time, relevant snippets are recalled and inserted into the prompt. It is cheap and fast to update.
- Knowledge graphs store knowledge in the deterministic structure of “entity–relation–entity” and excel at answering structured questions such as multi-hop reasoning and relationship queries.
Combining the two lets a graph constrain the retrieval scope and reduce hallucination. In essence, this uses deterministic external knowledge to compensate for the uncertainty of a probabilistic generative model.
7.3 Safety alignment: two complementary approaches
There are roughly two representative approaches to aligning today’s large models:
- Rule-driven: humans first define a set of behavioral guidelines, then the model critiques and revises its own outputs against those guidelines, and preference optimization internalizes the guidelines into the model;
- Data-driven: local context and values are deeply embedded in the training data, then preference optimization achieves value alignment, combined with targeted enhancement using industry knowledge.
The two may look like different routes, but they are complementary. The former is like an “external constitution”: clear boundaries and easy to audit, but it may miss the mark in complex local contexts. The latter is like a “growth environment”: it fits local expression habits, but it is hard to define boundaries precisely. A mature enterprise approach usually combines the two: rules set the red lines, data shapes the habits, and evaluation keeps calibrating.
8. Enterprise selection: a practical decision checklist
- Define the core task. Is it long-document analysis, code development, multimodal understanding, or high-concurrency dialogue?
- Build your own evaluation set. Sample several hundred representative cases from real business, covering both common scenarios and edge cases, focusing on:
- Long context: accuracy in locating a specified clause within a very long text;
- Code capability: correctness and test pass rate for multi-file changes in a real repository;
- Multimodal: field-level accuracy on your own invoices, reports, and images;
- Safety alignment: refusal rate on sensitive questions and the degree of adherence to response guidelines.
- Do the cost math. Beyond unit price, also evaluate concurrent throughput, average response latency, and the hardware investment for on-premises deployment.
- Validate the engineering support. Look at interface capabilities such as context management, function calling, structured output, and streaming responses, and whether the inference service can scale elastically.
- Run a small pilot, then scale up gradually, and establish ongoing monitoring.
For cloud deployment, you can assemble the pipeline using mature, general-purpose infrastructure:
- Object storage: holding training data, document corpora, and inference results;
- Container platform: supporting elastic scaling of the inference service;
- Message queue: smoothing peaks and troughs and making batch tasks asynchronous;
- Logging, monitoring, and alerting: tracking latency, throughput, error rates, and content-safety metrics;
- Knowledge base and vector retrieval: supporting the RAG pipeline.
This lets you focus your effort on business logic while allowing model capability to scale smoothly as the business grows.
9. Outlook
- Long context and sparse activation converge. The KV cache and attention overhead that long context brings naturally calls for sparsification. Future flagship models will likely be a combination of “sparse experts + efficient attention + very long context.”
- Native multimodality becomes standard. Text, images, audio, and video are modeled uniformly at the pretraining stage, and vision is no longer just a bolt-on organ for a language model.
- From “answering” to “executing.” The model’s role is evolving from a Q&A assistant into an Agent that can call tools and operate software, which raises the bar for long context, multimodality, and controllability.
- Large and small models cooperate. Complex tasks go to the flagship model, while high-frequency simple tasks are handled by distilled small models, delivering a better overall experience at lower cost.
Returning to the original question: Chinese large language models have no single “strongest” option — only the “most suitable” one for a specific scenario. Understanding the tradeoffs behind each technical route is what lets you move past parameter comparisons and make a genuinely rational technical choice.
