Why Chinese LLMs Are Taking Different Technical Paths: Core Capabilities Explained

    Chinese large language models are increasingly differentiated by their trade-offs in context length, coding and reasoning, deployment efficiency, multimodal support, and alignment. This article explains the engineering principles behind these differences, including KV cache costs, code-data strategies, model compression, multimodal architecture, Mixture-of-Experts routing, and practical evaluation methods for production model selection.

    1. Why LLM Technical Paths Are Diverging

    Early Chinese AI teams relied heavily on publicly available general-purpose architectures and training practices, focusing primarily on replication and catching up. As access to compute, data, and engineering expertise improved, teams began choosing different optimization priorities based on their resources and target use cases.

    These differences are most visible across four dimensions:

    • Context length
    • Coding and reasoning capabilities
    • Deployment cost
    • Modalities supported

    Compared with simply ranking models by benchmark scores, understanding the trade-offs across these dimensions is more useful for real-world engineering decisions.

    2. Long Context: From Attention Costs to Engineering Optimization

    The computational cost of standard self-attention grows quadratically with sequence length. During inference, however, the GPU memory consumed by the key-value cache, or KV cache, grows linearly with sequence length. Long-context capabilities therefore typically depend on three categories of techniques:

    • Improved positional encodings, allowing the model to represent token positions beyond the length used during training;
    • Sparse or block-based attention, reducing the range of tokens each token needs to attend to;
    • Cache compression, such as sharing key-value heads or quantizing the cache to lower precision.

    KV cache memory can be estimated with the following formula:

    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch=1, dtype_bytes=2):
    2. """Estimate KV cache memory in bytes during inference (keys and values are both stored)."""
    3. return 2 * layers * kv_heads * head_dim * seq_len * batch * dtype_bytes
    4. # Hypothetical configuration: 32 layers, 8 KV heads, 128 dimensions per head,
    5. # 128K context, and FP16 precision
    6. size = kv_cache_bytes(layers=32, kv_heads=8, head_dim=128, seq_len=131072)
    7. print(f"{size / 2**30:.1f} GiB") # Outputs 16.0 GiB

    This shows that even when a model does not have a particularly large parameter count, its cache overhead in long-context scenarios can exceed the memory required by the model weights themselves.

    Reducing the number of KV heads through grouped sharing, or quantizing the cache to 8-bit integers, can directly reduce this overhead. When selecting a model, teams should evaluate context length together with concurrency and precision requirements rather than looking only at the maximum advertised context window.

    3. Coding Capability: Data, Training Objectives, and Evaluation

    Coding performance is mainly determined by three factors.

    Data

    Training corpora should be deduplicated, filtered for low-quality files, and checked for license compliance. Function-level data, repository-level data, and comment-code pairs serve different purposes. Their proportions directly affect a model’s code completion and code understanding capabilities.

    Training Objectives

    In addition to standard next-token prediction, the fill-in-the-middle objective teaches a model to complete code when both the preceding and following context are known. This more closely matches real use cases in code editors.

    Evaluation

    Text similarity metrics cannot determine whether generated code is functionally correct. A more reliable approach is to execute generated functions in a sandbox and measure the pass rate using unit tests. The following example illustrates a simplified evaluation process:

    1. def pass_rate(candidates, tests, run_in_sandbox):
    2. """Return the proportion of problems for which at least one candidate
    3. passes the complete test suite."""
    4. solved = 0
    5. for problem_id, codes in candidates.items():
    6. if any(run_in_sandbox(code, tests[problem_id]) for code in codes):
    7. solved += 1
    8. return solved / len(candidates)

    The sandbox must isolate both network access and the file system. Otherwise, the evaluation process itself can introduce security risks.

    4. Model Compression: Running LLMs with Limited Resources

    The goal of model compression is to reduce inference costs while keeping accuracy loss within an acceptable range. Common approaches include:

    • Quantization: Compressing weights from 16-bit floating-point values to 8-bit or 4-bit integers. This can reduce memory usage by approximately half or to one quarter, but lower-bit formats may cause significant degradation on some tasks.
    • Distillation: Training a smaller model with the probability distribution produced by a larger model as soft labels, allowing the smaller model to learn richer relationships between possible outputs.
    • Pruning: Removing weights or structures that contribute relatively little to the output. Additional retraining is usually required to recover lost accuracy.
    • Speculative decoding: Having a smaller model generate candidate tokens first, then asking the larger model to verify them in batches. This can improve throughput without changing the output distribution.

    The benefits of each compression method depend on the task. A quantization scheme that causes almost no loss in general question answering may noticeably reduce performance in mathematical reasoning or long tool-calling workflows. Before deployment, every optimization should therefore be tested against the organization’s own business data.

    5. Multimodality: Unified Representations and Alignment

    A common multimodal architecture uses a vision encoder to divide an image into patches and encode them as vectors. A projection layer maps these vectors into the input space of a language model, which then processes image tokens and text tokens in a unified way. Training data often includes image-text pairs, chart question-answering samples, and document-understanding examples.

    This design presents two major challenges.

    The first is the trade-off between resolution and computation. High-resolution images produce a large number of image tokens, significantly increasing time to first response.

    The second is preserving fine-grained information. Small numbers in a table or axis labels in a chart require sufficient pixel-level information during encoding. A common engineering approach is dynamic tiling, in which the number of image regions is determined by the image content.

    6. Sparse Expert Architectures: How MoE Works and Common Misconceptions

    A Mixture-of-Experts (MoE) model divides the feed-forward layer into multiple expert subnetworks. A gating network selects a small number of experts for each token. The key distinction is between total parameters and active parameters: the total parameter count can be very large, while only a subset is activated during each forward pass.

    The gating process can be summarized in three steps:

    1. Calculate a score for each expert.
    2. Apply softmax to obtain routing weights.
    3. Select the top K experts and compute a weighted sum of their outputs.

    A simplified implementation is shown below:

    1. import numpy as np
    2. def top_k_gating(logits, k, temperature=1.0):
    3. """Apply temperature scaling and softmax to gating scores, then select
    4. the top K experts and their normalized weights."""
    5. scaled = logits / temperature
    6. probs = np.exp(scaled - scaled.max(axis=-1, keepdims=True))
    7. probs /= probs.sum(axis=-1, keepdims=True)
    8. topk_idx = np.argsort(-probs, axis=-1)[..., :k]
    9. topk_prob = np.take_along_axis(probs, topk_idx, axis=-1)
    10. topk_prob /= topk_prob.sum(axis=-1, keepdims=True)
    11. return topk_idx, topk_prob

    Temperature is often misunderstood as a switch that controls the number of activated experts. It does not work that way. Temperature only scales the scores before softmax, changing how concentrated the routing weights are.

    For two experts with scores [2, 1]:

    • At a temperature of 1, the weights are approximately [0.731, 0.269];
    • At a temperature of 5, the weights are approximately [0.550, 0.450], producing a more even distribution;
    • At a temperature of 0.5, the weights are approximately [0.881, 0.119], producing a more concentrated distribution.

    Regardless of the temperature, top-K routing activates exactly K experts. Temperature changes the relative contribution of the selected experts, not their number.

    MoE training must also address load balancing. If the gate consistently favors a small number of experts, the remaining experts receive insufficient training and computing resources are underused. Common solutions include adding an auxiliary loss that encourages a more uniform average routing rate and setting a capacity limit for each expert.

    In distributed training, experts may be placed on different devices. Tokens must then be routed across devices, making communication overhead a major system design bottleneck.

    7. Data and Value Alignment: Complementary Paths

    Model behavior can be controlled at two levels.

    Internal Model Constraints

    Principle-based methods ask a model to critique and revise its own responses according to explicit rules. Reinforcement learning from human preferences uses human rankings of multiple responses to train a reward model.

    These approaches can provide clearly defined behavioral boundaries. However, the rules require ongoing maintenance, and preference data can be influenced by the cultural backgrounds of the annotators.

    Data-Level Shaping

    The geographic distribution, languages, and communication styles represented in training data directly influence a model’s default output style. Localized data cleaning and instruction fine-tuning for specific application scenarios are both forms of data-level shaping.

    These two approaches are not mutually exclusive. One practical combination is to use data-level adjustments to improve performance in common scenarios while relying on internal model constraints to enforce minimum safeguards for high-risk outputs.

    Knowledge graphs can also serve as a supplement by providing verifiable structured facts through retrieval-augmented methods. However, their value should be demonstrated through task-specific evaluation rather than assumed based only on theoretical benefits.

    8. Model Selection and Deployment Recommendations

    For a specific project, the decision process can follow these steps:

    1. Define the task type: long-document analysis, code generation, edge deployment, or image-text understanding.
    2. Build an evaluation set from business data, including normal cases, edge cases, and high-risk cases.
    3. Measure key metrics such as time to first response, GPU memory per request, concurrent throughput, and the KV cache growth curve as context length increases.
    4. Review compliance requirements, including data retention, content moderation, and output traceability.
    5. Design a fallback strategy so the system can switch to a backup model or rule-based safeguard when the primary model is unavailable or its quality declines.

    9. Conclusion

    The differences between Chinese LLM development paths ultimately reflect different trade-offs among context costs, coding quality, deployment barriers, and modality coverage.

    Understanding the routing mechanism of sparse expert architectures, the growth pattern of KV cache memory, and the complementary roles of data and model alignment makes it possible to turn abstract benchmark comparisons into testable engineering decisions.

    For production deployment, evaluation results on the organization’s own tasks are more reliable than any single benchmark or metric.