Domestic Large Language Model Architectures: Technical Differences, Principles, and Selection Guide

    This guide explains how domestic large language models have evolved from reproducing mainstream Transformer designs to making architecture and training choices based on practical constraints. It compares long-context processing, code generation, inference efficiency, and multimodal capabilities, then explains how Mixture-of-Experts models use sparse activation, temperature-based routing, and load balancing to reduce per-token computation.

    1. From Reproduction to Independent Technical Choices

    The Transformer architecture, based on self-attention, was introduced in 2017 and has since become the common foundation for large language models. In the early stages, many domestic teams focused on reproducing mainstream model structures within this framework. Their primary efforts centered on training pipelines, data engineering, and compute scheduling.

    As compute resources accumulated and data quality improved, domestic teams began making independent choices in model architecture, training methods, and application scenarios.

    This shift has three major characteristics:

    1. Technical decisions moved from reproducing known architectures to designing around specific constraints, such as inference cost, memory usage, and Chinese-language coverage.
    2. Research priorities shifted from simply increasing parameter counts to improving parameter utilization.
    3. Evaluation expanded beyond general-purpose benchmarks to include specific tasks such as code generation, long-document processing, and multimodal understanding.

    These changes are important because today’s differences between models are driven primarily by different constraints and design priorities—not simply by which model has more parameters.

    2. Core Capabilities and Trade-Offs Across Four Directions

    The capabilities of a large language model can generally be divided into four dimensions: context length, code capability, inference efficiency, and multimodal capability. Different teams allocate research resources differently across these areas, and each direction involves distinct technical challenges and application boundaries.

    Direction Research Focus Main Technical Challenges Suitable Applications Common Evaluation Metrics Common Misconceptions
    Long context Attention computation and cache management KV cache size grows linearly with sequence length, while prefill computation grows quadratically Long-document question answering and long-codebase analysis Needle-in-a-haystack retrieval accuracy, long-document summarization quality, time to first token Looking only at the advertised context window while ignoring actual long-distance information utilization
    Code capability Ratio and quality of code in the training data Repository-level context understanding and verification of generated code correctness Code completion, bug localization, and unit test generation Executable test pass rate and code-completion acceptance rate Using human ratings instead of executable tests, which can make conclusions subjective
    Inference efficiency Quantization, distillation, and inference-engine adaptation Balancing accuracy and speed High-concurrency services, low-latency applications, and edge deployment Output tokens per second, cost per request, and P95 latency Comparing only peak throughput without evaluating performance with long inputs
    Multimodal capability Alignment between the vision encoder and language model Mapping visual information to text semantics and managing token growth caused by high-resolution inputs Receipt recognition, chart understanding, and document-layout analysis Image-text question-answering accuracy and table reconstruction accuracy Inferring performance on complex documents from results on a single simple image

    Model selection should begin by identifying which capability matters most to the business. The other dimensions can then be evaluated based on whether their trade-offs are acceptable. For example, a model with strong long-context performance will usually involve higher inference costs.

    3. Mixture-of-Experts: Reducing Per-Request Computation Through Sparse Activation

    The basic idea behind Mixture-of-Experts (MoE) is to divide the feed-forward network into multiple expert subnetworks. A gating network, or router, then selects a small number of experts for each input token. The full set of experts provides the model’s overall capacity, while each inference step uses only the selected experts.

    3.1 Computation Flow

    1. The gating network produces a score, or logit, for each expert for the current token.
    2. A softmax operation converts the scores into a normalized probability distribution.
    3. The top k experts are selected.
    4. The selected weights are renormalized, and the outputs of the selected experts are combined using a weighted sum.

    3.2 Total Parameters and Active Parameters

    The total parameter count of an MoE model includes all experts, while a single inference step activates only k of them. Total parameters represent the model’s capacity, whereas active parameters are more closely related to the computation required for an individual token.

    Assume there are E experts, each containing p parameters, and each token activates k experts. The expert portion contains E × p total parameters, while each token passes through k × p expert parameters.

    The parameters in the attention layers, embedding layers, and gating network are shared by all tokens and still participate in computation. They must therefore be included when estimating the actual computational cost.

    For example, assume:

    • E = 8
    • p = 1B parameters
    • k = 2
    • Shared parameters: 1B

    The expert portion contains 8B parameters. Including the shared parameters, the model has approximately 9B total parameters. Each token passes through 2B expert parameters plus 1B shared parameters, for approximately 3B active parameters.

    This model has a capacity close to that of a dense 9B model, while its per-token computation is closer to that of a dense 3B model.

    3.3 The Role of Temperature

    A temperature coefficient T can be applied to the softmax in the gating network by dividing each score by T before normalization. A higher T produces a smoother weight distribution, while a lower T produces a sharper distribution.

    Temperature does not change which experts are selected. Because softmax is a monotonic transformation, it preserves the ranking of the scores. Therefore, the set of experts selected by top-k routing is independent of T; temperature changes only the weights assigned to those experts.

    The number of experts activated during inference is determined by k. Some training strategies adjust the temperature or add noise during training to balance expert exploration and routing convergence. During inference, T and k are typically fixed so that routing remains stable for the same input.

    3.4 Load Balancing

    If routing remains concentrated on a small number of experts, the remaining experts receive insufficient training and part of the model’s capacity is wasted. Uneven routing can also overload some devices during inference and increase overall latency.

    Three common approaches are used to address this problem.

    First, an auxiliary load-balancing term can be added to the loss function. One common formulation measures the actual proportion of tokens received by each expert, f_i, and the average probability assigned by the gating network, P_i, and then calculates the sum of their products.

    The coefficient of the auxiliary loss must be tuned on a validation set. If it is too large, routing may be forced to become overly uniform, preventing experts from developing specialization and reducing model quality. If it is too small, routing may remain concentrated.

    Second, a capacity limit can be set for each expert. The limit is commonly estimated using:

    1. number of tokens per batch × k ÷ E × capacity factor

    Tokens that exceed the limit may bypass the layer through a residual path or be dropped. This approach has side effects that must be considered. A dropped token loses part of the computation, and the same sentence may produce slightly different results in different batches because congestion can vary.

    If the capacity factor is too small, the token drop rate may become excessive. If it is too large, the limit loses its purpose as a form of traffic control.

    Third, randomness can be increased during the early stages of training to prevent routing from becoming fixed too quickly. Once routing has stabilized around a small number of experts, it can be difficult to correct later.

    3.5 Example Code

    The following code uses only the Python standard library to demonstrate top-k routing for a single token. It runs on Python 3.8 or later and requires no additional dependencies.

    1. import math
    2. def softmax(logits, temperature=1.0):
    3. if temperature <= 0:
    4. raise ValueError("temperature must be positive")
    5. scaled = [x / temperature for x in logits]
    6. m = max(scaled) # Subtract the maximum value to prevent overflow
    7. exps = [math.exp(x - m) for x in scaled]
    8. total = sum(exps)
    9. return [e / total for e in exps]
    10. def top_k_router(logits, k, temperature=1.0):
    11. if not 1 <= k <= len(logits):
    12. raise ValueError("k must be between 1 and the number of experts")
    13. probs = softmax(logits, temperature)
    14. ranked = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)
    15. selected = ranked[:k]
    16. weight_sum = sum(probs[i] for i in selected)
    17. return {i: probs[i] / weight_sum for i in selected}
    18. def moe_forward(token_value, experts, gate_logits, k=2, temperature=1.0):
    19. weights = top_k_router(gate_logits, k, temperature)
    20. output = 0.0
    21. for idx, w in weights.items():
    22. output += w * experts[idx](token_value)
    23. return output, weights
    24. if __name__ == "__main__":
    25. experts = [
    26. lambda x: x * 2,
    27. lambda