MoE Architecture in Chinese LLMs: Core Capabilities Beyond Parameter Scale

    This article examines MoE architectures in Chinese large language models beyond headline parameter counts. It explains total versus activated parameters, analyzes four common MoE designs—shared-plus-routed experts, pure routed MoE, code- or task-aware routing, and hierarchical routing—and discusses why activated parameters matter more for deployment. It also covers expert load balancing, how coding ability can reveal expert specialization, and a practical framework for evaluating and selecting MoE models.

    1. Why 1.6 Trillion Parameters Do Not Equal 1.6 Trillion Units of Compute

    In recent years, large language models have grown from tens of billions of parameters to hundreds of billions and then to the trillion scale. During actual inference, however, not all parameters participate in computation at the same time. More models are adopting MoE, or Mixture of Experts, which splits a very large neural network into multiple relatively independent “expert modules.” A router then selects a small number of experts based on the input to process the current token.

    Therefore, when evaluating an MoE model, at least three concepts must be distinguished:

    1. Total parameter count: the sum of all experts, shared modules, embedding layers, and output layer parameters. It reflects the upper bound of the model’s knowledge capacity.
    2. Activated parameter count: the number of parameters that actually participate in computation when processing a single token. It directly affects inference compute.
    3. Number of experts and routing sparsity: more experts allow the model to express richer capabilities. Fewer activated experts usually lower inference cost, but make routing and load balancing harder.

    For example, if a model with a total parameter count at the trillion level activates only about tens of billions of parameters per token, its per-token compute cost may be close to that of a medium-sized dense model, while its theoretical knowledge capacity and task coverage are larger. This is the core reason MoE can pursue both “high capacity” and “low compute” at the same time.

    2. Four Typical MoE Architectures

    1. Shared Experts Plus Routed Experts

    This architecture consists of two parts: a set of shared experts that always participate in computation, and several routed experts dynamically selected according to the token.

    Shared experts handle general language ability, basic knowledge, and stable semantic representation. Routed experts selectively provide supplementary capabilities for different inputs. For example, mathematical reasoning, code generation, long-text understanding, and knowledge question answering may be assigned to different expert combinations.

    Its main advantage is stability. Because shared experts always participate in computation, even if the router is not accurate in some scenarios, the model still has a stable base computation path. At the same time, the shared module can reduce excessive divergence in capabilities among experts.

    But this approach also has costs: shared experts are called every time, so the model cannot fully enjoy the benefits of sparse computation. In addition, if the shared part is too large, overall compute will rise noticeably.

    2. Pure Routed MoE

    A pure routed architecture does not have fixed shared experts. All experts are dynamically selected by the router. A typical method is Top-K Gating, which calculates a matching score for each expert for each token and then selects the highest-scoring experts.

    If a model contains hundreds of experts and each token activates only two of them, most parameters remain inactive in the current computation. This approach can significantly expand total model capacity while controlling inference cost.

    The challenge of pure routed MoE is load balancing. If a large number of tokens are concentrated on a few experts, some devices become congested while other experts remain idle, ultimately reducing overall throughput. In engineering practice, load-balancing loss, capacity limits, routing noise, or dynamic scheduling mechanisms are usually introduced to prevent “popular experts” from becoming a system bottleneck.

    3. Code-Aware or Task-Aware Routing

    Some models optimize expert division based on the input task type, allowing certain experts to handle more content related to program structure, algorithm logic, API calls, and error repair.

    The key to this design is not simply labeling experts as “code experts.” Instead, it requires coordination across training data construction, routing objectives, and expert update strategies. For example, by mixing samples from different tasks, researchers can observe expert preferences for code tokens, natural language tokens, and structured text, and then use auxiliary losses to constrain experts to form relatively clear capability divisions.

    Task-aware routing helps improve performance in specialized scenarios, especially code generation, complex reasoning, and structured information extraction. But it can also create generalization issues: if expert division is too fixed, the model may not be able to flexibly combine capabilities in cross-domain tasks. Routing strategies therefore need to balance specialization and generality.

    4. Hierarchical or Multi-Level Routing

    More complex designs use two or more levels of routing. The first level determines the major category of the input, such as language understanding, mathematical reasoning, code processing, or knowledge retrieval. The second level then selects specific experts from the corresponding expert subset.

    Hierarchical routing can reduce the router’s search scope, lower the scheduling overhead of large-scale expert systems, and help form a clearer expert organization structure. However, the more levels there are, the higher the risk of error propagation: once the first-level classification is wrong, even highly capable downstream experts may not be invoked.

    3. Why Activated Parameters Deserve More Attention Than Total Parameters

    For deployers, activated parameter count is usually closer to actual compute cost than total parameter count. Total parameters mainly affect model weight storage, distributed sharding, and loading time. Activated parameters more directly affect per-token compute, memory bandwidth requirements, and inference latency.

    But it is not reasonable to assume that fewer activated parameters are always better. Too few activated parameters may mean the model invokes insufficient capability for complex tasks. Too many activated parameters weaken the cost advantage of a sparse architecture.

    Therefore, practical evaluation should consider:

    • Per-token compute;
    • Time to first token;
    • Continuous generation throughput;
    • Expert communication overhead;
    • Memory usage and weight loading time;
    • Performance changes under long context;
    • Routing stability under multi-user concurrency.

    In multi-GPU environments in particular, the bottleneck of MoE does not come only from matrix computation. Experts are usually distributed across different devices, and tokens may need to be transferred across devices. If network bandwidth is insufficient, communication time can offset the time saved by sparse computation.

    4. Expert Load Balancing Is Key to MoE Deployment

    Ideally, all experts receive a similar number of tokens. In actual training, however, the router tends to favor a small number of “high-quality” experts, creating load skew.

    Load skew causes three types of problems:

    1. Popular experts have long computation queues, increasing latency;
    2. Underused experts receive insufficient training, and their capabilities gradually degrade;
    3. Some tokens are dropped or downgraded because of expert capacity limits.

    Common solutions include adding auxiliary balancing losses, limiting per-expert capacity, introducing random routing perturbation, and using different scheduling strategies during training and inference. It is important to note that stricter balancing is not always better. Pursuing perfectly even distribution too aggressively may破坏 the natural division of labor among experts and cause all experts to gradually become similar.

    A more reasonable goal is “controlled imbalance”: allow some experts to take on more work in specific tasks, but avoid long-term, extreme traffic concentration.

    5. Why Coding Ability Can Reflect Expert Division in MoE

    Code tasks require semantic understanding, structural modeling, logical reasoning, and strict format control at the same time, making them well suited to testing whether an MoE architecture has truly formed effective expert division.

    A model with strong coding ability usually needs the following capabilities:

    • Understand the relationships among variables, functions, types, and control flow;
    • Generate complete implementations from natural language requirements;
    • Identify compilation errors, runtime errors, and boundary conditions;
    • Maintain global consistency across multiple files or long contexts;
    • Locate problems based on test results and modify code.

    If the router can stably assign relevant tokens to experts skilled in program structure and logical reasoning, the model will generally have advantages in code completion, program repair, and complex algorithm generation. But this does not mean that more “code experts” are always better. The key still lies in training data quality, collaboration among experts, and the router’s overall judgment of context.

    6. How to Choose and Evaluate MoE Models

    Enterprises and developers should not look only at the parameter scale promoted in marketing. Instead, they should build a multi-dimensional evaluation system.

    1. Task Fit

    General question answering, code generation, mathematical reasoning, long-document summarization, and structured extraction place different demands on model capabilities. Validation should use real business data rather than relying only on a single leaderboard.

    2. Deployment Conditions

    It is necessary to evaluate memory capacity, number of devices, interconnect capability, inference framework compatibility, and accuracy loss after quantization. Models with very large total parameter counts may be difficult to load efficiently on small-scale devices.

    3. Stability

    Output quality should be observed under different prompts, different input lengths, and high concurrency. Particular attention should be paid to whether expert load shows obvious skew and how latency changes in long-context scenarios.

    4. Total Cost

    Cost includes not only compute but also model storage, weight loading, cross-device communication, operations monitoring, and failure recovery. If a model with a larger parameter count has a reasonable activation ratio and stable throughput, its actual cost may not be higher than that of a smaller but compute-intensive model.

    7. Conclusion

    The real value of MoE is not simply stacking more parameters, but using routing mechanisms to let different experts handle different types of knowledge and tasks. Shared experts emphasize stability, pure routed structures emphasize sparse efficiency, task-aware routing emphasizes specialized capability, and hierarchical routing attempts to further reduce the scheduling cost of large-scale expert systems.

    Therefore, judging whether a trillion-scale model is “strong” should not rely only on total parameter count. More important is whether it can achieve high-quality routing, stable expert division, controllable load balancing, and compute efficiency in real business scenarios. For model users, understanding the relationship among total parameters, activated parameters, and system communication cost is the foundation for model selection and deployment planning.