Breaking Down Ultra-Large MoE Architectures: The Key Technologies Beyond Parameter Count

    As large language models scale from billions to hundreds of billions or even trillions of parameters, total parameter count alone no longer explains inference cost or model capability. This article examines how sparse computation, expert routing, load balancing, parallelism, and deployment strategies shape Mixture-of-Experts efficiency, and provides practical guidance for evaluating and operating MoE models.

    As large language models continue to scale, model sizes have grown from billions of parameters to hundreds of billions or even trillions. However, parameter count does not equal inference cost, nor does it directly determine model capability. The factors that truly shape efficiency often include sparse computation, expert routing, training stability, parallelism, and serving architecture.

    Mixture-of-Experts, or MoE, has become one of the major approaches for addressing these challenges. Its core idea is not to use every parameter during every inference step. Instead, the model is divided into multiple relatively independent expert modules, and each input token activates only a small number of them. This allows the model to expand its total capacity while keeping the computation required for a single inference step within a manageable range.

    1. Three Parameter Concepts That Are Easy to Confuse

    When analyzing an MoE model, at least three types of metrics should be distinguished.

    1. Total Parameter Count

    Total parameter count is the sum of the parameters in all experts, shared modules, attention layers, embedding layers, and output layers. It reflects the model’s theoretical capacity, but it does not mean that all parameters must be loaded and computed for every inference request.

    For example, a model may contain hundreds of experts, each with a large feed-forward network. Its total parameter count can therefore be extremely high. But if each token selects only two experts, far fewer parameters participate in the actual matrix computations for that token.

    2. Active Parameter Count

    Active parameter count refers to the number of parameters actually used by a token during a single forward pass. MoE models generally use a Top-K routing strategy to select a small number of experts, where K may be 1, 2, or another relatively small value.

    Active parameters directly affect:

    • Computation per token
    • Inference latency
    • Memory bandwidth pressure
    • Serving throughput
    • Compute cost per request

    For this reason, comparing models based only on total parameter count can lead to misleading conclusions. A more useful evaluation should consider total parameters, active parameters, and the number of routed experts per layer.

    3. Number of Experts and Expert Capacity

    The number of experts determines how many potential knowledge specializations the model can support. Expert capacity determines how complex a pattern each expert can learn. More experts may expand the theoretical representation space, but they also make routing, communication, and load balancing more difficult.

    If some experts are selected frequently while others are rarely used during training, the model may experience expert collapse or severe load imbalance. In that case, much of the model’s theoretical capacity remains underutilized.

    2. Three Typical MoE Architecture Patterns

    Based on whether experts are shared, how routing is performed, and how model capacity is organized, common MoE designs can be grouped into three categories.

    1. Shared Experts Plus Routed Experts

    This architecture typically includes one or more shared experts together with several dynamically routed experts. Shared experts process every token and provide general language capabilities, foundational knowledge, and stable pattern modeling. Routed experts are selected based on the input and may handle domain knowledge, complex reasoning, or specific writing styles.

    The basic flow can be represented as follows:

    1. Input token
    2. │
    3. ├── Shared experts: always active
    4. │
    5. └── Routing module: select the highest-scoring experts
    6. │
    7. ├── Expert A
    8. └── Expert B
    9. │
    10. Weighted aggregation
    11. │
    12. Output

    This design generally offers better stability. Because shared experts are always active, the model can still rely on a general-purpose path even when the routing network makes an imperfect decision. Shared modules can also reduce the need for different experts to repeatedly learn the same basic knowledge.

    The trade-off is that shared experts run for every token, so the active parameter count is not as low as in a purely sparse architecture. If the shared path is too large, it may also reduce the cost advantages of sparse computation.

    2. Purely Routed Sparse MoE

    A purely routed architecture generally does not include an always-active shared expert. Instead, the router selects a small number of candidates from the full expert pool. Each token is sent to one or more experts, and their outputs are combined using routing weights.

    A common Top-2 routing process can be represented as follows:

    1. routing_scores = Router(hidden_states)
    2. selected_experts = TopK(routing_scores, k=2)
    3. expert_outputs = Run(selected_experts, hidden_states)
    4. output = WeightedSum(expert_outputs, routing_scores)

    The main advantage of this design is stronger sparsity. As the total number of experts increases, the model’s overall capacity can expand significantly while the number of active experts per token remains relatively stable. This makes the architecture suitable for models that need to cover general knowledge, code, mathematics, long-context understanding, and multilingual capabilities at the same time.

    However, purely routed systems place greater demands on training and systems engineering. The router must prevent tokens from concentrating in only a few experts. Otherwise, some devices may become overloaded while others remain underutilized. Training commonly uses auxiliary load-balancing losses, expert capacity limits, routing noise, or capacity factors to mitigate this problem.

    3. Large Expert Pools with a Small Active Set

    Another design emphasizes very large total capacity with a small amount of computation per token. It uses a relatively large number of experts but activates only a few modules for each input.

    This architecture is suitable for applications that prioritize the model’s capability ceiling. A larger expert pool can support more fine-grained specialization, such as:

    • Factual knowledge
    • Formal reasoning
    • Program structure
    • Long-context modeling
    • Specific languages or domain-specific styles

    However, communication can become a bottleneck as the number of experts grows. In distributed training and inference, a token may need to be sent to an expert located on a remote device. The latency introduced by cross-device communication can sometimes offset the time saved through sparse computation.

    3. Routing Determines Whether MoE Is Truly Efficient

    The key question for an MoE model is not simply how many experts it has, but how it selects them.

    1. Top-1 Routing

    With Top-1 routing, each token is sent to only one expert. This minimizes computation and generally reduces communication overhead. However, the selected expert carries greater responsibility, and a routing error can directly affect output quality. Training stability therefore becomes more important.

    2. Top-2 Routing

    Top-2 routing selects two experts for each token and combines their outputs according to routing weights. It often provides a practical balance between computation cost and model quality, which is why it is widely used in engineering implementations.

    Compared with Top-1 routing, Top-2 routing reduces the impact of a single routing error, but it also increases active computation and communication.

    3. Load Balancing

    An effective routing system should distribute tokens as evenly as possible across experts. Load balancing is typically evaluated from two perspectives:

    • Whether token assignments are evenly distributed
    • Whether the total routing probability is evenly distributed

    If one expert becomes a long-term “popular expert,” it may experience computational congestion, while low-frequency experts receive insufficient training. In practice, auxiliary losses are often used to constrain the routing distribution, and each expert is assigned a maximum capacity. When that capacity is exceeded, extra tokens may be dropped, redirected, or processed through a residual path.

    4. Why Models Need a Complete Range of Sizes

    Large models are usually offered in multiple parameter configurations rather than as a single model size. The main reason is that hardware environments and application requirements vary significantly.

    Smaller models are suitable for:

    • Single-machine or edge deployment
    • Low-latency question answering
    • Batch text classification
    • Cost-sensitive applications

    Medium-sized models are suitable for:

    • Enterprise knowledge-base question answering
    • Code assistance
    • Multi-turn conversations
    • Document summarization and information extraction

    Larger models are more suitable for:

    • Complex reasoning
    • High-quality content generation
    • Long-context analysis
    • Unified multi-task services

    A larger configuration is not necessarily the best choice for every task. If the primary goal is throughput, a smaller model combined with appropriate quantization, batching, and caching may provide better cost efficiency than an ultra-large model.

    5. How to Evaluate an MoE Architecture Objectively

    An MoE model should not be evaluated using a single benchmark or parameter count. A multidimensional evaluation framework is more useful.

    1. Capability Metrics

    Relevant capabilities include knowledge question answering, mathematical reasoning, code generation, long-context understanding, multilingual processing, and instruction following. Different tasks may activate different experts, so the evaluation should cover multiple scenarios.

    2. Efficiency Metrics

    Important metrics include time to first token, per-token generation speed, concurrent throughput, memory usage, and the proportion of time spent on communication. For online services, average speed is not enough. Tail latency under high concurrency should also be measured.

    3. Stability Metrics

    These include performance variation across different input lengths, changes in routing distributions, expert load, and resource consumption during long-running operation. Some models perform well in low-concurrency tests but are affected by communication and scheduling overhead at production scale.

    4. Deployment Complexity

    Expert models generally require more complex parallelism strategies, including tensor parallelism, expert parallelism, pipeline parallelism, and data parallelism. Deployment must also account for whether experts can be evenly distributed across devices and whether loading model weights creates a startup bottleneck.

    6. Optimization Recommendations for Enterprise Deployment

    In production environments, MoE serving can be optimized in several ways.

    First, estimate inference cost primarily from active parameter count, then use total parameter count to assess memory requirements. Total parameter count determines how many weights must be stored, while active parameter count determines the main computation cost for each request. These two metrics should not be treated as interchangeable.

    Second, monitor expert load distribution. Record the number of tokens received by each expert, average processing time, and overflow ratio to identify routing imbalance as early as possible.

    Third, configure batching carefully. Dynamic batching can improve device utilization, but excessively large batches may increase scheduling complexity across different experts.

    Fourth, combine quantization with caching. Weight quantization can reduce memory consumption, while prefix caching and key-value caching can reduce repeated computation. However, precision testing is necessary to confirm that these techniques do not significantly affect critical tasks.

    Fifth, establish a tiered model strategy. Lightweight models can handle simple requests, while more complex tasks are forwarded to larger models. This approach can control overall cost without sacrificing quality where it matters.

    Conclusion

    The core value of MoE models lies in using sparse activation to achieve greater model capacity. Shared experts prioritize stability, purely routed architectures emphasize computational sparsity, and large expert pools focus on maximizing capability. None of these designs is universally superior. The best choice depends on training data, routing strategy, hardware topology, and business objectives.

    Therefore, an ultra-large model architecture should not be judged solely by its total parameter count. Evaluation should also consider active parameters, expert utilization, routing balance, communication efficiency, inference latency, and performance on real-world tasks.

    For developers and enterprises, the goal should not be to pursue the largest number. The more sustainable objective is to find the right balance among model capability, serving stability, and deployment cost.