Mixture-of-Experts Architecture: Sparse Activation, Gating, and Routing Explained
This article explains why Mixture-of-Experts (MoE) decouples model capacity from per-inference compute, describes the core expert, gating, and combination structure, and walks through Top-K routing with softmax temperature. It also covers load balancing and engineering considerations such as all-to-all communication, memory planning, and batching efficiency, then compares MoE with principles-based alignment and data/knowledge augmentation as complementary controllability approaches.
1. Why Mixture-of-Experts Architecture Is Needed
A common way to improve model capability is to widen or deepen the network. However, growth in parameter count directly increases the computation and memory required for each inference. In a dense model, every input passes through all parameters, so compute cost grows approximately linearly with parameter scale.
Mixture of Experts (MoE) offers another approach: split the model into multiple expert subnetworks and activate only a few of them for each input. The total parameter count can be large while per-inference computation remains controllable. This is the core value of sparse activation.
Understanding this requires distinguishing two concepts. The total parameter count determines model capacity and knowledge storage, while the activated parameter count determines inference cost. The lower the ratio between them, the more model capacity is carried per unit of compute.
2. Core Structure
A typical MoE layer contains three parts:
- Experts: usually feed-forward subnetworks with the same structure, each with independent parameters.
- Gating Network (also called Router): computes a score for each expert based on the input representation.
- Combination Module: produces the layer output by taking a weighted sum of the selected expert outputs.
In a Transformer, the MoE layer usually replaces the feed-forward layer, while the attention layer remains dense.
3. Gating and Top-K Routing
The gating network computation can be divided into four steps:
- Apply a linear transformation to obtain logits:
z = W_g · h, wherehis the input representation and the output dimension equals the number of experts. - Normalize with softmax to obtain probabilities:
p_i = exp(z_i / T) / Σ_j exp(z_j / T), whereTis the temperature coefficient. - Select the K experts with the highest probabilities and renormalize those K probabilities.
- Produce the output as
y = Σ_{i∈TopK} p̂_i · E_i(h), whereE_idenotes the computation of the i-th expert.
For example, assume there are 8 experts and K is 2. After gating, a token has two highest probabilities of 0.46 and 0.31. After renormalization, the weights are approximately 0.597 and 0.403, and the other 6 experts do not participate in computation for this token.
The following schematic code shows the core logic of single-layer routing:
import torchimport torch.nn.functional as Fdef topk_route(hidden, gate_weight, k=2, temperature=1.0):# hidden: [tokens, d_model], gate_weight: [num_experts, d_model]logits = hidden @ gate_weight.T # [tokens, num_experts]probs = F.softmax(logits / temperature, dim=-1) # Normalize after temperature scalingtopk_p, topk_idx = torch.topk(probs, k, dim=-1) # Select K experts per tokentopk_p = topk_p / topk_p.sum(dim=-1, keepdim=True) # Renormalize selected weightsreturn topk_idx, topk_p
This code omits expert computation and output merging. Its focus is the routing decision itself.
4. The Role of Softmax Temperature
The temperature coefficient T adjusts the shape of the probability distribution. It does not change the hard constraint of selecting K experts; it only affects the weight distribution and the stability of selection.
- When
Tis large, the distribution becomes more uniform. Multiple experts receive similar scores, selection results are more sensitive to small input changes, and load distribution becomes more dispersed. - When
Tis small, the distribution becomes sharp. Weights concentrate on a few experts, and routing becomes more deterministic, but some experts may remain idle for long periods while others are persistently overloaded.
In engineering practice, temperature is usually used as a tunable hyperparameter, and some practitioners adjust it gradually during training. When tuning it, validation-set metrics and expert utilization should be observed together. Looking at a single metric alone can easily lead to one-sided conclusions.
5. Load Balancing
Routers are prone to a “the strong get stronger” effect. When certain experts are selected frequently, they receive more gradients and become stronger, which makes them even more likely to be selected and creates positive feedback. There are three common ways to address this:
- Auxiliary loss: penalize the product of expert selection frequency and gating probability to encourage more uniform use of experts.
- Expert capacity limit (capacity factor): limit the maximum number of tokens each expert can process in a batch. Tokens beyond the limit can skip that expert, with the original input preserved only through the residual connection.
- Routing noise: add a small amount of random noise to logits during training to break deterministic routing bias and give more experts training opportunities.
6. Engineering Deployment Considerations
- Communication overhead: when experts are distributed across multiple devices, tokens need to be dispatched to the corresponding experts through all-to-all communication and then collected back. Network bandwidth is often the performance bottleneck.
- Memory planning: although only some experts are used in each computation, all expert parameters still need to reside in memory or be loaded through sharding. The memory budget should be estimated based on total parameter count, not activated parameter count.
- Batching efficiency: different tokens are routed to different experts, making intra-batch computation irregular. In practice, tokens are usually grouped and reordered by expert so that each expert processes a continuous batch of data at once, improving compute utilization.
- Training stability: the Top-K selection operation is non-differentiable, and gradients flow back only through the gating weights of selected experts. Therefore, the learning rate and auxiliary loss weight need to be tuned jointly; otherwise, routing can easily collapse.
- Evaluation metrics: beyond task accuracy, monitor expert utilization distribution, token drop rate, and average latency per token. Together, these metrics determine whether the system truly meets its design goals.
7. Comparison with Other Controllability Paths
Controllability in large models does not come only from architecture design. In industry, there are mainly two approaches, and they can complement each other.
- Principles-based alignment: write behavioral guidelines explicitly as rules, let the model review and revise its own outputs according to those rules, and use preference data to optimize output tendencies. This approach is well suited to defining behavioral boundaries.
- Data and knowledge augmentation: perform domain cleaning and supplement localized corpora at the training-data level, and introduce structured knowledge such as knowledge graphs to provide factual constraints for generation. This approach is well suited to improving factual accuracy and contextual fit.
In real systems, both are often used at the same time. The model internally learns value ordering, while external retrieval or knowledge bases handle factual verification. This division of labor covers a wider range of issues more completely.
8. Summary
MoE decouples parameter scale from per-inference cost through sparse activation. The gating network determines routing direction, softmax temperature adjusts weight distribution, Top-K limits the computation scope of each token, and load-balancing mechanisms ensure that all experts are used effectively. In deployment, communication bandwidth, memory planning, and batching efficiency must also be considered together. A shortfall in any one of these can offset the benefits of sparsity.
