MoE Selection Guide: Routing, Memory, and Deployment Checks
This guide explains how Mixture-of-Experts models differ from dense models in total versus active parameters, how gating, Top-K, and temperature affect expert selection, and how to compare common routing strategies and spot load imbalance. It includes the Mixtral 8x7B parameter ratio, a routing comparison table, and a minimal PyTorch gating example.
1. Dense Models and Sparse Activation
A dense model uses all of its parameters on every forward pass, so compute grows in step with parameter count. A Mixture-of-Experts (MoE) model replaces feed-forward network (FFN) layers with multiple expert subnetworks. A gating network then selects a small number of experts for each token.
Two numbers matter. Total parameter count sets model capacity and the weight size that must fit in memory. Active parameter count sets the compute for a single token and affects per-inference latency.
Take Mixtral 8x7B as an example (Mistral AI, Mixtral of Experts, 2024). Each layer has 8 experts, and each token selects 2 of them. The paper reports about 46.7 billion total parameters and about 12.9 billion active parameters per token, a ratio of about 3.6.
That ratio is only a rough reference, and the accounting method matters. Attention and embedding parameters are shared by all tokens and are used by every token, so they count as active parameters. Do not treat 3.6x as a direct cost multiplier. Real inference cost also depends on batch size, sequence length, hardware, and the inference framework.
2. Gating, Top-K, and Temperature
The gate is usually a linear layer. It maps a token’s hidden state to a score (logit) for each expert, then applies Softmax to obtain probabilities. The system takes the K experts with the highest scores and renormalizes those K probabilities as weights on the expert outputs.
Temperature T is applied to the logits: divide logits by T, then run Softmax. Softmax preserves relative order under positive scaling, so with a fixed Top-K, temperature does not change which experts are selected. It only changes the shape of the weight distribution:
- Larger T: weights are more even, and outputs from several experts are mixed in similar proportions.
- Smaller T: weights concentrate on the highest-ranked experts, and the output is closer to a single expert.
To change how many experts each token actually calls, adjust K or switch to threshold routing. Temperature only rescales weights, and it can change how many experts pass a threshold. Most public implementations do not set an extra temperature, which is equivalent to T = 1. Before changing T, compare quality and load distribution on a validation set. Do not tune it by intuition.
The table below compares three common routing strategies.
| Strategy | What sets experts per token | Role of temperature | Typical use | Main risk | Metrics to monitor |
|---|---|---|---|---|---|
| Fixed Top-K | Parameter K | Smooths weights | Most online inference with stable latency needs | Limited coverage of complex inputs | Share of load per expert |
| Threshold routing | Probability threshold | Changes how many experts exceed the threshold | Dynamic compute by input difficulty | Compute varies; latency is hard to predict | Distribution of experts per token |
| Capacity-limited routing | Top-K plus a capacity cap | Smooths weights | Distributed training with controlled expert load | Tokens over capacity are dropped or rerouted | Capacity factor, drop rate |
Capacity factor C means each expert processes at most C × (token count × K ÷ expert count) tokens. A smaller C makes load more even but drops more tokens.
The code below shows the core logic for temperature scaling and Top-K selection.
import torchimport torch.nn.functional as Fdef moe_gate_routing(hidden_states, gate_weight, temperature=1.0, top_k=2):"""hidden_states: [num_tokens, hidden_dim], on the same device as gate_weightgate_weight: [hidden_dim, num_experts]returns: top_k_indices [num_tokens, top_k], top_k_weights [num_tokens, top_k]"""assert hidden_states.device == gate_weight.device, "inputs and gate weights must be on the same device"num_experts = gate_weight.shape[-1]assert temperature > 0, "temperature must be positive"assert 1 <= top_k <= num_experts, "top_k must be in [1, number of experts]"# Gate in float32 to reduce ranking jitter among similar probabilitieslogits = torch.matmul(hidden_states.float(), gate_weight.float())probs = F.softmax(logits / temperature, dim=-1)top_k_probs, top_k_indices = torch.topk(probs, top_k, dim=-1)top_k_probs = top_k_probs / top_k_probs.sum(dim=-1, keepdim=True)return top_k_indices, top_k_probs
Before running this code, the following conditions must hold:
- Inputs and gate weights must be on the same device, or a device-mismatch error is raised.
- Gating uses float32. Some implementations do this to reduce ranking jitter among similar probabilities. Frameworks differ; follow the documentation of the framework you use.
- Common values of top_k are 1 or 2. Mixtral uses 2. Some newer models select more experts per token.
- The default temperature is 1.0. Change it only after experiments show a benefit.
3. Three Engineering Issues and How to Check Them
1. Load balance
Symptom: a few experts receive most tokens while others stay nearly idle. In multi-GPU training, some GPUs wait for long periods.
Cause: the gate tends to keep selecting a small set of experts, which reinforces itself, so experts do not specialize enough. Skewed input distributions make this worse.
Checks:
- Count tokens assigned to each expert in a batch and compute the maximum share.
