Large-Scale Mixture-of-Experts (MoE) Architecture: Key Design Considerations
Mixture-of-Experts separates "how big the model is" from "how much compute each pass uses," decoupling total parameters (knowledge capacity) from activated parameters (latency and throughput). This article walks through the engineering tradeoffs behind trillion-parameter, few-billion-activated MoE models: capacity versus activation versus quality, routing design beyond Top-k, shared versus purely routed experts, encouraging genuine expert specialization, training stability at scale, and inference cost and system shape. It closes with a four-question framework for choosing an MoE architecture based on latency budget, capability gaps, deployment coverage, and cluster memory and communication limits.
Mixture-of-Experts (MoE) separates “how big the model is” from “how much compute each pass uses”: total parameters determine knowledge capacity, while activated parameters determine the compute and latency of a single forward pass. Trillion-scale total parameters with a few billion activated are not simply a dense network scaled up—they are a set of mutually constraining choices across routing, expert division of labor, training stability, and inference systems. The discussion below follows the problems you actually hit in engineering, without being tied to any specific model.
1. Distinguishing capacity, activation, and quality
A dense model uses all of its parameters at every step, so capability and cost are almost linearly bound. MoE replaces the feed-forward network with a group of experts in some layers; a router selects a few experts for each token, and the rest perform no matrix multiplication on that token. This gives rise to three quantities that are easy to conflate:
- Total parameters: the sum of all experts and the backbone, reflecting the capacity to store patterns.
- Activated parameters: the parameters actually involved in computing a single token, closer to what drives latency, throughput, and memory-bandwidth pressure.
- Effective training volume: the high-quality tokens each expert actually sees. No matter how many experts there are, if they go without data for a long time, they are only “nominal capacity.”
So “total parameters far exceeding activated parameters” only means the architecture has the potential for sparse computation; it does not automatically mean better quality. When evaluating, look at whether the task distribution is covered by experts, whether routing is stable, and whether the activated scale matches the target latency. Directly comparing “who is bigger” between two systems with similar activated parameters often leads to misleading conclusions: the side with larger total capacity relies more on routing to send tokens to the right experts, and once routing drifts, the extra parameters sit idle.
2. Routing: Top-k is just the starting point
The most common gating in the industry scores experts and takes the top k (Top-k), where k is usually small to keep activation under control. The scores generally come from a linear projection of the token’s hidden state, then normalized into weights, and the selected experts’ outputs are summed by weight. The sketch below only illustrates the data flow, not a specific framework’s interface:
scores = router(hidden_state) # [num_experts]topk_idx, topk_w = topk(softmax(scores), k)y = sum(topk_w[i] * expert[topk_idx[i]](hidden_state))
The tradeoff for k is straightforward: a larger k lets each token blend more experts for smoother representation, but activated parameters, communication, and memory bandwidth all rise together; too small a k makes the cost of misrouting higher, since one misassigned expert almost single-handedly determines that token’s feed-forward result. In practice you also handle ties, noise, and temperature: in early training the scores are often near-uniform, and if they harden into hard selection too early, experts differentiate prematurely or even collapse. A common approach is to keep light noise or an auxiliary loss during training and use deterministic Top-k at inference, avoiding jitter in production results.
Routing needs to not only “select accurately” but also “separate cleanly.” If a large number of tokens continuously flood a few experts, the rest get sparse gradients, the capacity advantage disappears, and the popular experts become compute and communication hotspots. Load balancing is therefore not a tuning decoration but a prerequisite for MoE to train at all. Common constraints include: expert selection frequency within a batch stays close to uniform, each expert has an upper limit on its token quota, and an auxiliary loss penalizes over-concentrated routing. A quota that is too strict hurts specialization; too loose and hotspots reappear—so this must be judged against the task distribution rather than a single validation set.
3. Shared experts vs. purely routed experts
One stabilizing approach is “a few always-on shared experts plus per-token routed experts.” Shared experts participate for every token, carrying general language patterns, formatting, and common sense; routed experts are activated only on specific tokens, carrying incremental capability. The benefit is that even when routing is temporarily inaccurate, the shared pathway still holds up baseline quality, and performance degrades more gently when expert load is uneven. The cost is that shared experts are computed every step, so a portion of the activation budget cannot be made sparse, and the total number of experts usually doesn’t get very large.
Another approach relies almost entirely on routed experts, with few or no shared experts, turning as much of the parameter budget as possible into switchable capacity. It fits the scaling route of “large total parameters, controlled activation,” and makes it easy to produce multiple sizes from the same training stack: small variants reduce the number of experts or narrow expert width, large variants increase the number of experts, while the backbone and data recipe stay consistent—convenient for choosing a size by hardware. The risks are equally clear: if routing and balancing fail, there’s no shared pathway to fall back on, so performance depends more heavily on gating quality; and if you only change the number of experts across variants without re-tuning the learning rate, batch size, and balancing weights, the optimal hyperparameters for small and large models won’t automatically align.
Neither route wins outright. Shared experts lean toward stability and predictable latency; pure routing leans toward capacity elasticity and size coverage. When choosing, first fix the activation budget (determined by latency and GPU count), then decide how much of that budget must always stay on.
4. Making experts specialize rather than randomly partition
Expert differentiation comes from the interaction of data and routing, not from cutting matrices into blocks. If code, long documents, dialogue, and tool calls differ by an order of magnitude in proportion in the training corpus, the router sends the high-frequency patterns that are easy to score to a few experts, while low-frequency but high-value capabilities stay chronically undertrained. On the engineering side, you can sample by domain proportion at the data-recipe stage so each expert has a chance to see enough gradient; you can also introduce task cues into the routing features (for example, whether the current segment is structured text or contains strong syntactic constraints), making it easier for gating to gather similar tokens into the same group of experts. This kind of “task-aware routing” explains a common phenomenon: with similar total activated parameters, a system that has co-designed routing and data for a particular task clearly outperforms one where “experts are just randomly sharded.”
Specialization has limits too. Experts that are too narrow hurt cross-domain combinations—for example, when a response must both narrate and modify code, tokens bounce between experts, increasing communication and creating style inconsistency. A more robust design is “coarse-grained domains plus allowing multiple experts to still compete within a domain,” rather than one person one specialty with no overlap. When evaluating, look at general capability and specialized capability separately, to avoid a single aggregate score masking weak spots.
5. Training stability: after scaling up, routing often breaks first
When MoE scales up, the instabilities already present in dense models get amplified: loss spikes, expert collapse, inconsistent routing statistics across data-parallel replicas, and biased gradients from dropped tokens (capacity overflow). Actionable principles include:
- Treat the balancing loss, capacity factor, and routing temperature as hyperparameters on par with the learning rate, not as defaults.
- Monitor each expert’s selection rate, average weight, and output norm, rather than only the total loss. A selection rate that stays near zero or near one for a long time both signal failed division of labor.
- Under expert parallelism and data parallelism, aggregate balancing statistics over the correct communication domain; otherwise each replica “looks balanced” on its own while the global picture is highly imbalanced.
- Make the overflow policy explicit: whether tokens exceeding capacity are dropped to the residual, reassigned to shared experts, or deferred to the next step changes the gradient bias and must match the evaluation methodology.
Shared-expert structures are usually more resilient to these disturbances; a pure large expert pool demands tighter monitoring. Either way, don’t substitute an unverified “more parameters means more stable” for the curves.
6. Inference: activated parameters set the bill, total parameters set the system shape
Serving cost is mainly determined by activated compute, movement of expert weights, and collective communication. If expert weights can’t stay resident on every accelerator, and each step has to fetch the selected experts from elsewhere, then bandwidth and topology often become the bottleneck before FLOPs do. Expert parallelism places different experts on different devices, and all-to-all sends tokens to the device hosting the expert and back; when tokens are few and experts are many, communication startup overhead can eat up the time saved by sparse computation. So small-batch, low-latency scenarios are sometimes better served by a moderately dense or fewer-experts configuration.
Memory has to be planned on the same side: backbone weights, resident experts, the KV cache, and temporary buffers for routing and balancing. As context grows longer, KV-cache growth is unrelated to expert weights—no amount of activation sparsity saves the attention cache. Capacity planning should compute “whether experts are resident” and “sequence length” separately, then combine them into single-GPU and cluster budgets.
For the serving form, you can use a container platform to host multiple replicas, and use monitoring and alerting to track expert popularity, queue latency, and overflow rate. On Baidu AI Cloud, this kind of observability can rest on general container platforms plus logging and monitoring/alerting capabilities: what you’re watching is whether expert load is imbalanced and whether tail latency is caused by communication, not just average QPS. Once you find a persistent hotspot, go back to the training or fine-tuning stage to adjust data and balancing, rather than just replicating the hotspot experts a few more times in production—replication can relieve queuing but won’t make the cold experts make up for their undertrained capability.
7. How to make the architecture choice
You can compress the decision into four questions:
- How large an activation does the latency budget allow? From this, work backward to k, expert width, and whether to keep shared experts.
- Is the capability gap in general stability or a specific specialty? When a specialty is clearly lacking, prioritize the data recipe and routing features over blindly adding experts.
- Do you need one recipe to cover multiple deployment sizes? If so, unify the backbone and tokenizer, use expert scale as the tier, and re-tune balancing and learning rate for each tier.
- Can cluster communication and memory support “weights not resident”? If not, reduce the number of experts or raise the resident fraction, accepting lower total parameters.
MoE’s strength lies not in any isolated parameter number but in whether capacity, activation, routing quality, and system bandwidth are aligned. Shared experts trade for stability, pure routing trades for elasticity, task-aware routing trades for specialized performance, and load balancing determines whether this paper capacity can become usable gradients. Designing and observing under the above constraints gets you closer to a deliverable system than chasing a single scale metric.
