MoE LLM Architecture: Sparse Activation and Attention Optimization
As public benchmark scores converge, architecture choices often matter more than leaderboard rankings for real-world deployment. This article explains how MoE models trade sparse activation for capacity, how routing and load balancing work, why KV cache grows in long-context inference, how attention compression and wide-backbone/narrow-expert designs affect efficiency, and how to compare MoE and dense models using measurable engineering metrics.
1. Why Architecture Matters More Than Benchmark Scores: Choosing Between MoE and Dense Models
As public evaluation scores gradually converge, mainstream large language models may differ by only a few percentage points on leaderboards. However, their differences in inference cost, deployment difficulty, scalability, and long-text support often matter more than leaderboard scores when deciding whether a model is practical.
For developers, the key questions are not simply “which model has the higher score,” but:
- How should you choose between MoE and Dense models?
- How do activated parameters affect deployment cost?
- Why does KV cache expand rapidly in long-context scenarios?
The core architectural differences of modern large language models mainly come from two design areas:
- Parameter usage: Dense models activate all parameters for every token, while Mixture-of-Experts (MoE) models activate only a subset of expert networks for each token.
- Attention and inference optimization: This focuses on how to balance representation capability, compute cost, and memory usage.
Understanding these architectural trade-offs is the foundation for model selection, cost estimation, and deployment tuning.
2. MoE: Trading Sparse Activation for Capacity
2.1 Sparse Activation vs. Dense Models
In a traditional Dense model, every input token passes through all parameters in the network. Model capability generally increases as parameter count grows, but inference cost also rises with parameter count.
Mixture-of-Experts (MoE) introduces sparse activation. In an MoE model, the total parameter count can be very large, but each token only passes through a small subset of expert networks.
For example, in some public MoE models, the total parameter count may reach tens of billions, hundreds of billions, or even more, while the number of parameters actually activated per token may be only a fraction of the total.
The key benefit of this design is decoupling “model capacity” from “compute per forward pass”: the model can store more knowledge in its parameters without activating all parameters for every token.
However, this benefit is not unconditional. The actual gains of MoE depend on routing quality, expert load balance, communication bandwidth, memory capacity, and inference framework optimization. If routing is unstable, expert load is severely imbalanced, or expert-parallel communication overhead is too high, the theoretical advantages of MoE may be offset by engineering costs.
2.2 Routing Mechanism: Pseudocode, Tensor Shapes, and Load Balancing
The router is central to MoE. It computes scores for each expert for every token and selects the Top-K experts to process that token.
The example below is pseudocode. It focuses on tensor shapes and routing flow rather than production-level efficiency.
import torchdef moe_router(x, router, top_k=2):# x: [B, S, D]# router outputs a score for each of E experts for each tokenlogits = router(x) # [B, S, E]topk_scores, topk_ids = logits.topk(top_k, dim=-1)# topk_scores: [B, S, K]# topk_ids: [B, S, K]gate = torch.softmax(topk_scores, dim=-1) # [B, S, K]return gate, topk_idsdef dispatch_to_expert(x, expert_ids, experts):"""Pseudocode: dispatch tokens to their corresponding experts according to expert_ids.x: [B, S, D]expert_ids: [B, S]experts: list[Module]. Each expert takes [B, S, D] and returns [B, S, D].Production implementations usually use grouping, expert capacity limits,token dropping/fallback, and expert-parallel communicationinstead of looping over tokens one by one."""B, S, D = x.shapeout = torch.zeros_like(x)for b in range(B):for s in range(S):e = int(expert_ids[b, s])out[b, s] = experts[e](x[b:b+1, s:s+1])[0, 0]return outdef moe_forward(x, experts, router, top_k=2):# x: [B, S, D]gate, topk_ids = moe_router(x, router, top_k)# gate: [B, S, K]# topk_ids: [B, S, K]out = torch.zeros_like(x) # [B, S, D]for k in range(top_k):expert_id = topk_ids[..., k] # [B, S]y = dispatch_to_expert(x, expert_id, experts) # [B, S, D]out = out + gate[..., k].unsqueeze(-1) * yreturn out
In real systems, the router must do more than select experts. It also has to handle expert capacity, load balancing, and training stability.
A common approach is to define an expert capacity factor. If there are (N) tokens, (E) experts, and Top-K = (K), the total number of assignments is approximately (N \times K). The theoretical average number of assignments per expert is approximately (N \times K / E). The configured capacity can be:
capacity = ceil(N * K / E * capacity_factor)
If an expert receives more tokens than its capacity, the system may:
- drop overflow tokens,
- keep the original input,
- reroute tokens to other experts,
- or use a residual fallback path.
Different strategies affect quality and latency, so they must be validated against the specific model and serving framework.
To reduce expert hotspots or idle experts, training often includes a load-balancing loss. A simplified form is:
L_balance = α * E * Σ(f_i * p_i)
Here:
- (f_i) is the fraction of tokens routed to expert (i);
- (p_i) is the average router probability assigned to expert (i);
- (E) is the number of experts;
- (\alpha) is a weighting factor.
The goal of this loss is to encourage tokens to be distributed more evenly across experts and prevent a small number of experts from handling most of the computation over long periods.
3. Attention Mechanisms: Balancing Compute and Representation
3.1 Why KV Cache Grows in Long-Context Scenarios
Attention layers must balance representation capability and computational overhead. As sequence length increases, KV cache grows significantly and becomes a major bottleneck for long-text inference.
For ordinary multi-head attention, KV cache can be roughly estimated as:
KV Cache ≈ 2 × L × S × H_kv × d_head × bytes_per_element
Where:
2represents Key and Value;- (L) is the number of layers;
- (S) is the sequence length;
- (H_{kv}) is the number of KV heads;
- (d_{head}) is the dimension per head;
bytes_per_elementis the number of bytes per value.
If FP16 is used, this is usually 2 bytes. If INT8 or quantized KV cache is used, it should be adjusted according to the quantization format.
This means that sequence length, layer count, number of attention heads, and per-head dimension all directly affect KV cache size.
In long-text scenarios, KV cache can become a major pressure point for memory and bandwidth even when the total model parameter count is not especially large.
A class of techniques called Multi-head Latent Attention (MLA) reduces KV cache size by compressing Key-Value representations into a lower-dimensional latent space.
Its benefit depends on:
- the compression dimension,
- the attention implementation,
- kernel optimization,
- and inference framework support.
After latent compression, KV cache is no longer simply determined by “number of heads × head dimension,” and may instead be driven mainly by the latent dimension.
Therefore, when evaluating MLA, you should look beyond the name and examine the actual cache layout, decode-phase memory usage, and long-context throughput.
3.2 Wide Backbone and Narrow Experts: An Example Structure
In some public model configurations or community reproduction configurations, you may see a structure with a relatively wide backbone hidden dimension and relatively narrow hidden dimensions per expert.
For example, some examples may show:
- backbone hidden dimension: about 7168;
- expert hidden dimension: about 2048.
These numbers are only examples and should not be treated as a universal rule for all MoE models.
Different model families, training objectives, expert counts, activation ratios, and inference frameworks can all affect the final configuration.
This type of structure may provide several potential benefits:
- A wide backbone network can retain expressive capacity for global features.
- Narrower expert hidden dimensions can reduce compute per activated expert.
- With proper expert parallelism and communication optimization, per-token compute may be reduced.
- If combined with low-rank attention or KV compression, it may also alleviate long-context memory pressure.
However, these benefits are not automatic.
If communication between experts is frequent, routing load is imbalanced, kernels are not optimized for narrow experts, or memory is mainly consumed by inactive expert weights, the advantages of a “wide backbone, narrow experts” structure may not be fully realized.
When evaluating this kind of structure, it is better to observe per-token compute, peak memory usage, communication overhead, and end-to-end throughput together.
4. Sparsity and Scaling Laws: Benefits and Limits of Adding More Experts
Scaling law research suggests that, under controllable total compute budgets and training stability, increasing sparsity can improve model capacity.
Here, sparsity means increasing the ratio of total parameters to activated parameters.
This explains why many MoE designs increase the number of experts: they expand the amount of storable knowledge while keeping activated parameters unchanged or only slightly increased.
However, “more experts is always better” is not an unconditional conclusion.
Increasing the number of experts introduces several engineering problems:
- Routing becomes harder, and load balancing becomes more difficult.
- Expert-parallel communication increases, especially all-to-all or token dispatch overhead.
- Inactive experts still occupy memory, so memory estimation cannot rely only on activated parameters.
- Training stability may degrade, requiring more careful initialization, normalization, router loss, and capacity control.
- If expert sharding is poorly designed during inference, tail latency may become worse.
Therefore, sparsity design should look for a turning point rather than blindly increasing the number of experts.
A reasonable approach is to jointly evaluate:
- total parameters,
- activated parameters,
- training budget,
- inference memory,
- communication bandwidth,
- and routing stability.
5. Inference Efficiency: What Are the Risks of Changing the Number of Attention Heads?
The number of attention heads is often discussed but easily misunderstood.
A common misconception is that reducing the number of attention heads will necessarily reduce compute and memory usage.
The actual effect depends on whether q/k/v widths, hidden dimensions, or per-head dimensions are also changed.
If a model originally has many attention heads, and reducing the head count is accompanied by reducing q/k/v projection width or hidden dimension, then compute per step and some memory usage may decrease.
However, if only the number of heads is reduced while total width is kept constant—for example, by increasing per-head dimension to maintain the same q/k/v total width—then compute and KV cache may not decrease significantly.
In that case, the benefit may mainly come from implementation-level scheduling simplification rather than a reduction in theoretical compute.
Reducing the number of attention heads may also introduce risks:
- Representation capability may decline, potentially affecting complex tasks.
- Pattern diversity in multi-head attention may be reduced.
- If projection layers are not adjusted accordingly, some implementations may not achieve the expected memory savings.
- In long-context scenarios, if KV cache still grows with head count and head dimension, memory pressure may not be relieved.
A more careful statement is therefore:
Reducing the number of attention heads may bring inference benefits, but it should be combined with synchronized adjustments to q/k/v width, hidden dimension, or head dimension, and validated through stress testing.
Recommended validation steps include:
- Accuracy validation: compare perplexity, downstream task performance, and small-scale evaluation sets before and after the change.
- Throughput validation: measure tokens/s under fixed batch size and fixed input/output lengths.
- Latency validation: observe time-to-first-token, per-token latency, and P95/P99 latency separately.
- Memory validation: measure peak usage of weights, KV cache, activations, and temporary communication buffers.
- Long-context validation: test at sequence lengths such as 8K, 32K, and 128K.
- Batching validation: test both single-concurrency and multi-concurrency scenarios to avoid drawing misleading conclusions from single-request results.
6. How to Systematically Compare Different Architectures
When evaluating two large model architectures, you should not look only at total parameter count or leaderboard scores.
It is better to build a comparison across dimensions such as capacity, sparsity, attention, memory, and latency.
| Metric | Formula or estimation method | Measurement method | Risks or common mistakes |
|---|---|---|---|
| Total parameters vs. activated parameters | Activated parameters ≈ total parameters × per-token activation ratio | Check the model configuration and confirm the actual activation path in the forward pass | Compute estimation cannot rely only on total parameters, but memory estimation also cannot look only at activated parameters |
| Expert count and routing strategy | Top-K, router loss, capacity factor | Measure token assignment distribution, expert utilization, and overflow rate | Overheated experts can cause idle capacity and higher tail latency |
| KV cache | Normal attention: 2 × L × S × H_kv × d_head × bytes_per_element; compressed attention: estimate based on latent dimension |
Measure memory usage at different sequence lengths in an actual inference engine | Long-text scenarios cannot be evaluated by model parameters alone |
| Attention heads and hidden dimension | Hidden dimension is usually approximately equal to number of heads × per-head dimension | Compare q/k/v width, KV cache, and actual throughput | Reducing head count without reducing width may bring limited benefit |
| Throughput and latency | tokens/s, TTFT, TPOT | Use realistic business lengths and concurrency for stress testing | Paper specifications cannot replace end-to-end performance |
| Memory usage | Weights + KV cache + activations + communication buffers | Monitor an actual inference service in production-like conditions | Inactive MoE expert weights may still reside in memory |
The purpose of this comparison is to turn architectural questions into measurable engineering metrics.
Only when design items such as “number of experts,” “number of heads,” and “hidden dimension” are mapped to formulas, measurement methods, and risk controls can you form reproducible engineering judgments.
7. Practical Recommendations
Choose the architecture based on the scenario first: MoE or Dense?
For knowledge-intensive tasks where higher memory usage and more complex deployment are acceptable, MoE’s sparse capacity may be more valuable.
If low latency, low memory usage, and simple deployment are priorities, Dense models or MoE models with smaller activated parameters may be safer.
Long-text and high-concurrency scenarios require particular attention to KV cache, attention compression, and memory bandwidth.
How to estimate activated parameters?
Do not rely only on model names or promotional parameter counts.
Check the configuration for:
- total parameters,
- activated parameters,
- number of experts,
- Top-K,
- hidden dimensions,
- and per-layer structure.
During deployment, activated parameters are often a better starting point for compute estimation, but memory estimation must also include:
- all expert weights,
- KV cache,
- activations,
- and communication buffers.
What should a MoE deployment checklist include?
At minimum, check:
- expert-parallel communication overhead,
- token dispatch cost,
- expert capacity factor,
- routing load balance,
- overflow token handling,
- KV cache strategy,
- quantization support,
- and long-context compression support.
Compared with vague statements such as “using cloud platform capabilities,” these are the direct key items for MoE deployment.
Stress test with real business workloads, not just paper specifications.
Stress tests should cover different:
- input lengths,
- output lengths,
- concurrency levels,
- and batching strategies.
Focus on:
- throughput,
- time-to-first-token,
- per-token latency,
- peak memory usage,
- expert utilization,
- and long-text stability.
Only after validation under realistic workloads can you determine whether an architectural advantage can translate into service-level benefits.
8. Conclusion
When large models become hard to separate on public leaderboards, real technical competitiveness is reflected in every architectural trade-off:
- capacity from sparse activation,
- stability from routing and load balancing,
- long-text efficiency from latent attention,
- and the balance between expert count and attention head count.
Understanding these underlying mechanisms and translating them into estimable, measurable, and testable engineering metrics is essential for making selection and deployment decisions that balance cost, latency, and quality.
