MoE LLM Architecture Explained: Total Parameters vs. Activated Parameters vs. Context Window

    Mixture of Experts (MoE) models break the habit of judging an LLM's scale by parameter count alone. This article separates three metrics that are often conflated—total parameters, per-token activated parameters, and context window—explains how the router selects experts for sparse activation, and covers why small activated parameters do not automatically mean low inference latency. It closes with the engineering trade-offs to weigh when deploying and selecting an MoE model.

    Starting With a Counterintuitive Observation

    When people talk about the “size” of a large language model, they tend to treat the total parameter count as the only yardstick. With sparse Mixture of Experts (MoE) architectures, that yardstick stops working.

    “Large” means at least two different things for an MoE model. On one hand, it may hold an enormous pool of experts, with total weights in the hundreds of billions or even trillions. On the other hand, when processing any single token, only a small fraction of that pool actually participates in computation. So a model with “1T total parameters, 32B activated per token” does not run all 1T parameters for every token. Conversely, a model with “80B total parameters, 3B activated per token” should not be dismissed as a 3B-capability model.

    The confusion comes from mixing three key metrics: total parameters, per-token activated parameters, and context window. To actually read an MoE model correctly, you need to keep these three measurements strictly separate.

    1. Aligning the Metrics: Total Parameters, Activated Parameters, and Context Window Are Not the Same Thing

    Total parameters: knowledge capacity and deployment cost

    Total parameters refers to the sum of parameters across every expert network, attention layer, embedding table, gating router, and supporting module in the model repository.

    This number sets the model’s theoretical ceiling: a larger expert pool implies richer knowledge coverage and greater potential for specialization. But the costs are equally clear:

    • High weight-loading cost: all experts must reside in GPU memory or host memory. Even if a token touches only a few of them, the storage and loading overhead cannot be avoided.
    • Complex distributed deployment: as parameters grow, the combination of tensor parallelism, pipeline parallelism, and expert parallelism becomes more involved, and the communication topology requires finer design.
    • Large GPU/host memory footprint: this is directly tied to “small activated parameters” and is also the cost item most easily overlooked when deploying MoE models.

    A common mistake is to assume that because activated parameters are small, the model can run on a single GPU. In practice, total parameters determine whether the model can fit into memory at all, and that is often the first hurdle of deployment.

    Per-token activated parameters: the compute budget for a single forward pass

    The key mechanism of MoE is that it does not run the full model every time. Instead, a router dynamically selects a subset of experts for each token. This figure is closer to a “per-token forward compute budget,” and it is used to approximate compute consumption.

    For example (these numbers illustrate the metric only and do not refer to any specific product):

    • If a model has 1T total parameters and activates 32B, the forward compute for a single token is roughly comparable to that of a 32B dense model, even though its underlying knowledge capacity far exceeds 32B.
    • If a model compresses activated parameters to around 3B, per-token compute is indeed very small, yet it still has to maintain over a hundred gigabytes of total weights.

    The important point: activated parameters measure the order of magnitude of compute, not the ceiling of capability. The compute savings from sparse activation are offset by the capacity that an enormous expert pool provides.

    Context window: how long a sequence the model can handle

    The context window belongs to a different dimension from the other two. It constrains how long an input sequence (in tokens) the model can take in at once, and it is determined jointly by the positional encoding scheme, the attention implementation (for example, whether linear or sparse attention is used), and the memory budget at inference time.

    For engineering practice, the context window directly determines how much document length, how many conversation turns, and how much knowledge a single request can carry. It cannot be solved by stacking more total parameters, and it does not grow automatically because activated parameters are small. It has to be considered separately in both architecture design and inference configuration.

    2. How MoE Works: Routers and Experts

    To understand why the metrics above hold, you need to go back to the core structure of MoE. In a dense model, every token passes through the same feed-forward network (FFN). MoE replaces that single FFN layer with several parallel expert networks and adds a gating router that decides which experts handle each token.

    A typical flow looks like this (illustrative pseudocode):

    1. def moe_layer(x):
    2. # x: [num_tokens, hidden_dim]
    3. # 1) The router scores every token against every expert
    4. logits = router(x) # [num_tokens, num_experts]
    5. topk_weights, topk_idx = topk(softmax(logits), k=2) # select the Top-K experts
    6. # 2) Compute only the selected experts, achieving sparse activation
    7. output = zeros_like(x)
    8. for token_i in range(x.shape[0]):
    9. for w, e in zip(topk_weights[token_i], topk_idx[token_i]):
    10. output[token_i] += w * expert[e](x[token_i])
    11. # 3) Load-balancing loss, encouraging tokens to spread evenly across experts
    12. loss += load_balancing_loss(logits)
    13. return output

    Several engineering details matter here:

    • Top-K selection: k = 1 or k = 2 is common. A larger k increases activated parameters and may improve quality, but raises compute and communication overhead.
    • Sparse activation: only the selected experts participate in computation. This is exactly where “activated parameters far smaller than total parameters” comes from.
    • Load balancing: if all tokens flow toward a few experts, most experts sit idle and compute utilization drops sharply. Training therefore usually adds an auxiliary loss that pushes the routing distribution toward uniform.
    • Shared experts: some designs include shared experts that always participate in computation and carry general knowledge, while the sparse experts handle domain-specific specialization.

    3. Why Small Activated Parameters Do Not Mean Low Latency

    This is one of the most commonly misunderstood points in engineering. Small activated parameters mean a small compute magnitude, but real inference latency is shaped by several additional factors:

    • Communication overhead: when experts are sharded across devices (expert parallelism), each token must be routed between devices, and all-to-all communication becomes the bottleneck. The fewer and more scattered the activated experts, the more frequent this communication may become.
    • Uneven load under expert parallelism: when certain tokens hit a few hot experts, some devices are overloaded while others sit idle, and overall latency is dragged down by the slowest device.
    • Batching and utilization: under high concurrency, merging tokens from multiple requests into batches can substantially improve throughput, but sparse activation sends different tokens in a batch to different experts, which increases scheduling complexity.
    • KV cache: during long-context inference, the KV cache used by attention grows linearly with sequence length and can far exceed the cost of expert computation.
    • Quantization and inference framework: weight quantization, operator fusion, and scheduling strategy at the serving layer affect final latency no less than the model structure itself.

    Evaluating the inference cost of an MoE model therefore means looking at total parameters (memory and bandwidth), activated parameters (compute), and routing plus communication (parallelism strategy) together, rather than fixating on a single number.

    4. Engineering Trade-offs for Deployment and Model Selection

    In practice, you can weigh the following dimensions:

    • Capacity versus cost: larger total parameters mean stronger knowledge and a larger expert space, but memory and distributed deployment costs rise steeply. Match the parameter magnitude to how much knowledge depth and domain coverage the business requires.
    • Choosing the active ratio: the ratio of activated to total parameters directly determines per-token compute. A low active ratio suits scenarios pursuing high throughput and low-cost online serving; a higher active ratio is safer in quality-sensitive scenarios.
    • Context strategy: a longer context delivers more capability but puts more pressure on the KV cache and memory. Choose the attention implementation and maximum sequence length according to business needs.
    • Elastic scaling: building inference services on general cloud capabilities such as containerized platforms, message queues, and object storage, together with monitoring and alerting that close the loop on memory, latency, and throughput, is the baseline engineering practice for running MoE services reliably.

    5. Summary

    Reading an MoE model comes down to adopting a two-layer view of scale: total parameters describe knowledge capacity and deployment cost, per-token activated parameters describe the compute budget of a single step, and the context window describes how much sequence the model can process. The three are not interchangeable, and none of them can be converted directly into “capability” or “latency.” At the architecture level, sparse activation through routers and experts is the essence of how MoE saves compute. At the engineering level, load balancing, expert-parallel communication, the KV cache, and inference framework optimizations together determine real serving performance. Once you have this framework in mind, you can make grounded decisions about selection, deployment, and tuning instead of being misled by a single parameter.