Open-Source LLM MoE Architecture Explained: How Sparse Activation Decouples Capacity from Compute

    This article breaks down the sparse Mixture of Experts (MoE) architecture that powers today's large open-source language models. It explains how experts and gating routers work, why total parameters and activated parameters must be evaluated separately, how parameter scaling and data scaling reinforce each other, which evaluation dimensions actually matter, and what to watch for when deploying a 700B-class MoE model in production—ending with practical guidance for developers and enterprises choosing and running these models.

    Introduction: Why Large Models Are Going Sparse

    For the past two years, capability gains in large models have largely followed one simple path: make the parameter count bigger and feed in more data. Under a dense architecture, however, parameter growth is almost linearly coupled with the compute spent at inference time—every step up in capability brings a matching step up in serving cost. For developers and enterprises that want to put large models into real production, that overhead is hard to sustain.

    The sparse Mixture of Experts (MoE) architecture has become the mainstream answer to this problem. Its core idea is to give a model an enormous reserve of parameters while activating only a small subset of them on each inference pass, decoupling “capacity” from “compute cost.” Take an open-source MoE model with roughly 744B total parameters, about 40B activated per inference, 256 experts, and 80 layers: this design makes the engineering value of the approach easy to see. Using this class of ultra-large sparse models as a thread, this article unpacks the architecture, its scaling behavior, the dimensions worth evaluating, and the key deployment considerations.

    1. MoE Fundamentals: Experts and Gating

    An MoE layer can be understood as replacing a conventional single feed-forward network (FNN) layer with a group of parallel “expert” subnetworks, plus a “gating network” (router) that decides which ones run.

    • Experts: Each expert is typically an independent feed-forward subnetwork, specialized in—or “responsible for”—features from different distributions, modalities, or semantic spaces.
    • Router (gating network): For every input token, the router scores the match against each expert, picks the top few highest-scoring experts (usually Top-1 or Top-2), and sends the token only to the selected experts.

    As a result, even though the model’s total parameter count is huge, a single token passes through only a handful of experts per layer while the rest stay “dormant.” That is the essence of sparse activation.

    The routing logic can be expressed with the following illustrative pseudocode:

    1. def moe_layer(x, experts, router, top_k=2):
    2. # The router scores every expert for each token
    3. scores = router(x) # shape: [tokens, num_experts]
    4. topk_val, topk_idx = scores.topk(top_k, dim=-1)
    5. # Normalize the weights of the selected experts
    6. gates = softmax(topk_val, dim=-1)
    7. out = zeros_like(x)
    8. for k in range(top_k):
    9. idx = topk_idx[:, k] # index of the k-th selected expert
    10. expert_out = experts[idx](x) # compute only for tokens routed to this expert
    11. out += gates[:, k:k+1] * expert_out
    12. return out

    The logic makes one thing clear: compute cost is driven by top_k and the size of each expert, and is largely independent of the total number of experts. More experts means more “knowledge capacity,” while per-token compute stays broadly under control.

    2. Key Distinction: Total Parameters vs. Activated Parameters

    The single most important thing to understand about MoE models is that you can no longer judge them by one number. Two metrics have to be separated:

    1. Total parameters: the combined parameter count across all experts. This sets the ceiling on the model’s knowledge capacity.
    2. Activated parameters: the parameters that actually participate in computation for a single token. This determines the compute cost of one inference pass.

    For the example model, roughly 744B total parameters against about 40B activated puts the activation ratio below 6%. That leads to one critical engineering conclusion: GPU memory / RAM footprint is determined by the total parameter count, while inference compute—especially in compute-bound phases—is determined by the activated parameter count.

    Practically, deploying a model like this means:

    • You need enough GPU memory or RAM to hold all the experts, even the ones this particular request never touches; otherwise loading simply fails.
    • Actual throughput and single-request latency, however, are closer to those of a dense model in the tens-of-billions range than to those of a 700B+ dense model.

    This is exactly where the cost-performance advantage of sparse MoE comes from: much larger knowledge capacity in exchange for inference overhead close to a much smaller model. Compared with the previous generation’s common dense or smaller-scale MoE configurations (for example, 355B total with 32B activated), roughly doubling total parameters while raising activated parameters moderately—combined with more substantial training data—is the typical path behind this generation of open-source models taking a capability leap.

    3. How Parameter Scaling and Data Scaling Reinforce Each Other

    Simply stacking parameters does not guarantee better capability. The scaling law widely accepted in the industry holds that model capability is jointly determined by parameter count, training data volume, and compute, and that a sensible ratio must be maintained among the three.

    In the example model, pretraining data grew from roughly 23T tokens in the previous generation to about 28.5T tokens (around 27T for pretraining and around 1.5T for mid-training). Two practices behind this are worth noting:

    • Pretraining: massive, high-quality, deduplicated general-purpose corpora that build the model’s foundational knowledge and language ability. Data quality often determines the model’s ceiling more than raw quantity does.
    • Mid-training: after general pretraining, targeted data supplementation and annealing aimed at specific capabilities—such as code, reasoning, and long context—to strengthen the target abilities without eroding generality.

    Scaling parameters and data in tandem is the underlying reason this generation of open-source models can approach leading closed-source models across a range of composite benchmarks. It also suggests something about how to judge a model: not just “how big is it,” but whether its scaling ratios are well balanced and whether training was carried through thoroughly.

    4. How to Evaluate a Large Model’s Capability Properly

    Parameters and architecture are only paper specifications. What actually determines practical value is capability evaluation. For a new model, cross-validate across several dimensions rather than relying on a single leaderboard:

    1. Composite capability index: the industry has various “composite intelligence index” leaderboards that weight and aggregate benchmarks spanning language understanding, knowledge Q&A, mathematical reasoning, code, and more. They are useful for coarse-grained horizontal comparison, but check whether the evaluation sets have been contaminated by leaderboard gaming.
    2. Human preference arena evaluation: real users compare anonymized model outputs head-to-head and vote, producing a ranking based on human subjective preference. This kind of evaluation is closer to real experience and is especially useful for judging conversational text quality and code generation quality.
    3. Domain-specific evaluations: evaluating code generation, mathematical reasoning, long-text understanding, and instruction following separately is what tells you whether a model fits your business.

    Broadly, public results show leading open-source models now score close to closed-source benchmarks on composite capability indices, and rank at the front of the open-source field in human-preference evaluations such as text arenas and code arenas—placing them in the same tier as top closed-source models overall. That said, a gap with closed-source models remains on some of the hardest reasoning tasks, so selection should be based on hands-on testing in your specific scenario rather than rankings alone.

    5. Key Engineering Considerations for Deployment

    Putting a sparse MoE model like this into production raises a few practical questions you cannot avoid:

    • GPU memory / RAM capacity: the total parameter count dictates that the full set of weights must be loaded. A model in the 700B class typically requires coordinated GPU memory across multiple cards and multiple nodes, so budget hardware carefully before deployment.
    • Inference throughput and latency: fewer activated parameters keep per-request compute manageable, but uneven expert routing in MoE can cause load skew—hot experts get saturated while cold ones sit idle. In practice, strategies such as load-balancing loss and expert capacity limits are introduced to mitigate this.
    • Distributed parallelism: models at this scale usually combine tensor parallelism, expert parallelism, pipeline parallelism, and other strategies to shard across many devices. Communication overhead and sharding strategy both need careful tuning.
    • Inference frameworks and quantization: combining weight quantization (such as low-bit quantization) to reduce memory pressure with efficient batching and KV cache management is key to raising throughput and controlling cost.

    For teams that want to validate quickly rather than build their own cluster, cloud capabilities can carry this whole workflow: use a container platform to spin up inference services elastically, object storage to host model weights and datasets, and logging plus monitoring and alerting to track throughput, latency, and memory utilization. On a cloud platform such as Baidu AI Cloud, for example, these general-purpose building blocks can be combined on demand, letting teams focus on model selection and effectiveness validation instead of building infrastructure from scratch.

    6. Practical Advice for Developers and Enterprises

    Overall, here is what to keep in mind with this new generation of sparse MoE models:

    1. Size against activated parameters, not total parameters: estimate inference cost from activated parameters first, then use total parameters to gauge the hardware threshold—avoiding both “it looked big, so I assumed I couldn’t afford it” and “it looked small, so I assumed it wouldn’t fit” misjudgments.
    2. Let scenario testing be the deciding factor: use leaderboards for initial screening, but always finish with A/B comparisons on evaluation sets close to your own business, with particular attention to hard-capability scenarios like code and reasoning.
    3. Take load balancing on the deployment side seriously: MoE’s routing characteristics mean its production behavior depends heavily on tuning the inference framework and parallelism strategy.
    4. Make good use of elastic infrastructure: prioritize cloud containers, storage, and monitoring for fast validation and elastic scaling, and leave cost control and operational complexity to mature platform tooling.

    Conclusion

    By decoupling total parameters from activated parameters, the sparse MoE architecture lets large models retain enormous knowledge capacity while keeping per-inference cost within an acceptable range—and that is the key reason the newest open-source models can close in quickly on top closed-source benchmarks. For developers, truly mastering this paradigm means understanding not only the principles of experts, gating, and sparse activation, but also making sound trade-offs across the entire chain from selection to evaluation to deployment. Only then does powerful model capability turn into stable, economical, deployable productivity.