Open-Source Sparse MoE Models: Architecture, Training, and Deployment

    Open-source large models are rapidly closing the gap with their closed-source counterparts, and sparse architectures are a key part of that story. This article uses the GLM-5 technical report as a lens to explain how sparse Mixture-of-Experts (MoE) models convert a modest increase in compute into much larger knowledge capacity. It walks through the main architectural changes—MLA compressed KV caching, Multi-Token Prediction, and DSA native sparse attention—alongside training techniques such as the Muon optimizer with Muon Split and fully asynchronous Agent RL. It also covers how to judge data sufficiency for sparse models and, critically, how to estimate memory and avoid common pitfalls when deploying them, including tensor/expert parallelism, quantization choices, and the stability signals worth monitoring. Readers will come away with a practical framework for selecting, costing, deploying, and monitoring large-scale sparse MoE models in production.

    Why Open Source Is Closing In on Closed Source

    The capability gap between open-source and closed-source large models is shrinking quickly, and sparse architectures are a central variable behind that trend. a16z has noted in its AI weekly reporting and frontier analysis that top closed-source models still hold a clear advantage, yet the gap to open-source models has narrowed noticeably. Verifiable references include the a16z site and its AI section (https://a16z.com/ , where you can search for a16z AI Weekly, AI frontier, and technical-report material), as well as Zhipu AI’s GLM-5 Technical Report released on February 22, 2025 (searchable under the THUDM GitHub organization at https://github.com/THUDM ; the Zhipu AI official release is also recommended as the primary source).

    The discussion this report sparked centered on three directions. First, how ultra-large sparse expert models achieve stronger knowledge capacity with fewer activated parameters. Second, how architectural changes—MLA, Multi-Token Prediction, and DSA sparse attention—work together. Third, how asynchronous Agent reinforcement learning and a self-built RL infrastructure improve training efficiency. Rather than simply restating conclusions, this article follows a main thread of “architecture—data—training—inference—engineering” to unpack where the capabilities of open-source sparse models come from and what matters for enterprise adoption.

    From public information, the key figures for this generation of GLM-5 can be summarized as follows:

    Dimension Previous gen GLM-4.5 This gen GLM-5 Change
    Total parameters 355B 744B ~2.1x
    Activated parameters 32B 40B ~1.25x
    Experts / layers 355 / 92 256 / 80 Fewer experts, relatively fewer layers
    Pretraining data 23T tokens 28.5T tokens ~1.24x

    One point deserves special emphasis when reading this table: you must distinguish “stored parameters” from “activated parameters.” Total parameters represent the weight scale kept on disk and set the lower bound on deployment cost; activated parameters represent the scale of parameters that actually participate in computation per token, and they dominate the compute and latency of inference. GLM-5 doubles its total parameters while adding only about a quarter more activated parameters. This is precisely the core value of sparse MoE: trading an acceptable increase in compute for a much larger knowledge capacity.

    How Sparsity Reshapes Compute and Memory

    The essence of MoE (Mixture of Experts) is conditional computation. A dense model invokes all of its parameters when processing each token, whereas an MoE model uses a gating network to select a small number of experts for each token. For GLM-5, with 744B total parameters and 40B activated parameters, the activation ratio is roughly 5.4%. This means the compute per token is closer to that of a 40B dense model, while the model’s overall knowledge capacity can approach that of a 744B parameter scale.

    This leads to two direct consequences:

    1. Compute cost stays relatively controllable. Inference compute scales more with activated parameters than with total parameters.
    2. Memory cost cannot be judged by activated parameters alone. All weights must reside in memory or be swapped in and out efficiently, so memory pressure is far greater than for a dense model with the same activation level.

    Common sparse structures include MoE, sparse attention, and structural pruning. They can be combined, but any combination must be validated end to end.

    Structure Sparsity target Main benefit Main cost
    MoE Parameters High compute-to-knowledge ratio High memory use, load imbalance, routing jitter hurting stability
    Sparse attention Attention computation Saves time and memory on long sequences Complex to implement, must align with the training objective
    Structural pruning Parameters Regular structure, good for inference acceleration Accuracy-loss risk, usually needs retraining to recover

    Reducing Experts From 355 to 256: What Does “Fewer and Deeper” Mean?

    The previous configuration can be read as “355 experts / 92 layers,” while this generation uses “256 experts / 80 layers.” For estimation purposes, this article uses a simple rule of thumb: divide total experts by the number of layers to observe the average expert scale each MoE layer can spread across. Under this rule, the previous generation is about 355/92 ≈ 3.9, and this generation is 256/80 = 3.2.

    A caveat is worth adding: if the 355/256 figures in the report refer to the number of experts selectable per layer rather than the total number of experts, the phrasing should instead be “experts routed per layer.” Here, we adopt the “total experts divided by layers” convention to explain combinatorial complexity; it does not represent every detail of the model’s underlying implementation.

    Fewer experts with a relative change in layer count may bring two kinds of benefits. First, less routing ambiguity means a more stable gate. Second, a smaller set of experts participating per layer makes communication more concentrated and easier to optimize with fused kernels. On the other hand, fewer experts must each carry more capacity, raising the bar for load balancing and expert utilization. Common practices include load-balancing regularizers, a capacity factor, expert replication, and dropless training. The report does not fully disclose whether shared experts are used, but from an engineering standpoint, shared experts help absorb general-purpose capabilities and avoid every expert relearning the same general knowledge—a design worth attention when scaling total parameters.

    Architecture Change 1: How MLA and Muon Split Work Together

    The report highlights three key architectural changes in this generation: MLA + Muon Split, Multi-Token Prediction, and DSA sparse attention.

    MLA: Compressing the KV Cache Into a Low-Dimensional Latent Space

    MLA (Multi-head Latent Attention) shares its origin with DeepSeek-V3. Its core idea is to compress the dimensionality of the KV cache, thereby reducing the cache bandwidth pressure during long-text inference. The main line of the attention mechanism—from MHA to MQA/GQA, and then to MLA—has always revolved around the KV cache:

    • MHA: each head has its own K and V, the largest cache, stable quality.
    • MQA/GQA: multiple heads share K and V, a significant cache reduction, but some capacity is lost.
    • MLA: hidden states are first compressed into a low-dimensional latent vector, then up-projected to restore K and V when needed; the cache mainly stores the compressed latent vector.

    A simplified view:

    1. Standard attention:
    2. K = W_k · h
    3. V = W_v · h
    4. out = softmax(Q·K^T / sqrt(d)) · V
    5. MLA:
    6. c = W_dkv · h # low-dimensional latent vector; the cache mainly stores c
    7. K = W_uk · c
    8. V = W_uv · c

    MLA’s engineering value goes beyond saving memory. Because K and V are both linear functions of c, the up-projection matrices can be absorbed into the query side during inference, so the KV cache only keeps the low-dimensional c. This achieves decoding efficiency close to multi-query attention at a modest memory cost, which is especially well suited to high-concurrency, long-context serving.

    Muon: A Momentum Optimizer With Matrix Orthogonalization

    Muon is an optimizer for 2D weight matrices. Beyond the usual momentum update, it approximately orthogonalizes the momentum matrix so that parameter updates are more balanced across directions, avoiding a few singular values dominating training. The following pseudocode is only for illustrating the principle, not a runnable production implementation:

    1. def muon_step(W, g, state, lr=1e-3, momentum=0.95):
    2. # W: weight matrix, shape [out_features, in_features]
    3. # g: gradient matrix, same shape as W
    4. # state: optimizer state, must be initialized beforehand
    5. if "buf" not in state:
    6. state["buf"] = torch.zeros_like(W)
    7. buf = state["buf"]
    8. buf.mul_(momentum).add_(g, alpha=1 - momentum)
    9. # newton_schulz takes a 2D matrix as input and outputs an approximately orthogonalized matrix
    10. O = newton_schulz(buf)
    11. W -= lr * O
    12. return state

    Here, newton_schulz approximates the orthogonalization result through a few matrix multiplications, typically targeting the orthogonal factor in a polar decomposition. It is not simple gradient clipping; it changes the balance of parameter updates across different scales.

    Why Muon Clashes With MLA

    The training team found that combining Muon with MLA performed worse than the simpler GQA-8 scheme. The reason is that the orthogonalization approximation is highly sensitive to matrix shape. Common open-source implementations use empirical scaling factors to compensate for a matrix’s aspect ratio—for example, flattening the three sides of certain long, thin matrices. But matrices in MLA can be even more elongated, so empirical rules do not necessarily hold.

    This is not simply “a bad optimizer” but a coupling conflict between “the optimizer and the structure.”

    Muon Split: Orthogonalizing per Attention Head

    The team’s solution is Muon Split: instead of orthogonalizing the entire projection matrix as one block, orthogonalize each attention head separately.

    1. # Before: orthogonalize the whole projection matrix as one block
    2. # W: [n_head * d_head, d_model]
    3. O = orthogonalize(momentum_buf)
    4. # After: split by attention head and orthogonalize each head independently
    5. for head in range(n_head):
    6. W_h = W[head * d_head : (head + 1) * d_head, :]
    7. buf_h = momentum_buf[head * d_head : (head + 1) * d_head, :]
    8. O_h = orthogonalize(buf_h)
    9. W_h -= lr * O_h

    After splitting by head, each block is closer to a regular shape, and the orthogonalization approximation is no longer disturbed by extreme aspect ratios. This is also isomorphic to MLA’s design logic: MLA itself gives each head an independent latent space, so the optimizer should adjust update scales independently within each head’s subspace. The engineering lesson is that when an optimizer conflicts with a structure, align the optimizer’s granularity to the structure’s natural units first, rather than rushing to reject the overall architecture.

    Architecture Change 2: How Multi-Token Prediction Improves Efficiency

    The second key change is Multi-Token Prediction (MTP). A standard language model predicts only the next token at each position, whereas MTP lets the model predict several future tokens at each position simultaneously.

    A simplified structure:

    1. # Simplified MTP training objective, n=3 for illustration only
    2. h = backbone(x)
    3. loss = ce(main_head(h), y[t + 1])
    4. for k in range(1, n + 1):
    5. h_k = mtp_head[k](h, embed(y[t + k]))
    6. loss += lambda_k * ce(h_k, y[t + 1 + k])

    MTP’s value lies mainly in two layers:

    1. Denser training signal. The same piece of text provides longer-range supervision, forcing hidden states to encode planning information further ahead.
    2. Self-speculative decoding at inference. The MTP head acts like a draft model built into the model itself, proposing several candidate tokens in one forward pass, which the backbone then verifies in parallel.

    Note that MTP’s inference benefit depends on the acceptance rate. It pays off more on highly predictable tasks such as code generation; in open-ended creative writing, the acceptance rate may drop. A more robust strategy is therefore to “use it heavily during training and enable it selectively at inference.”

    Architecture Change 3: How DSA Sparse Attention Cuts Costs

    The third focus is DSA (DeepSeek Sparse Attention). In standard dense attention, computing token i requires attending to all preceding tokens; compute complexity grows quadratically with sequence length, and the KV cache grows linearly. In long-context scenarios, this overhead can exceed the expert forward computation itself.

    There are two families of solutions. One is to let each token look at less, such as local windows or block sparsity. The other is to let each token look more precisely—use a lightweight indexer to pick the few most relevant tokens first, then apply full attention to them. DSA belongs to the latter, and its core component is the indexer:

    1. # Indexer illustration
    2. score[i, j] = indexer(q_i, k_j)
    3. topk_idx = causal_topk(score, k) # k is far smaller than the sequence length
    4. out[i] = full_attention(q_i, k[topk_idx[i]], v[topk_idx[i]])

    The keyword for DSA is “natively trainable.” Sparse selection exists from early pretraining rather than being retrofitted onto a dense model after the fact; the selection process also participates in gradient flow, so the model can learn to actively seek evidence. Mainstream implementations typically use fine-grained selection at the token or small-block level, closer to retrieval on demand.

    The frequently cited “20B tokens matching 943.7B tokens” claim needs careful interpretation. It does not simply claim that a 20B-token sparse model is fully equivalent to a complete 943.7B-token model. Rather, under the report’s assumptions or illustrative framing, it discusses sample-efficiency differences under boundary conditions such as equal parameters, equal compute, and a fixed data mix.

    Training volume Sparse attention model Dense attention model
    20B tokens Converges earlier Not yet caught up
    200B tokens Level or ahead Continuing to close
    943.7B tokens Holds performance Catches up

    A more reasonable conclusion is that sparse models may hit their ceiling earlier with fewer tokens, while dense models offer higher throughput but may need more tokens to catch up. This advantage depends on the compute budget and comparison conditions; comparing directly without equal compute, equal parameters, and the same data mix easily leads to misleading conclusions.

    How to Judge Data Sufficiency

    Architecture determines how a model learns; data determines what it learns. This generation’s pretraining data grows from 23T tokens to 28.5T tokens, which can be broken down into roughly 27T pretraining tokens and 1.5T mid-training tokens.

    Estimated against total parameters, 28.5T / 744B is about 38 tokens per parameter—higher than the ~20 tokens-per-parameter reference line traditionally cited for Chinchilla. So it is not accurate to simply say it is “still below Chinchilla.” A more precise statement is that, measured against total parameters, this generation’s data scale is not low. For sparse MoE, however, data sufficiency must be assessed using both total and activated parameters together. Total parameters are closer to knowledge capacity, and activated parameters are closer to compute per step; the two should not be conflated.

    Mid-training sits between pretraining and post-training. Common goals include strengthening high-value capabilities such as code, math, and logical reasoning, extending the context window, and introducing agent trajectories and tool-call samples to prepare for post-training.

    How Asynchronous Agent RL Improves Training Utilization

    In the post-training stage, the team adopts fully asynchronous Agent RL: interaction between agents and the environment is decoupled from parameter updates. Synchronous RL often faces two bottlenecks: generation and training wait on each other, and long-tail tasks hold back the whole batch. The core mechanisms of asynchronous RL include:

    • The generation pool runs continuously, and the inference cluster keeps writing experience.
    • The training cluster continuously samples from the buffer and updates.
    • Weights are periodically synced to the inference cluster.
    • Training is stabilized through staleness control, importance-sampling correction, and a maximum-staleness limit.

    Illustrative code:

    1. rollout_pool = AsyncRolloutPool(env_tools, max_staleness=K)
    2. replay = ReplayBuffer()
    3. while not converged:
    4. batch = replay.sample(batch_size, staleness_weighted=True)
    5. loss = policy_loss(batch) + kl_penalty(batch)
    6. loss.backward()
    7. optimizer.step()
    8. if step % sync_interval == 0:
    9. rollout_pool.update_weights(model.state_dict())

    The differences between synchronous and asynchronous operation:

    Dimension Synchronous Agent RL Fully asynchronous Agent RL
    GPU utilization Generation and training idle in turns Both ends closer to full load
    Sample freshness High Staleness present, needs correction
    Long-tail tolerance Whole batch waits for the slowest Each task enters the pool when done
    Engineering complexity Low High
    Applicable stage Short tasks, small scale Large scale, long-horizon agent training

    Note that asynchronous RL does not mean never pausing. A more accurate statement is that it keeps generating within each weight-sync cycle, significantly reducing idle time, while still being affected by sync frequency, staleness control, and resource queuing.

    Why the In-House slime Infrastructure Matters

    Agent RL is demanding on infrastructure: training, inference, and environment execution are heterogeneous workloads; long-tail interactions are frequent; policy drift is fast; and reward construction is complex. Once a general-purpose framework is adapted up to a point, it often requires extensive modification, and building in-house can actually be more economical.

    slime’s approach is to add a thin abstraction layer: rather than rebuilding the inference engine, it treats a mature inference engine as a hot-swappable generation service and wraps data flow, reward computation, and weight synchronization around it. This reuses the inference engine’s accumulated strengths in throughput, scheduling, and kernels, while concentrating in-house effort on differentiated parts such as asynchronous scheduling, weight synchronization, and agent environment management. Supporting parameters like partial rollouts, dynamic batch size, and the rollout buffer map respectively to the three problems of long tails, padding waste, and waiting. The value of such tuning is that, in most cases, you can improve training efficiency without modifying core code.

    Estimating and Avoiding Pitfalls in Inference Deployment

    The most common mistake in deploying sparse MoE is estimating cost with dense-model intuition. A MoE model with 40B activated parameters may need compute close to a 40B dense model, but its memory demand may approach the weight scale of 744B parameters.

    A recommended estimation framework:

    1. Total memory ≈ weight memory + KV cache + activations and temporary tensors + framework communication overhead
    2. Weight memory ≈ total params × bytes per param
    3. For example: 744B × 2 bytes (BF16) ≈ 1488 GB
    4. Using FP8/INT8 can significantly reduce weight memory, but accuracy regression must be evaluated
    5. KV cache ≈ num_layers × 2 × num_kv_heads × head_dim × seq_len × batch × bytes

    Deployment best practices include:

    1. Estimate memory by total parameters, then add KV cache and concurrency overhead.
    2. For long-context scenarios, prioritize combining MLA with native sparse attention.
    3. When combining tensor parallelism with expert parallelism, watch for load skew caused by hot experts.
    4. Choose quantization by scenario: keep higher precision on the training side, and reduce precision on the inference side only where accuracy stays controllable.
    5. MTP heads can be combined with self-speculative decoding to reduce the cost of a separate draft model.
    6. Scheduling should distinguish long-context from short requests so they do not drag each other down.

    Beyond basic metrics, you should also monitor hot-expert load, KV quantization accuracy regression, cold-expert offload latency, cache hit rate, first-token latency, and memory watermark.

    Practical Checklist and Summary

    Stage Key action Common pitfall
    Selection & evaluation Evaluate on your own business data, not just public leaderboards Public leaderboards differ greatly from business distribution
    Cost estimation Estimate memory by total params, KV cache, concurrency, and quantization Estimating only by activated parameters
    Deployment Combine tensor parallelism, expert parallelism, and quantization Ignoring load skew from hot experts
    Long-context optimization Prefer models with native sparse attention and MLA Using post-hoc truncation instead of native sparsity
    Stability monitoring Track first-token latency, throughput, and memory watermark Lacking capacity warnings and expert load monitoring

    If your team is evaluating a sparse MoE model of similar scale, you can start by answering three questions: How do you choose a sparse MoE? How do you estimate memory? What problems do MLA and DSA each solve? Taking it further into the business, you need to clarify the input-length distribution, concurrency scale, latency requirements, memory budget, and quantization precision tolerance.

    Sparsity is not a silver bullet; it is a systems engineering effort that coordinates algorithms, systems, and engineering. It gives open-source models stronger knowledge capacity, and it also demands more precise engineering judgment from enterprises in selection, evaluation, deployment, and monitoring.