Sparse Mixture-of-Experts LLMs: Architecture and Training Techniques Explained
This article explains the design of a new-generation sparse Mixture-of-Experts large language model. It covers how the gap between total and active parameters affects cost, how attention changes reduce KV cache pressure, how the attention structure is optimized, how multi-token prediction and sparse attention work, and what to consider when deploying such a model.
1. Parameter Scale: Total Parameters Do Not Equal Per-Inference Cost
The most closely watched figure for this model is the gap between its total parameter count and its active parameter count. The model has about 744 billion total parameters, but only about 40 billion are activated during a single inference pass. The model uses a Mixture of Experts (MoE) architecture.
A conventional dense model uses all of its parameters at every layer. Generally, larger parameter counts mean higher training costs, greater memory requirements, and more computation per inference. The MoE architecture instead splits the network into multiple expert modules. A routing mechanism selects a small number of experts for each input token. As a result, the model can hold a very large parameter capacity while using only part of it for each computation.
From an engineering perspective, this design addresses two problems at once:
- Expanding model capacity: More experts can each learn different types of patterns, such as code, mathematics, knowledge question answering, and long-text understanding.
- Controlling compute per token: Each token passes through only a few experts, so total parameter count does not translate linearly into inference cost.
The report discloses that the model contains 256 experts and an 80-layer network structure. Compared with the previous generation, which had about 355 billion total parameters and 32 billion active parameters, the new version increases both total capacity and actual compute per token. The pretraining data volume also rose from about 23 trillion tokens to 28.5 trillion tokens. About 27 trillion tokens were used for pretraining, and about 1.5 trillion tokens were used for mid-training.
MoE does not reduce every cost unconditionally. Although active parameters decrease, all expert weights may still need to be loaded or distributed across different devices. The more accurate advantage of MoE is a lower compute cost per token. It does not eliminate storage, communication, and scheduling overhead.
2. Model Capability: Look at Both Overall Ability and Practical Efficiency
Public test results show that the model scored highly on a comprehensive intelligence benchmark and ranks at the top among open-source models. It also performs competitively on competition-style evaluations for text question answering and coding.
A single score does not tell the full story. Model evaluations usually cover at least the following dimensions:
- Knowledge and reasoning: Factual understanding, logical analysis, and solving complex problems.
- Coding: Code generation, debugging, refactoring, and multi-file tasks.
- Instruction following: Whether the model accurately follows formatting rules, constraints, and multi-step requirements.
- Long-context ability: Locating information in long documents, summarizing content, and reasoning across sections.
- Stability and consistency: Whether results are reliable across repeated samples and whether style or conclusions drift.
- Inference efficiency: Time to first token, generation speed, memory usage, and concurrency.
Claims such as “first among open-source models” or “close to leading closed-source models” should be interpreted together with the test set, evaluation protocol, sampling parameters, and hardware environment. For enterprise users, the key question is not the leaderboard position. It is the model’s accuracy, latency, cost, and maintainability on their own business tasks.
3. MLA and Chunked Optimization: Compressing the KV Cache for Long Contexts
The model’s attention mechanism follows the Multi-Latent Attention (MLA) approach. Its core goal is to compress the Key and Value vectors, which form the KV cache in attention.
During autoregressive generation, the model must store the Keys and Values of tokens it has already processed so that later tokens can continue attending to them. The longer the context, the larger the KV cache. For workloads with many concurrent requests, the KV cache often becomes the memory bottleneck.
Standard multi-head attention stores fairly complete Key and Value representations for each attention head. Multi-latent attention instead maps this information into a more compact latent space. During inference, the model can recover or use attention values from a smaller cache, which reduces memory usage for long-context workloads.
The potential benefits include:
- Supporting longer contexts on the same hardware;
- Accommodating more requests in high-concurrency serving;
- Reducing memory pressure from the KV cache;
- Lowering the memory read and write cost of long-text inference.
This structure also increases the complexity of training and implementation. The compressed representation must keep enough information. If the compression ratio, projection method, or training stability is handled poorly, attention expressiveness can decline.
The report notes that when the research team trained this attention structure with a Muon-style optimizer, the results at one point trailed those of a simpler grouped-query attention approach. The team then proposed a “Muon Split” modification. Instead of orthogonalizing an entire projection matrix at once, it processes each attention head separately.
From an optimization standpoint, this change matters because different attention heads usually serve different functions. Transforming the whole matrix uniformly can hide structural differences between heads. Processing each head separately reduces interference between them and makes optimization better match how the attention module is organized.
4. Multi-Token Prediction: Improving Generation Efficiency and Representation Learning
The core idea of multi-token prediction is that, during training, the model predicts several future tokens at once instead of only the next token.
A traditional autoregressive language model uses the following objective:
Input: today's weatherPredict: good
Multi-token prediction extends this to:
Input: today's weatherPredict together: good, suitable for going outside, ...
This does not mean inference can always generate multiple tokens in a single step. The main value lies in the training objective. The model must anticipate text at more distant positions, which may help it learn stronger local planning, syntactic structure, and semantic continuity.
This approach may bring three benefits:
- Stronger training signal: One context is paired with several future prediction targets, which increases the information density of each training sample.
- Better long-range modeling: The model considers subsequent content, not only the immediately following token.
- Support for more efficient decoding: When consistency checks are satisfied, multi-token prediction can be combined with speculative decoding to reduce the wait for token-by-token generation.
Multi-token prediction also adds training heads, target design, and loss-balancing questions. If the prediction distance is poorly chosen, the model may focus too narrowly on local patterns or introduce noise into distant predictions. The actual effect therefore depends on the number of prediction steps, the loss weights, and how the extra objective works alongside the main language modeling objective.
5. DSA Sparse Attention: Reducing Computation for Long Sequences
Standard self-attention cost typically grows with the square of the sequence length. As the context expands from a few thousand tokens to several hundred thousand, the attention matrix grows quickly, and both compute and memory pressure become significant.
Sparse attention takes a different approach. Instead of having each token attend to all historical tokens, it selects a subset of important positions based on position, content, or routing results. This converts dense all-to-all attention into sparse connections.
A general sparse attention pipeline can be summarized as:
Input sequence↓Generate query representations↓Filter candidate historical positions↓Run fine-grained attention only on candidate positions↓Fuse local and global information
The main difficulty of these methods is not reducing computation itself. The harder problem is avoiding the loss of important information. If the filtering step wrongly removes key context, the model may miss facts, fail at cross-section reasoning, or lose track of references.
Sparse attention therefore requires balancing several factors:
- Recall of candidate positions;
- Sparsity ratio and actual speedup;
- Ability to preserve important information;
- Consistency between training and inference;
- Communication overhead in distributed environments.
Sparse attention is valuable for long-document question answering, code repository analysis, log retrieval, and multi-turn dialogue. In deployment, theoretical complexity alone does not show the real gain. The team also needs to check whether the hardware kernels support sparse computation, whether memory access stays contiguous, and whether the serving framework includes the matching optimizations.
6. The Upgrade from the Previous Generation: Structural Coordination Matters More Than Scale
The upgrade does not come from simply adding parameters. Total parameters, active parameters, and training data all grew together. This means the model expanded its capacity, compute budget, and knowledge coverage in a coordinated way.
Each change targets a different bottleneck:
- MoE architecture addresses the tension between model capacity and compute per token;
- Multi-latent attention eases KV cache pressure in long contexts;
- Per-head optimization improves training stability for complex attention structures;
- Multi-token prediction strengthens the training signal and explores more efficient generation;
- Sparse attention reduces wasted computation in very long sequences.
These designs are not independent of each other. The more complex the architecture, the higher the demands on the optimizer, parallelization strategy, communication mechanism, and training data quality. The gain from any single module can be offset by bottlenecks elsewhere. For example, fewer active parameters do not guarantee cheaper inference. If experts are unevenly loaded, some devices may be overloaded, communication may stall, and serving throughput may drop.
7. Key Considerations for Engineering Deployment
Teams planning to deploy this kind of model should focus on the following areas.
1. Expert Load Balancing
The router should distribute tokens evenly across experts. If many tokens concentrate on a few experts, those experts become overloaded. This can affect training stability and inference latency.
2. Distributed Communication
Experts are usually spread across multiple compute devices, so token routing introduces cross-device communication. Actual throughput depends on network bandwidth, communication topology, and scheduling efficiency, not only on compute capacity.
3. Memory Planning
Planning must account for model weights, activations, the KV cache, communication buffers, and batch space. For long-context serving, the KV cache may become the limiting factor earlier than the model weights.
4. Evaluation Set Adaptation
General benchmark scores cannot replace evaluation on your own tasks. Enterprises should build test sets that include real questions, long documents, domain terminology, code samples, and unusual inputs. They should measure accuracy, stability, latency, and cost separately.
5. Serving Strategy Design
Serving policies can vary by request length and task type. For example, short requests can be prioritized for low latency, long-document tasks can use sparse attention and cache reuse, and high-concurrency workloads need careful control of batch size and expert load.
8. Summary
The technical value of this new-generation model lies not only in its 744 billion total parameters. It also lies in a system design built around capacity, efficiency, and long-context ability. MoE expands model capacity, while active parameters control compute per token. Multi-latent attention reduces KV cache pressure. Per-head optimization improves training for complex attention structures. Multi-token prediction provides a richer learning signal. Sparse attention offers a new compute path for very long sequences.
For developers, the key is not memorizing parameter counts. The key is understanding which problem each technique solves, what new costs it introduces, and how to verify its benefits under real hardware and business constraints. Model architecture, training methods, inference frameworks, and evaluation systems must be considered together before deciding whether a large model suits a production environment.
