Large-Scale Sparse MoE Model Architecture Explained: Total vs. Activated Parameters, Training Data, MLA, and Deployment
This article explains the architecture of a large-scale sparse Mixture-of-Experts (MoE) model with roughly 744B total parameters and 40B activated per token, 256 experts, and 80 layers. It covers the difference between total and activated parameters, how the model's training data grew to 28.5T tokens, and how its key mechanisms work: MoE routing and load balancing, MLA's compressed KV cache, the Muon Split approach to per-head optimization of attention projections, and multi-token prediction. It also explains how to interpret benchmark and arena results. For practitioners, it offers deployment guidance on separating capacity from compute cost, choosing parallelism strategies, balancing expert load, managing long context, and building layered evaluation.
1. Understanding the Model from Its Parameter Counts
Large language model development has been shifting away from simply increasing parameter counts. Instead, the industry is pursuing larger total capacity and lower compute per inference step in parallel. The model described in the source material uses a sparse Mixture-of-Experts (MoE) architecture.
Models of this kind typically contain many expert networks. However, not all experts take part in computing every input token. A routing module selects a few experts based on the semantic features of each token, and only those experts run the forward pass. As a result, it helps to separate two concepts that are easy to confuse:
- Total parameters: the combined size of all experts, shared layers, the embedding layer, and the output layer. This represents the model’s theoretical capacity.
- Activated parameters: the number of parameters actually involved in computing a given token during inference. This mainly reflects the compute cost of a single step.
- Number of experts: the number of subnetworks the routing module can choose from.
- Number of layers: the number of stacked transformation layers, which determines how deep information processing goes.
According to the source material, this generation has about 744B total parameters. It activates about 40B parameters per token and has 256 experts across 80 layers. By comparison, the previous generation had about 355B total parameters and 32B activated parameters. The new model therefore expands its parameter capacity considerably, but compute per step has not grown in proportion to the total parameter count.
This reflects a core design principle of sparse models: expand the expert pool to increase knowledge capacity and task coverage, while keeping the actual compute cost of each inference step under control. Still, fewer activated parameters does not mean total system cost is determined by 40B alone. Deployment must also account for the storage of all parameters, routing communication, GPU memory capacity, expert load balancing, parallelism strategy, and context length.
2. What a Larger Training Dataset Means
The source material indicates that pretraining data grew from about 23T tokens in the previous generation to 28.5T tokens. Of that total, about 27T tokens are used for pretraining and about 1.5T for mid-training.
More data is valuable not only because there is more of it. Equally important are changes in data structure, quality, and the stages of training in which data is used. Large models typically draw on the following types of data:
General text data
Used to learn language patterns, factual expression, connections between pieces of knowledge, and how to organize long-form text.Code and technical data
Helps the model understand code structure, API calls, debugging logic, and algorithm implementations.Mathematical and logical data
Strengthens multi-step reasoning, symbolic processing, and the ability to break down complex problems.Synthetic and filtered data
Quality filtering, deduplication, difficulty stratification, and automated evaluation increase the effective density of the training corpus.Mid-training data
Typically used after core capabilities have formed, to further strengthen long context, code, reasoning, or specific task distributions.
More data therefore does not automatically translate into linear gains in capability. Other factors also have a substantial impact:
- data quality;
- duplication rates;
- domain coverage;
- the mix of data used during training;
- context length;
- curriculum learning strategy;
- how effectively the model absorbs the data.
3. How the MoE Architecture Works
In a traditional dense model, every token passes through the same set of feed-forward networks. MoE models split these feed-forward networks into multiple experts and add a router. Conceptually, the process can be summarized as follows:
Input token↓Shared representation layer↓Router computes expert weights↓Select a few experts↓Experts process independently↓Weighted merge of outputs
Suppose the model has 256 experts, but each token selects only a few of them. In that case, most experts are not activated at all during the current computation step. This structure offers three main benefits:
- Higher model capacity: the model can hold more parameters and support more fine-grained specialization.
- Controlled compute: each token runs only a subset of the expert networks.
- Task specialization: different experts may gradually develop preferences for code, math, natural language, or particular expression patterns.
MoE also introduces new engineering challenges, the most typical being load imbalance. If many tokens are routed to a small number of experts, some devices become overloaded while others sit idle. Communication and compute efficiency then suffer. For this reason, training usually incorporates load-balancing mechanisms, capacity constraints, or routing regularization terms.
In production, the total parameters of an MoE model still need to reside somewhere in memory or GPU memory. Compared with a dense model, MoE may reduce the matrix computations performed per step, but it does not necessarily reduce the complexity of model loading, distributed communication, or system management.
4. MLA and Low-Cache Attention Design
According to the source material, the model uses MLA (Multi-Head Latent Attention). Its core goal is to compress the key-value cache, also known as the KV Cache.
During autoregressive generation, the model continuously stores the keys and values of previous tokens. When the context is long, the KV Cache can quickly consume a large amount of GPU memory. With many concurrent requests, the cache often becomes the main bottleneck for the inference system.
Traditional attention mechanisms generally need to store relatively large key and value representations. A low-dimensional latent scheme compresses them first, then reconstructs them or maps them back to the space needed for attention computation when required. Its main benefits include:
- Lower KV Cache usage per request;
- Better GPU memory utilization in long-context scenarios;
- Higher concurrency on the same hardware;
- Fewer request interruptions or reductions in batch size caused by insufficient cache capacity.
Compressing the attention cache, however, is not simply a matter of using fewer dimensions. Several factors need to be balanced:
- compression ratio;
- reconstruction error;
- positional encoding handling;
- training stability;
- inference accuracy.
If compression is too aggressive, long-range dependency information may be lost. If it is not aggressive enough, the memory savings become hard to realize.
5. Muon Split: Optimizer Adjustments for Attention Projections
The source material also notes a finding from training. When a certain class of matrix optimization methods was applied directly alongside MLA, the results were worse than with a simpler grouped-query attention (GQA) scheme. To address this, the team introduced the Muon Split approach.
The key here is not simply switching to a different optimizer. It is changing how the optimization operation is applied to the weight matrices. A conventional approach may treat an entire projection matrix as a single unit and orthogonalize it or apply a related transformation. Muon Split instead tends to split the matrix by attention head and apply the corresponding processing to each head separately.
An attention head can be understood as a relatively independent channel for extracting information. Different heads may focus on different types of relationships, such as:
- local word dependencies;
- long-range references;
- structured formats;
- code indentation and syntactic relationships;
- relationships between mathematical symbols.
When all heads’ projection parameters are processed as a whole, structural differences between heads may be obscured. Splitting by head allows optimization to preserve the geometric properties of each subspace in more detail. This can improve training stability and representation quality.
This approach shows that large model architecture optimization is often not a matter of replacing a single module. It requires considering the following factors together:
- How the attention structure organizes representations;
- How the optimizer updates projection matrices;
- Whether heads should share statistical properties;
- Whether parameter transformations would disrupt the existing representation space;
- The balance between training stability and final generalization performance.
6. Why Multi-Token Prediction Matters
Multi-token prediction is another notable change listed in the source material. A conventional autoregressive model typically predicts only the next token at each step. It then adds that token to the context and predicts the next one.
Multi-token prediction instead aims to let the model predict tokens at multiple future positions within a single forward pass. It can serve two purposes:
- Training enhancement: helps the model learn longer-range local planning;
- Inference acceleration: generates several candidate tokens first, then uses a verification mechanism to confirm which ones are acceptable.
The potential advantage is less repeated per-token computation, which may improve generation throughput, especially when producing long text or code. However, the approach also faces error accumulation. If earlier candidate tokens are unreliable, the quality of predictions at later positions may decline quickly.
Multi-token prediction therefore usually needs supporting mechanisms such as candidate verification, confidence checks, or fallback strategies. Real-world gains also depend on hardware parallelism, batch size, output length, and task type. More predicted positions does not guarantee a proportional speedup.
7. How to Interpret Model Evaluation Results
According to the source material, the model has achieved high scores in some composite intelligence benchmarks and open arenas, and leads among open models. Results like these should be understood with the limitations of evaluation metrics in mind.
First, a single score cannot fully represent a model’s capabilities. Composite benchmarks usually cover multiple dimensions, including:
- knowledge Q&A;
- math reasoning;
- code generation;
- instruction following;
- long-context understanding.
However, the weighting of different tasks and their data distributions can affect the final ranking.
Second, arena-style evaluations are closer to real conversational experience. They can reflect answer quality, clarity of expression, and user preferences. At the same time, they can be affected by prompts, task distribution, judging mechanisms, and version changes.
A more sound approach is to build a multi-dimensional evaluation system that covers:
- accuracy on foundational knowledge;
- math and logic reasoning;
- code generation and repair;
- long-context retrieval quality;
- tool-calling reliability;
- hallucination rate and factual consistency;
- time to first token and overall throughput;
- cost per request;
- stability under high concurrency.
For enterprise users, offline validation on real business data is usually more informative than a single public ranking.
8. Deployment and Engineering Best Practices
When deploying a sparse model of this scale, it helps to plan around the following areas.
1. Separate Capacity Cost from Compute Cost First
Total parameters determine the demands of model storage and distributed loading. Activated parameters mainly affect per-step computation. Neither can substitute for the other, so memory, GPU memory, bandwidth, and compute requirements should be assessed separately.
2. Design a Sound Parallelism Strategy
MoE models usually need to combine several forms of parallelism:
- data parallelism;
- tensor parallelism;
- pipeline parallelism;
- expert parallelism.
Expert parallelism distributes different experts across different devices, but it adds cross-device routing communication. Network bandwidth and topology are therefore critical.
3. Monitor Expert Load Balancing
Input distributions in production may differ from the training data, causing some experts to be accessed more frequently. Teams should continuously monitor expert hit rates, token distributions, device utilization, and communication latency. This helps prevent local hotspots from slowing down the overall service.
4. Manage Long Context Effectively
Low-cache attention mechanisms help ease KV Cache pressure, but they do not make context length unlimited. Strategies such as session truncation, summary compression, retrieval augmentation, and cache reuse are still needed to keep actual memory usage under control.
5. Build a Layered Evaluation System
Before launch, test quality, latency, throughput, and stability together. Design separate test sets for different scenarios, such as code, customer support, retrieval Q&A, and document analysis. This avoids relying on a single generic metric in place of business validation.
9. Summary
The technical approach of this new generation of sparse MoE models can be summarized in four directions:
- Expanding total parameter capacity;
- Controlling activated compute;
- Compressing long-context caches;
- Improving training optimization.
Within this framework, each mechanism has a distinct role:
- MoE expands model capacity at a lower per-step compute cost.
- MLA-style mechanisms focus on KV Cache efficiency and long context.
- Per-head optimization strategies aim to improve training stability.
- Multi-token prediction explores ways to reduce repeated computation in autoregressive generation.
However, larger models also bring greater system engineering complexity. What ultimately determines how well a model performs in production is not just its parameter count or public benchmark ranking. Data quality, training stability, hardware interconnect, expert balancing, inference serving architecture, and validation methods tailored to specific business needs all matter as well. By understanding these underlying mechanisms, enterprises and developers can make more reliable technical decisions about model selection, deployment planning, and performance optimization.
