Ultra-Large Sparse MoE Models Explained: Architecture, Training, Evaluation, and Deployment
This article breaks down the new generation of ultra-large sparse mixture-of-experts (MoE) models and explains what actually matters for developers and technical teams beyond raw parameter count. It covers the model's core specs (744B total parameters, ~40B activated per inference, 256 experts, 80 layers, 28.5T training tokens), how MoE improves cost-performance, and the key architectural and training choices behind it: Multi-head Latent Attention (MLA) and per-head orthogonalization (Muon Split), multi-token prediction, sparse attention, and the data strategy behind 28.5T tokens. It also includes a MoE-vs-dense selection checklist, guidance on the gap between benchmark scores and real-world results, production engineering concerns such as expert parallelism and KV cache management, typical use cases in code intelligence and enterprise knowledge processing, and the risk and compliance boundaries enterprises should check before adoption.
Start with the conclusion: what these models are good for
For developers and technical teams, the most important thing about the new generation of ultra-large sparse expert models is not the parameter count itself, but how the model balances total capacity, activated parameters, long-context efficiency, training-data quality, and engineering serviceability. This article answers three practical questions—how MoE models reduce inference cost, how long context is optimized, and how to choose open-weight models—by walking through four layers: architecture, training, evaluation, and deployment.
These ultra-large sparse expert models fit scenarios that require private deployment, controllable cost, data isolation, fine-tuning, and complex reasoning. They are not a one-size-fits-all replacement for every model. Instead, they work best as a high-performance base, combined with retrieval-augmented generation (RAG), tool calling, code execution environments, test frameworks, and business workflows.
If your team mainly works on general dialogue, code generation, long-document understanding, enterprise knowledge Q&A, or complex task planning, these models offer high practical value. But if you only need low latency, low cost, simple classification, or small-scale Q&A, a smaller dense model may be a better fit.
Model fundamentals: 744B total, 40B activated, 28.5T tokens
These models use a typical sparse mixture-of-experts (MoE) architecture. The core parameters are: 744B total parameters, roughly 40B activated parameters per single inference, 256 experts, and 80 network layers. Compared with the previous generation, total parameters grew from 355B to 744B, while activated parameters rose from 32B to 40B. This shows that the increase in capability does not depend entirely on expanding activated parameters, but is achieved by enlarging total capacity and strengthening expert expressiveness.
On the training-data side, total training volume increased to 28.5T tokens, with about 27T tokens for pre-training and about 1.5T tokens for mid-training. For a model of this scale, data volume is not the only thing that matters—data quality, mixing ratios, deduplication, filtering, and staged training are equally important. Pre-training mainly provides broad language ability, world knowledge, and a code foundation; mid-training is used more to improve instruction following, complex reasoning, code engineering, long-text understanding, and performance in specific domains.
In public evaluations or specific leaderboard contexts, this type of model reaches roughly the 50-point range on certain composite intelligence index benchmarks, putting it at a leading position among open-weight models. However, leaderboards, prompts, temperature settings, and decoding strategies all affect scores. Developers should not treat public scores as equivalent to private-business performance, and should instead run small-scale validation against their own scenarios.
How MoE architecture improves cost-performance
The essence of the MoE architecture is to split one large model into multiple expert sub-networks, then use a router to select a small number of experts to participate in computation for each token. This lets total model capacity be very large while limiting the number of parameters actually used per token. For inference services, this design eases memory and compute pressure; for training, it allows the model to keep scaling at larger parameter counts while controlling the complexity of a single forward pass.
But MoE is not an inherently efficient architecture. It faces several typical challenges. First, routing load balancing is hard—some experts may be over-selected while others are under-trained. Second, expert parallelism introduces additional communication overhead. Third, the inference stage must manage a large number of expert weights, which places higher demands on memory bandwidth and caching strategy. Fourth, in long-context scenarios, attention computation and the KV cache become new bottlenecks.
So when selecting an MoE model, you cannot look only at total and activated parameters. You also need to consider expert-parallelism efficiency, routing stability, memory budget, concurrent throughput, and fine-tuning cost.
MLA and Muon Split: long-context compression and per-head orthogonalization
The new generation of models adopts MLA (Multi-head Latent Attention) in its attention mechanism. The core idea is not to store the full Key and Value directly, but to compress KV information into a lower-dimensional latent space, reducing KV cache usage during inference. This optimization is valuable for long-text scenarios, because in traditional attention the KV cache grows linearly with sequence length, and long-context serving is often limited by memory bandwidth and cache capacity.
During training, the team found that using a certain class of matrix-orthogonalization optimizer together with MLA did not always outperform a simpler grouped-query attention scheme. More precisely, this was an observation under a specific optimizer, learning-rate, and initialization configuration, and it may be affected by the optimizer’s update method, the structure of the projection matrices, and the coupling between attention heads. This conclusion should be validated against the official technical report or reproduction experiments and cannot be generalized to all training configurations.
Understood as Muon Split, the process can be seen as an engineering strategy that orthogonalizes projection matrices after splitting them by attention head. It is not a simple “whole-matrix orthogonalization,” but rather an attempt to preserve finer-grained structural constraints along each attention-head dimension.
Illustrative code:
def split_and_orthogonalize(weight, num_heads, head_dim):heads = []for i in range(num_heads):start = i * head_dimend = start + head_dimhead_weight = weight[:, start:end]heads.append(orthogonalize(head_weight))return heads
Note that the code above is pseudocode. The orthogonalize function is a placeholder that must be replaced with a concrete orthogonalization implementation, and the shape and slicing direction of weight depend on the specific layout of the projection matrix. If in a real implementation the weight matrix is stored by input dimension or output dimension, the slicing approach may need to be adjusted.
The point of splitting by head is that it preserves the optimizer’s ability to constrain the distribution of matrix parameters while avoiding over-homogenization of different attention heads. For tasks that need fine-grained token routing, long-context retrieval, and complex semantic modeling, this strategy helps maintain training stability, but it still needs to be judged against actual experimental data.
Multi-token prediction: stronger training signal and long-range modeling
The second type of change is an enhancement at the training-objective level. Traditional autoregressive models usually predict only the next token, while the new generation of models introduces a multi-token prediction mechanism. In short, during training the model is asked not only to predict the current token but also several future tokens. This design does not change the autoregressive generation method at inference time, but it increases information density during training.
The value of multi-token prediction shows up in three main ways. First, it makes the model learn longer-range dependencies. Single-token prediction tends to make the model over-focus on local context, whereas multi-step prediction forces stronger long-range structural awareness. Second, it improves sample efficiency. Within the same training sample, the model can obtain multiple supervision signals at once, which reduces data waste in ultra-large-scale training. Third, it helps improve the sense of “planning” in code, math, and complex reasoning tasks. Many complex tasks are not single-step generation but require laying out structure in advance—for example, first determining function structure when writing code, then filling in local details. Multi-token prediction lets the model more naturally learn this step-by-step decision process.
For engineering configuration, multi-token prediction usually requires tuning the number of prediction steps, the auxiliary-loss weight, and the task-type mix. Too few prediction steps yield limited gains; too many may introduce noise and increase training complexity. In practice, parameters are typically tuned according to model scale, data quality, and task distribution.
Sparse attention: reducing long-context computation complexity
The third type of change is a sparse attention mechanism. Standard full attention has quadratic computational complexity on long sequences, which creates two problems: high compute consumption, and heavy pressure on KV cache and memory bandwidth. Sparse attention reduces overall computation by calculating attention relationships only between some key tokens.
Common approaches to sparse attention include local-window attention, global-anchor attention, block-sparse attention, and dynamic-routing attention. Local windows focus on nearby context and are good at capturing local syntax and local dependencies; global anchors retain a few key tokens to maintain global information flow; block sparsity divides the sequence into blocks and performs limited computation between them; dynamic routing selects token regions to attend to based on content relevance.
Combining sparse attention with low-rank KV compression can further improve long-context serving efficiency. Low-rank compression solves the problem of “each token’s KV representation being larger,” while sparse attention solves the problem of “too many relationships to compute between tokens.” Together, they optimize long-context capability at both the representation level and the computation level.
Common tuning mistakes in practice include: setting the sparse window too small, causing loss of local information; setting the number of global anchors too low, causing cross-segment reference drift; choosing an unreasonable block size, causing lower hardware utilization; and pursuing only speed while ignoring accuracy on long-text Q&A, long-code understanding, and multi-turn dialogue. It is recommended to validate through ablation experiments: separately disabling global anchors, adjusting local-window size, and changing block-sparse granularity, then comparing first-token latency, throughput, memory usage, and task accuracy.
Training-data strategy: quality and mixing behind 28.5T tokens
Increasing data scale from 23T to 28.5T does not mean simply adding more data. For ultra-large model training, data governance matters more. Pre-training data mainly provides broad language ability, world knowledge, and a code foundation; mid-training data is usually used to improve model performance on high-quality instructions, complex reasoning, code engineering, long-text understanding, and specific-domain tasks.
In building the data, several layers generally need attention. First, deduplication. If large-scale corpora contain a lot of repeated content, the model’s memorization increases but generalization drops. Second, quality filtering. Low-quality text, garbled content, templated content, and overly ad-heavy text all hurt the model’s expressive ability. Third, mixing-ratio adjustment. A reasonable proportion must be kept among general text, code, math, encyclopedias, technical documentation, and multi-turn dialogue. Fourth, staged training. The pre-training stage emphasizes coverage, while mid-training emphasizes capability calibration and task alignment.
Below is an illustrative data-mixing idea—not an official ratio, and not to be applied directly:
- General web and encyclopedia text: about 40%
- Code and engineering documentation: about 25%
- Math, logic, and reasoning data: about 20%
- Long-form and structured documents: about 10%
- Safety and compliance samples: about 5%
This kind of ratio needs to be adjusted dynamically based on the target scenario. If the model is mainly used for intelligent coding, the proportion of code and engineering documentation needs to increase; if it is mainly used for enterprise knowledge Q&A, long documents, tables, retrieval-augmented data, and multi-turn dialogue samples matter more. A more reliable approach is to build small-scale ablation experiments and observe the gains of different ratios on target metrics.
Reading the evaluation: the gap between leaderboard scores and real performance
Composite intelligence index benchmarks usually cover multiple dimensions such as knowledge understanding, logical reasoning, code generation, math ability, instruction following, and multi-turn dialogue. A model reaching roughly the 50-point range on some public evaluations indicates it already has fairly strong general task-processing ability. Being at the front among open-weight models also means it is no longer a lopsided model that only performs well on specific leaderboards.
Human-preference evaluations and competitive code benchmarks are closer to real usage scenarios. The former focuses on whether answers are natural, whether they match user intent, and whether they carry usable information; the latter focuses on whether code runs, whether it follows engineering conventions, and whether it handles complex context. If the new generation of models performs close to leading closed-source systems on these evaluations, it means it already has a high level in terms of actual interaction experience.
Still, developers should keep evaluation limitations in mind when using these models. First, public leaderboards may carry a risk of data contamination and cannot fully represent private-scenario performance. Second, different prompts, temperature settings, and decoding strategies can significantly affect results. Third, code tasks must be validated with compilation, testing, and static checks—not just by looking at the generated output. Fourth, long-context tasks require attention to loss of intermediate information, reference drift, and cross-segment consistency. Fifth, enterprise knowledge-base scenarios usually need retrieval augmentation, access control, and citation mechanisms; relying on parameter memory alone is not reliable.
MoE vs. dense model: a selection checklist
If your team is evaluating whether to adopt these ultra-large sparse expert models, you can refer to the following checklist:
- High proportion of long context—for example, long-document Q&A, long code-repository understanding, meeting-minutes analysis—favor models with MLA, KV compression, or sparse attention capabilities.
- Strong private-deployment needs—for example, finance, government, healthcare, and legal scenarios—should prioritize open-weight, fine-tunable, auditable models.
- High inference cost but complex tasks—MoE models can carry larger capacity with lower activated parameters, but you need to assess expert parallelism, memory bandwidth, and communication overhead.
- High concurrent request volume—focus on evaluating expert-routing load, dynamic batching, KV cache hit rate, and first-token latency.
- Sensitive to fine-tuning cost—confirm whether the model supports LoRA, full-parameter fine-tuning, continued pre-training, or domain adaptation, and assess data-preparation cost.
- Code tasks as the main workload—focus on code evaluations, compilation pass rate, unit-test generation ability, and defect-localization ability, rather than composite leaderboards alone.
Engineering deployment: from model capability to service capability
The real value of an ultra-large model ultimately depends on engineering capability. For MoE models, inference serving needs to focus on expert parallelism, tensor parallelism, pipeline parallelism, and dynamic batching. When there are many experts, uneven routing load can cause some devices to idle while others overload. In production, it is therefore usually necessary to monitor expert-activation distribution, request latency, cache hit rate, and long-text throughput.
For KV cache management, long-context scenarios require more refined eviction and reuse strategies. Techniques such as tiered caching, prefix caching, and session-level caching can keep high-frequency system prompts, tool definitions, and repeated context in higher-priority storage, reducing repeated computation. For intelligent-coding scenarios, code context often contains a lot of repeated project structure, dependency descriptions, and prompt templates; prefix caching can noticeably improve response speed.
For stability, you need logging services, monitoring and alerting, and metrics tracking. Key metrics include: first-token latency, per-token generation latency, long-context request throughput, expert-routing entropy, memory usage, cache hit rate, abnormal-output rate, and code compilation pass rate.
For enterprise applications, you can also connect model capabilities to general cloud infrastructure such as object storage, message queues, container platforms, logging services, and monitoring/alerting, forming a complete intelligent-application pipeline. For example, a code-generation service can write model output to object storage, place asynchronous tasks in a message queue, deploy the inference service on a container platform, and track task quality through logging services and monitoring alerts.
Typical use cases: code intelligence and enterprise knowledge processing
One of the best-fit scenarios for these ultra-large sparse expert models is intelligent coding and code assistance. Code tasks usually require the model to understand long context, cross-file dependencies, interface definitions, and engineering conventions. The MoE architecture can improve capabilities such as code generation, code completion, unit-test generation, code review, defect localization, and refactoring suggestions through expert specialization. Multi-token prediction helps the model plan variable naming, exception handling, and call-chain structure in advance when generating functions.
Another typical scenario is enterprise knowledge processing. Enterprise documents often contain tables, flowcharts, policy descriptions, and technical manuals, and long-context capability plus sparse attention can help the model better handle complex documents. But enterprise deployment cannot rely on model parameters alone; it also needs retrieval-augmented generation, permission isolation, data masking, and citation tracing. Only then can it balance accuracy, security, and auditability.
Risks, boundaries, and the enterprise compliance checklist
Although these models are capable, there are still boundaries to watch in practice. First, ultra-large model inference cost remains high—even though activated parameters are lower than total parameters, expert weights, cache, and communication overhead cannot be ignored. Second, MoE routing may be non-deterministic, so the same request can show subtle differences under different expert combinations. Third, long context does not equal full understanding—the model may miss key information or confuse intermediate segments. Fourth, code tasks must be validated by testing and cannot be judged correct by generated text alone. Fifth, enterprise scenarios need to address data compliance, access control, and output auditing to avoid leaking sensitive information.
For enterprise deployment, it is recommended to establish checklist items: Is data masking complete? Is permission isolation effective? Is there an audit log for outputs? Are sensitive entities filtered? Is citation tracing possible? Have code outputs been compiled and tested? Do long-document answers support source localization? Do multi-turn dialogues avoid leaking prior sensitive information?
Summary: the engineering value of sparse expert large models
The technical value of the new generation of ultra-large sparse expert models lies not in simply stacking parameters, but in jointly improving model value through architecture optimization, data governance, training-objective design, and engineering serviceability. Its core design includes: building a large-capacity, low-activation inference model with 744B total parameters, 40B activated parameters, and 256 experts; improving long-context training stability with MLA and per-head orthogonalization; strengthening long-range modeling and sample efficiency with multi-token prediction; reducing long-sequence computation complexity with sparse attention; and improving overall task performance with 28.5T tokens of data and mid-training.
For developers and enterprise technical teams, the truly valuable way to deploy is not to simply replace the existing model, but to restructure data flows, toolchains, and evaluation systems around the business scenario. Only when there is a closed loop among model capability, retrieval augmentation, task orchestration, monitoring/alerting, and security compliance can ultra-large sparse expert models genuinely become stable, reliable, and scalable productivity.
