MoE Foundation Model Explained: MLA, MTP, and Sparse Attention for Lower Long-Context Costs

    This article examines a publicly released MoE foundation model and explains how MLA KV compression, Muon Split optimizer adaptation, multi-token prediction, and DSA sparse attention are combined to reduce long-context inference and training costs. It covers parameter and activation scale, benchmark positioning, architecture tradeoffs, evaluation guidance, and practical deployment checks for production teams.

    Introduction: Foundation Models Are Moving Beyond Parameter Scale

    The first reaction to a new foundation model is often simply that it has become larger. When the technical report of a new open-source Mixture of Experts (MoE) foundation model was released, many practitioners described it as one of the best open-source models currently available. However, the report’s significance extends beyond size. It combines attention compression through MLA low-rank KV compression, optimizer adaptation through Muon Split, training-signal enhancement through multi-token prediction, and long-context acceleration through sparse attention within the same model. The shared goal is to lower the unit cost of long-context inference and training.

    This shift suggests that foundation-model competition is moving from raw parameter accumulation toward effective intelligence delivered per unit of compute. This article uses the information disclosed in the public technical report as its primary basis. It follows a practical path: fundamentals, benchmark positioning, architecture changes, evaluation and selection, and deployment. For each architectural change, it explains the general problem being solved, provides a conceptual analysis and illustrative code, and lists validation checks that teams can execute.

    Note: The parameter counts, activated counts, training-token totals, and benchmark rankings discussed in this article come from the model’s public technical report and are stated as of the disclosed version. Complete design details and hyperparameters should be based on the official report.

    Fundamentals: What 744B Total Parameters and 40B Activated Parameters Mean

    The basic configuration is 744B total parameters, 40B activated parameters per inference, 256 experts, and 80 layers. Compared with the previous generation, there are four major changes.

    Dimension Previous generation Current generation Change Business impact
    Total parameters 355B 744B About doubled Higher knowledge capacity and specialization ceiling; weight memory demand nearly doubles; deployment usually requires multi-node, multi-GPU infrastructure.
    Activated parameters per token 32B 40B +25% Modest increase in compute per token; limited increase in inference latency and compute cost.
    Sparsity ratio (total / activated) About 11× About 19× Significantly higher More knowledge carried per unit of compute; better cost-efficiency for long-context and high-concurrency scenarios.
    Total training tokens 23T 28.5T (27T pretraining + 1.5T mid-training) +24% Mid-training targets long context, code, and reasoning; these specialized capabilities should be validated in internal evaluations.
    Layers Fewer 80 More layers More layers must be cached; memory requirements should be evaluated together with MLA.

    Two details are easy to overlook.

    First, most of the growth is in capacity, not in compute. Total parameters have doubled, but activated parameters have increased by only 8B. The increase in computation per token during inference is therefore limited, while the knowledge capacity that can be carried is nearly doubled. This is achieved by introducing more experts, effectively decoupling capacity growth from per-token computation growth. This is one reason MoE can scale to a large size while keeping single-inference cost within an acceptable range.

    Second, the data recipe includes a distinct mid-training stage. Of the 28.5T training tokens, 27T are used for pretraining and 1.5T for mid-training. Mid-training refers to continued training performed after pretraining and before post-training, typically using a smaller amount of higher-quality or more targeted data. Common approaches strengthen specific capabilities such as high-quality corpora, long context, code, and reasoning. This layered strategy has become increasingly common in recent technical reports from multiple organizations, although exact data ratios and corpus compositions should be based on each organization’s own report.

    This layering is essentially a division of labor. Pretraining covers knowledge and language distributions as broadly as possible. Mid-training concentrates resources on filling weak areas. Post-training then addresses interaction problems such as alignment and instruction following. When rebuilding a training-data pipeline, mid-training data must still be subject to strict deduplication, quality filtering, and mixing policies. Otherwise, it can easily turn into repetitive corpus drilling rather than targeted capability enhancement.

    Benchmark Positioning: The Gap Between Open and Closed Models Is Narrowing

    The technical report discloses two main types of external evaluation results. Rankings mentioned below refer to results as of the disclosed report or specified evaluation version and may change in future retests.

    1. Composite intelligence index: In version v4.0 of this index, the model scores 50, ranking first among open-source models and only behind the leading closed-source models available internationally.
    2. Arena-style evaluations: The model ranks first among open-source models in both text arenas and code arenas, placing it overall in the same tier as some flagship closed-source models.

    There are three useful ways to interpret these results.

    • Look at the tier, not just the rank. The gap between the top open-source model and the top closed-source model has narrowed to a small difference within the same tier. For many enterprise applications, this means open-source models are already feasible substitutes for flagship closed-source models in many general scenarios.
    • Look at specialty dimensions. Overall scores only reflect average performance. Code-arena rankings are more relevant for developer-productivity products, while text-arena performance is closer to conversational, writing, and customer-support scenarios.
    • Look at reproducibility. Arena-style evaluations are based on pairwise blind voting from real users, which is closer to real experience. However, they can be influenced by prompt distribution, answer length, and formatting style. Composite-index evaluations are easier to compare across models, but they cannot cover every business scenario. These results should not be treated as a guarantee of performance or ranking.

    Evaluation results are therefore best used as a screening mechanism for candidate models, not as the final selection conclusion.

    Architecture Overview: Three Changes, Each Targeting a Different Cost

    This generation differs from the previous one in three main ways:

    • MLA + Muon Split: Uses low-rank compression to reduce the memory cost of KV cache, while adapting an orthogonalized momentum optimizer to the MLA structure.
    • Multi-token prediction (MTP): Predicts multiple future tokens at each position, increasing training-signal density and data efficiency.
    • DSA sparse attention: On ultra-long sequences, each query only attends to a small number of the most relevant tokens, reducing attention computation cost.

    These three changes target efficiency bottlenecks at different levels: memory bottlenecks, where KV cache grows rapidly as context length increases; optimization bottlenecks, where training becomes harder to stabilize and accelerate as parameter scale grows; and computation bottlenecks, where the quadratic growth of attention becomes difficult to tolerate as sequence length increases.

    Change 1: MLA + Muon Split

    MLA: Compressing KV Cache into a Low-Dimensional Latent Vector

    The attention mechanism used by the model is Multi-head Latent Attention (MLA). The core idea is not to cache full Key and Value tensors for every layer. Instead, the hidden state is first projected into a low-dimensional latent vector. Only that latent vector is cached, and the Key and Value tensors for each attention head are reconstructed through an up-projection when needed.

    This is equivalent to replacing a full KV cache with a low-rank bottleneck. Open-source implementations commonly use a latent dimension around 512, while the concatenated KV dimension of all KV heads in traditional multi-head attention can reach tens of thousands. The resulting memory difference is substantial.

    Let L be the number of layers, D the KV dimensions cached per layer per token, b the number of bytes per value, S the sequence length, and B the batch size. The KV-cache memory can be approximated as:

    1. KV cache (bytes) ≈ 2 × L × D × b × S × B

    The coefficient 2 accounts for Key and Value. Larger D or longer S increases memory pressure, especially in long-context settings. MLA reduces D from the concatenated dimension of all KV heads to the latent-vector dimension, lowering the expression proportionally.

    Two points should be noted. MLA usually retains a small set of rotary-position-encoding-related dimensions separately to express positional information, which slightly increases cache size. Even with this overhead, total cache usage is still substantially lower than in an uncompressed design.

    Why Long-Context Scenarios Benefit Most

    In traditional implementations, processing a 128K-token context requires caching full Key and Value tensors for every layer and every request. As multi-turn dialogue accumulates context, and as concurrent requests increase, KV cache can become the dominant consumer of memory. This directly limits three things: maximum concurrency, maximum supported context length per instance, and overall cost per token.

    After MLA compression, latent vectors can be processed more uniformly with batch matrix operations, which is generally friendlier to GPU-based accelerators. It also creates a positive cycle: less cache means more available memory, which allows larger batches, which increases throughput. This mechanism is a major reason teams migrating to MLA-style architectures can improve long-text service costs. However, the actual improvement depends on public benchmarks, reproducible experiments, or internal stress tests, because implementations and kernels can differ significantly.

    Training Problem: Optimizer Assumptions and Matrix Operations

    MLA’s computation path includes many down-projection and up-projection low-rank decomposition structures. Introducing an optimizer such as Muon creates a mismatch.

    Muon is a class of momentum optimizers based on matrix orthogonalization (MomentUm Orthogonalized by Newton-Schulz). It uses a weight matrix as the update unit, performs an approximate orthogonalization on the momentum matrix, and then updates the parameters. Intuitively, this makes updates more evenly distributed across the singular directions of the matrix rather than concentrated in a few dominant directions. Compared with classic elementwise adaptive optimizers, it often converges faster in large-scale training. Multiple organizations have verified this in public technical reports. However, it has one prerequisite: parameters must be two-dimensional matrices.

    • Conflict with non-matrix parameters: MLA contains one-dimensional normalization weights and biases. Forcing them into a matrix representation is mathematically meaningless.
    • Conflict with low-rank decomposition structures: Down-projection and up-projection matrices lose some of the benefits of the decomposition if they are orthogonally updated as whole matrices, because matrix-level update behavior can leak into both sides of the low-rank decomposition.
    • Update scale requires recalibration: Orthogonalization changes the structure of the update scale, so the learning rate must be recalibrated. Otherwise, training can become unstable.

    The team’s final approach is to split parameter groups, called Muon Split.

    Split Strategy and Illustrative Implementation

    Parameters are grouped according to whether they are suitable for matrix-style updates:

    • Matrix parameter group: Two-dimensional hidden weight matrices are assigned to the orthogonalization-based optimizer to exploit matrix-level updates for better training efficiency.
    • Standard parameter group: One-dimensional parameters, embedding layers, output layers, and MLA low-rank decomposition matrices are assigned to a mature elementwise adaptive optimizer to preserve training stability.

    In the code below, muon_lr is the learning rate for the orthogonalization optimizer, and adam_lr is the learning rate for the elementwise adaptive optimizer. They are configured independently. model is the model object being trained.

    1. def split_param_groups(model, muon_lr, adam_lr):
    2. """Split parameters into two groups based on whether they are suitable for matrix-style updates (illustrative)."""
    3. muon_params, adam_params = [], []
    4. for name, p in model.named_parameters():
    5. if not p.requires_grad:
    6. continue
    7. # Only two-dimensional hidden weight matrices enter the Muon group.
    8. # One-dimensional parameters, embeddings, output heads,
    9. # and MLA down/up-projection matrices use the standard optimizer group.
    10. if p.ndim == 2 and "embed" not in name and "lm_head" not in name \
    11. and "q_a_proj" not in name and "kv_a_proj" not in name:
    12. muon_params.append(p)
    13. else:
    14. adam_params.append(p)
    15. return [
    16. {"params": muon_params, "lr": muon_lr, "use_muon": True},
    17. {"params": adam_params, "lr": adam_lr, "use_muon": False},
    18. ]

    In engineering practice, two details must be handled alongside this split. First, learning rates should be decoupled: each group has its own learning rate to prevent orthogonalized updates from interfering with standard parameters. Second, weight decay and gradient clipping should be calibrated for each group according to its update scale to reduce the risk of spikes in later training stages.

    Benefits, Costs, and Validation

    Change Main benefit Main cost Suggested validation
    MLA Significantly lower KV-cache memory on the inference side, improving long-context concurrency and unit cost; on the training side, reduces optimizer-state memory pressure. Additional positional-encoding-related dimensions in cache; small extra computation from up-projection; dedicated kernels are needed to realize actual gains. Fix model and input length; compare per-sequence KV memory before and after compression; compare maximum throughput under different concurrency levels; confirm that the inference backend natively supports MLA.
    Muon Split Two-dimensional hidden weight matrices benefit from matrix-level updates, enabling faster training convergence; parameter grouping keeps more parameters on the standard optimizer path. Requires explicit parameter grouping and dual optimizer-state management; distributed sharding must match group information, otherwise updates may be missed or duplicated; hyperparameter tuning space grows. Run loss-curve and gradient-norm comparisons on small-scale models; check that every parameter is updated exactly once across all parallel dimensions; monitor spikes during learning-rate warmup.

    Change 2: Multi-Token Prediction (MTP)

    Problem: Training Signals Are Too Sparse

    Standard autoregressive training predicts only the next token at each position. A sequence of length L provides only L supervision signals. For training runs involving tens of trillions of tokens, low signal density means that the same data may require more passes to converge. It can also lead to insufficient modeling of farther context, making long-range dependencies harder to learn.

    Method: Predict Multiple Future Tokens

    The model adds several prediction modules outside the main model. Each module predicts a token at a farther position, and an additional output head computes the loss.

    Variable conventions in the code below (the following is pseudocode, used only to illustrate how losses are combined, and cannot be run directly):

    • num_mtp_depth: MTP depth, i.e., the number of additional future tokens predicted.
    • weight: loss-weight table indexed by depth, usually decreasing with depth.
    • lm_head / head: vocabulary projection heads; the main model and MTP modules may share them or use separate ones.
    • embed: token embedding layer.
    • labels_shift(d): the target sequence obtained by shifting labels right by 1 + d positions, i.e., predicting the token at position t + 1 + d.
    • h: hidden states output by the backbone.
    1. # MTP training illustration: the main model predicts t+1, while MTP modules predict tokens farther ahead.
    2. h = backbone(input_ids) # Main model hidden states
    3. loss = cross_entropy(lm_head(h), labels_next1) # Main task: predict t+1
    4. for depth in range(1, num_mtp_depth + 1): # Each depth predicts one token farther ahead.
    5. h_mtp = mtp_module[depth](h, embed(input_ids)) # Reuse the main model representation
    6. loss = loss + weight[depth] * cross_entropy(head, labels_shift(depth))

    Structural illustration:

    1. Input tokens ──> Backbone ──> Main output head ──> Token t+1
    2. │ │
    3. │ └──> MTP module 1 ──> Token t+2
    4. └─────> MTP module 2 ──> Token t+3

    Benefits

    • Higher signal density: The same text can provide multiple prediction targets, improving data utilization.
    • More forward-looking representations: The model is forced to encode context information from farther positions, which can help with long-range dependencies and planning-related abilities.
    • Natural support for speculative decoding: Trained MTP modules can act as lightweight draft models. They can propose multiple candidate tokens at once, and the main model can verify them in parallel. The higher the acceptance rate, the greater the end-to-end decoding speedup.

    Costs and Considerations

    • Training memory and computation increase because of the additional modules and output heads.
    • Loss weights require tuning. Predicting tokens farther ahead is harder, and overly large weights can interfere with main-task convergence, so weights usually decrease with depth.
    • MTP does not need to be enabled all the time during inference. Teams can use regular decoding or MTP-accelerated decoding depending on latency and throughput requirements.
    • Post-training strategies must remain consistent. If fine-tuning and alignment treat MTP modules differently from pretraining, draft-model quality may degrade, and speculative-decoding acceptance rates can drop significantly.

    Change 3: DSA Sparse Attention

    Problem: Quadratic Growth of Attention Computation

    In standard attention, every token computes relevance with all historical tokens, leading to O(L²) complexity. Doubling the sequence length makes computation roughly four times larger; a tenfold increase makes it roughly one hundred times larger. In contexts of 128K tokens or more, this becomes a major source of inference latency and cost.

    Method: Select Tokens First, Then Compute Attention

    1. A lightweight scorer, or indexer, assigns a relevance score to each historical token for every query token.
    2. Only the top-k highest-scoring tokens are retained, forming a sparse attention mask.
    3. Attention computation is performed only over these k positions.
    1. # Sparse attention illustration: score -> top-k selection -> sparse attention
    2. scores = indexer(q, k_cache) # Lightweight scorer; can use low precision or a low-rank representation
    3. indices = scores.topk(k, dim=-1).indices # Select the k most relevant tokens
    4. out = sparse_attention(q, k_cache, v_cache, indices) # Compute only over these k positions

    Necessary qualification about complexity. The main attention part (QK^T and weighted summation over the selected k positions) has complexity O(L · k · d). When k is constant and much smaller than L, this is a substantial reduction compared with standard attention’s O(L² · d). However, this is only a partial conclusion:

    • If the indexer still scores every historical token individually, the scoring stage itself may contain quadratic overhead, approximately O(L² · d_idx). In that case, end-to-end complexity may still contain a quadratic term, although with a much smaller constant than full attention.
    • Only under certain conditions can overall complexity be approximated as O(L · k): when the indexer uses low-rank projection, such as low-dimensional inner products against compressed latent vectors; when it uses compressed or pooled historical representations; when it adopts blockwise scoring; or when a dedicated sparse kernel fuses scoring and selection.
    • Complexity convention: The complexities above refer to floating-point operation magnitude of the attention part for a single sequence, excluding MoE/MLP computation. L is sequence length, k is the sparse budget, d is head dimension, and d_idx is scorer dimension. Actual latency also depends on memory bandwidth and kernel implementation, so teams must confirm with their own stress tests rather than estimating from FLOPs alone.

    Combining with MLA: Structural Reuse

    DSA and MLA are naturally complementary. MLA already prepares a compressed low-dimensional KV representation for each layer, which is suitable as input features for the scorer. The scorer only outputs scalar scores, making it much cheaper than full attention. After sparse selection, attention only needs to expand full Key and Value tensors over a small number of positions, which fits MLA’s cache structure seamlessly. Thus, “compressed cache” reduces memory usage, while “sparse computation” reduces compute cost, producing additive benefits.

    Engineering Tradeoffs

    • Stability of the selection strategy: Top-k is a discrete operation. Training requires a reasonable way to backpropagate gradients, and the scorer may need appropriate supervision to prevent unstable training caused by selecting wrong positions.
    • Kernel support: Sparse attention needs custom kernels. Without them, a generic framework may fall back to inefficient implementations, increasing overhead.
    • Long-context extrapolation: The ability of the sparse pattern to generalize to ultra-long sequences should be validated around target lengths, not only with short-text evaluations.

    How the Three Changes Work Together: A Sparsification Thread

    Viewed together, the three changes point in the same direction: improving sparsification and resource-allocation efficiency.

    • MoE’s sparse activation prevents capacity growth from causing proportional compute growth.
    • MLA’s low-rank compression prevents context growth from causing proportional memory growth.
    • DSA’s sparse attention prevents sequence growth from causing proportional attention-compute growth.
    • Multi-token prediction generates more effective learning signals from the same amount of training data.

    The actual magnitude of these combined benefits must be verified through public benchmarks, reproducible experiments, or internal stress tests, because results can differ substantially across kernel implementations and hardware conditions. For enterprises, the implication is that when evaluating whether a foundation model fits their business, they should look beyond parameter scale and overall scores. They should assess performance under real constraints such as long context, high concurrency, and low cost.

    Reading Evaluations and Model Selection

    How should MoE models be selected? Start with three constraints. Is the business long-document analysis, code generation, or conversational interaction? The relative importance of long context, reasoning depth, and tool calling is completely different. Use constraints first to eliminate unsuitable options, then compare scores.

    What evaluations matter for code and long-text scenarios? Composite intelligence indexes are useful for shortlisting candidates. Arena-style results help judge real interactive experience. Code arenas are especially relevant for developer-productivity products, while text arenas are closer to dialogue, writing, and customer-support scenarios.

    How should an internal evaluation set be built? Construct 100 to 500 representative samples from real business data. Cover typical tasks and edge cases such as ultra-long inputs, multi-turn references, structured outputs, and failure retries. Form an internal regression set and fix a scoring rubric.

    How should cost and latency be estimated? Use target context length and concurrency to estimate memory and throughput requirements, then judge the feasibility of single-GPU or multi-GPU deployment. The formulas are described in the deployment checklist below.

    When evaluating models, three cautions matter. Avoid relying on a single leaderboard ranking. Be aware of possible data contamination, where evaluation data overlaps with training data. Pay attention to whether the prompt distribution of the evaluation matches your business.

    Deployment Checklist

    Step 1: Lock Down Input and Output Token Lengths

    Measure the input-length distribution (P50/P95/max) and output-length distribution of the business workload. Long-text RAG commonly uses 32K to 128K input tokens and 1K to 4K output tokens. Conversational scenarios often have short inputs but many turns, so prefix caching should be enabled. Without these two distributions, all later estimates are speculative.

    Step 2: Estimate GPU Memory

    1. Weight memory ≈ total parameters × bytes per parameter ÷ expert-parallel degree
    2. BF16: 744B × 2B ≈ 1.4 TB
    3. FP8: 744B × 1B ≈ 744 GB
    4. Peak KV cache ≈ 2 × L × D × b × S̄ × B ÷ 1024³ (GB)
    5. L: number of layers (80 in this model)
    6. D: cached dimensions per layer per token
    7. b: bytes per value
    8. S̄: average sequence length
    9. B: concurrent batch size
    10. Per-instance GPU memory ≈ weight shard + peak KV cache × (1 + 20%–30% headroom)
    11. + activation/communication buffers
    12. + CUDA context and memory fragmentation

    Illustrative calculation (numbers are only for order-of-magnitude explanation, not official data): If D ≈ 576 (512 latent-vector dimensions plus positional-encoding dimensions), b = 2, and L = 80, then per token per layer the cached size is about 2 × 80 × 576 × 2 ≈ 184 KB. A 128K-token sequence would need about 23 to 24 GB of KV cache. With an uncompressed design where the KV concatenated dimension is 8192, the same length would need about 340 GB. The difference illustrates the magnitude of MLA’s benefit.

    Step 3: Define Concurrency and Deployment Specifications

    Use target QPS × average total tokens ÷ measured single-GPU throughput to estimate the required number of GPUs. Then validate whether the memory formula in Step 2 allows the workload to fit. Note that MoE expert parallelism introduces all-to-all communication. If cross-node bandwidth is insufficient, throughput can fall significantly below theoretical values. It is advisable to compare different parallel-sharding strategies under the same evaluation conditions.

    Step 4: Validate MTP Draft-Model Acceptance Rate

    • Collect metrics: average accepted length (number of draft tokens accepted per decoding round), token acceptance rate, and the time share of drafting versus verification.
    • Threshold convention: MTP is worthwhile only when the average accepted length is large enough that the saved main-model decoding steps exceed the extra cost of draft generation plus verification. Empirically, when average accepted length is below about 1.5 to 2 tokens, the benefit is often offset by overhead. This threshold is only a rule of thumb and must be confirmed on your own hardware and business prompts.
    • Test by scenario: Code-like, high-determinism tasks usually have higher acceptance rates than open-ended creative tasks, so MTP can be enabled selectively through staged rollouts.

    Step 5: Validate Sparse-Attention Kernel Performance

    1. Confirm the kernel path is actually active: Check whether the inference backend supports MLA or sparse attention, avoiding a generic fallback implementation. A profiler can be used to confirm the kernel names that are actually invoked.
    2. Break down latency: Measure indexer scoring, top-k selection, and the core sparse-attention computation separately. Verify that scoring truly uses low-rank or blockwise strategies rather than full scoring.
    3. Sweep k values: Measure throughput and accuracy at multiple k values and identify the inflection point.
    4. Check numerical consistency: Compare output error against dense attention on short sequences to confirm that the sparse approximation does not harm key tasks.
    5. Test length extrapolation: Stress-test at training length, twice the length, and the maximum target business length. Observe whether the latency curve deviates from linear growth.

    Step 6: Continuous Monitoring After Launch

    Establish internal regression evaluation and online metric monitoring. Focus on failure rates for long-context tasks, first-token latency, and cost per token. This helps detect capability regressions after version upgrades.

    Conclusion

    The most noteworthy aspect of this technical report is not the 744B parameter count itself, but the balance it strikes between parameter growth and efficiency optimization: MLA reduces memory pressure, Muon Split adapts the optimizer to the new attention structure, multi-token prediction increases training-signal density, and sparse attention controls the computation cost of long sequences.

    Together, these changes form a technical path of “sparsification plus efficiency.” For developers, understanding this path and mastering its validation criteria — which benefits are structural, which depend on kernel implementations, and which must be confirmed through internal stress tests — is more valuable than memorizing any single parameter.