Open-Source MoE Architecture Explained: MLA, MTP, DSA, and Asynchronous RL

    This article analyzes a roughly 40-page MoE large-model technical report and breaks down how a 744B-total / 40B-activated open-source model has narrowed the gap with closed flagship models. It walks through scale fundamentals, the MLA low-rank attention mechanism and its KV-cache savings, multi-token prediction (MTP), DSA sparse attention, Muon Split optimizer adaptation, and a fully asynchronous agentic RL framework. For each design, it explains the specific bottleneck being solved, where the gains come from, and what the trade-offs are, then covers deployment, evaluation, and risk considerations so you can decide what to port into your own training or inference systems.

    Introduction: Open-Source Models Are Closing the Gap, but the Real Story Is Bottlenecks and Trade-offs

    Top-tier closed-source models still lead in overall capability, yet the gap between the open-source camp and flagship models has narrowed noticeably. According to several public technical reports and leaderboard figures, open-source models now score in a high range on comprehensive benchmarks and have begun to sit in the same conversation tier as closed-source flagships.

    A technical report of roughly 40 pages on a large MoE model has drawn attention. Its most frequently cited keywords include: MLA low-rank attention, MTP multi-token prediction, DSA sparse attention, Muon Split optimizer adaptation, and a fully asynchronous agentic reinforcement learning framework. Using this report as the primary material, this article breaks things down in the order of scale fundamentals → architecture evolution → training paradigm → supporting infrastructure → engineering deployment.

    One point deserves emphasis: dissecting each block does not mean copying each item. For developers who want to evaluate or borrow these techniques, the more valuable questions are:

    • What specific bottleneck does each design solve?
    • Where do the gains come from, and what is the cost?
    • Which ideas can transfer into your own model training or inference system?
    • What precision, memory, kernel, and evaluation risks must be confirmed before transferring them?

    1. Scale Fundamentals: How 744B Total Params and 40B Activated Params Shape Deployment

    1.1 Key Numbers: What Total vs. Activated Params Actually Mean

    Metric Previous Gen Current Gen Change
    Total params 355B 744B ~2.1x
    Activated params per token 32B 40B ~1.25x
    Number of experts — 256 —
    Layers — 80 —
    Training data (pretraining + mid-training) 23T tokens 28.5T tokens (27T + 1.5T) ~1.24x

    The core of MoE (Mixture of Experts) is sparse activation. Each layer contains multiple parallel expert networks, and a router selects a few experts per token. As a result, “total params” and “per-token compute” are decoupled.

    The ratio logic in this report is fairly clear:

    • Total params roughly double: model capacity expands mainly on the expert side, to carry more knowledge, patterns, and long-tail capability.
    • Activated params grow only about a quarter: inference-side per-token compute grows relatively mildly.
    • Training data expands in step: pretraining lays the foundation, while mid-training shapes capabilities in a targeted way.

    But the cost is also clear: all expert weights must stay resident in memory, so memory capacity and access pressure are far higher than for a dense model with the same activated count. In other words, MoE lowers the per-token compute cost but does not lower the parameter storage cost.

    1.2 Benchmark Results: A High-Score Range by the Report’s Own Figures

    By the report’s figures, the model scores 50 on a comprehensive evaluation and ranks near the top among open-source models on text and code arena leaderboards that reflect real user preferences, putting it in the same tier as top flagship models overall.

    The significance of this kind of result is that the competition focus for open-source models has shifted from “can it be used” to “how large is the remaining gap with closed-source flagships in specific scenarios.” However, leaderboards fluctuate with version updates, question changes, and the passage of time. When making a real selection, you should still cross-check against your own business evaluation set rather than relying on a single public leaderboard.

    1.3 Training Data: Pretraining Builds the Base, Mid-Training Transfers Capability

    The 28.5T tokens break into two parts:

    • 27T pretraining tokens: learn general language, knowledge, reasoning, code, and other foundational abilities.
    • 1.5T mid-training tokens: directionally shape base capabilities at relatively low cost, for example strengthening reasoning, code, or agent behavior.

    What developers should pay attention to is the “data quality first” ratio philosophy. The report stresses that high-quality curated data can improve key capabilities more efficiently than ordinary data of the same volume. This means post-training and mid-training are not about simply piling up tokens, but about placing high-value data during the windows when key capabilities form.

    2. MLA Low-Rank Attention: How It Works, What It Costs, and How Muon Split Adapts

    2.1 MLA: Trading Low-Rank Compression for KV-Cache Memory Efficiency

    The model in this report uses MLA (Multi-head Latent Attention) for its attention mechanism. It shares a common origin with another mainstream MoE technical route, and its goal is to preserve expressive power while reducing KV-cache pressure during long-context and high-concurrency inference.

    How it works: MLA jointly compresses each token’s key (Key) and value (Value) into a low-dimensional latent vector. During inference, only this vector needs to be cached; before use, it is up-projected back into per-head keys and values via an up-projection matrix. At the same time, the positional-encoding components are “decoupled” into a small dimension shared across heads, so positional awareness is separated from the low-rank compression.

    The per-layer KV cache of different attention schemes is compared below, using a consistent definition: the per-layer, per-token cache formula, computed for relative scale with (c=4D), (r=D/2), (H=128). Here (H) is the number of heads, (D) the per-head dimension, (c) the latent dimension, and (r) the decoupled positional component dimension.

    Scheme Per-layer per-token cache formula With (c=4D, r=D/2, H=128) Relative to MHA
    Multi-head attention (MHA) (2HD) (2 \times 128D = 256D) 1
    Multi-query attention (MQA) (2D) (2D) 1/128
    Grouped-query attention (GQA, 8 groups) (2 \times 8D) (16D) 1/16
    MLA (c + r/2) (4D + D/4 = 4.25D) (4.25 / 256 \approx 1/60.2)

    Under the same assumptions, MLA’s per-layer, per-token cache is about (1/60) of MHA, far below MHA. But compared with MQA, MLA does not simply compress to the minimum; instead, it retains more expressive power through a low-rank latent representation. Each head’s query can still keep its full dimension, which makes it better suited to scenarios sensitive to long context, high concurrency, and inference cost.

    2.2 The Problem: Muon with MLA Underperforms a Simpler Scheme

    Training revealed a counterintuitive result: using a Muon-style optimizer, whose core is matrix orthogonalization, together with MLA, the training quality could not catch up with the structurally simpler GQA-8 scheme.

    The basic idea of Muon is:

    • For a 2D weight matrix, first accumulate first-order momentum.
    • Then use Newton-Schulz iteration to approximate the “orthogonal polar factor” of the momentum matrix, effectively scaling all non-zero singular values to the same scale.
    • Finally, update the parameters using the orthogonalized matrix multiplied by the learning rate.

    This balances the update step sizes across different directions and reduces the anisotropy of parameter updates. In practice, it often converges faster than traditional adaptive optimizers.

    But Muon and MLA may “clash.” From an optimization-geometry perspective: MLA concatenates the key-value projections of multiple heads into the same latent space. The matrix contains both low-dimension key projections and higher-dimension value projections, and the heads have a functional division of labor. Applying orthogonalization uniformly to the entire concatenated matrix forces all singular directions to the same update scale, which tends to flatten the importance differences that should exist between different heads and subspaces, thereby weakening the differentiated representation that MLA relies on.

    2.3 The Solution: Muon Split, Orthogonalizing Per Head

    The team’s solution is Muon Split:

    • Instead of orthogonalizing the whole concatenated projection matrix.
    • Split the matrix per attention head, then perform orthogonalized updates head by head.

    This way, each head gets an update step size matched to its own scale, preserving MLA’s low-rank structure while restoring Muon’s convergence advantage.

    The following code is an illustrative implementation to show the idea of “per-head split + Newton-Schulz orthogonalization.” It is not equivalent to a production-grade Muon implementation; real training also needs engineering details such as distributed strategy, numerical stability, kernel efficiency, and gradient clipping.

    1. import torch
    2. def orthogonalize(m: torch.Tensor, steps: int = 5, eps: float = 1e-7) -> torch.Tensor:
    3. """
    4. Illustrative: approximate the matrix's polar factor with Newton-Schulz iteration.
    5. Input m has shape [d_out, d_in].
    6. Output has the same shape as input, and the input's dtype is preserved.
    7. Normalization: Frobenius norm, used to pull the iteration into a more stable range.
    8. """
    9. X = m.to(torch.float32)
    10. X = X / (X.norm(p="fro") + eps)
    11. for _ in range(steps):
    12. # Push non-zero singular directions gradually toward 1, keep zero singular directions at 0
    13. X = 0.5 * (3.0 * X - X @ X.transpose(-1, -2) @ X)
    14. return X.to(m.dtype)
    15. def muon_split(grad: torch.Tensor, heads: int) -> torch.Tensor:
    16. """
    17. Split the projection matrix's momentum matrix by head, then orthogonalize per head.
    18. grad: momentum matrix with shape [heads * d_out, d_in].
    19. Returns: update matrix with the same shape as input.
    20. """
    21. if grad.ndim != 2:
    22. raise ValueError("grad must be 2D: [heads * d_out, d_in]")
    23. if heads <= 0 or grad.shape[0] % heads != 0:
    24. raise ValueError("heads must divide grad.shape[0]")
    25. d_out = grad.shape[0] // heads
    26. out = torch.empty_like(grad)
    27. for h in range(heads):
    28. block = grad[h * d_out:(h + 1) * d_out, :]
    29. out[h * d_out:(h + 1) * d_out, :] = orthogonalize(block)
    30. return out

    A more common equivalent approach in real engineering is to first reshape the projection weights into a 3D form [heads, d_out, d_in] by head, then perform orthogonalization over the batch dimension to reduce kernel launch overhead. The general lesson from this change is: an optimizer’s geometric assumptions must match the model’s structural characteristics.

    2.4 Transfer Notes

    If you want to port MLA or a Muon-style optimizer to your own model, focus on validating:

    • Whether orthogonalization is done on the whole matrix or per head/per subspace.
    • Whether the momentum matrix is split by attention head.
    • Whether the number of orthogonalization iterations is sufficient.
    • Whether float32 is used for stable computation.
    • Whether gradient-scale anomalies or loss spikes appear early in training.
    • Whether convergence speed has been strictly compared against a baseline GQA scheme.

    3. Multi-Token Prediction (MTP): Strengthening the Training Signal

    3.1 From “Guessing the Next Word” to “Guessing Multiple Words at Once”

    Traditional autoregressive language models predict only the next token at each position, so the supervision signal is relatively sparse. Multi-token prediction (MTP) adds several lightweight prediction modules on top of the main prediction objective: given the same context, it predicts the 2nd, 3rd, and further future tokens, effectively lengthening the model’s “prediction horizon” during training.

    Taking the prediction of two future tokens as an example, the illustrative code is:

    1. import torch
    2. import torch.nn as nn
    3. # Illustrative: the model body outputs a hidden state for each position
    4. class ModelWithMTP(nn.Module):
    5. def __init__(self, vocab_size, hidden_size, num_layers):
    6. super().__init__()
    7. self.embed = nn.Embedding(vocab_size, hidden_size)
    8. self.blocks = nn.ModuleList([Block(hidden_size) for _ in range(num_layers)])
    9. def forward(self, ids):
    10. # Stacked blocks with a causal mask, producing a hidden state per position
    11. return self.blocks(self.embed(ids))
    12. # Illustrative: multi-token prediction loss in the training loop
    13. mtp_head = nn.Linear(hidden_size, vocab_size) # Can chain lightweight prediction modules; simplified here
    14. mtp_depth = 2 # Number of extra tokens to predict
    15. lam = 0.3 # Auxiliary loss weight
    16. total_loss = main_loss = criterion(main_logits, labels)
    17. for k in range(1, mtp_depth + 1):
    18. # Align to the k-th future token; a real implementation must handle sequence boundaries and padding
    19. shifted = torch.roll(labels, shifts=-k, dims=1)
    20. loss_k = criterion(mtp_head(h), shifted)
    21. total_loss = total_loss + lam * loss_k

    3.2 Benefits and Costs

    Benefits:

    • Provides a richer supervision signal at each position, so the model learns cross-token local planning and structural constraints earlier.
    • Especially helpful for tasks like code and reasoning that require “planning ahead.”
    • After sufficient training, these prediction modules can also be reused for inference acceleration, giving the dual value of “enhanced learning during training and assisted generation during inference.”

    Costs:

    • The extra prediction modules increase training compute and memory usage.
    • The auxiliary-loss weight needs tuning; if too large, it may interfere with the main objective.
    • If usage differs between the training and inference phases, the benefits may not fully materialize.

    The general lesson is that the design of the pretraining objective is itself an important architectural lever. You can change the kind of capability the model learns without changing the final inference structure.

    4. DSA Sparse Attention: Filter First, Then Compute, to Cut Long-Context Cost

    4.1 The Dense-Attention Bottleneck: Quadratic Compute and Linear Cache Growth

    Standard attention compute grows quadratically with context length, while the KV cache grows linearly. In scenarios like long-document understanding, multi-turn agent tasks, and repository-level code understanding, these two costs quickly become the main bottleneck.

    4.2 How It Works: Filter First, Then Compute

    The idea behind DSA (sparse attention) is “filter first, then compute”:

    1. Use a lightweight index module to quickly estimate a relevance score between each query and its candidate keys.
    2. Each query keeps only the top (k) highest-scoring keys for the attention computation.
    3. All other keys are skipped directly.

    This reduces each query’s compute from being proportional to context length to being proportional to (k).

    The code below is an illustrative implementation, focused on causal masking, Top-k applying only to visible keys, and the definition of gather. For bidirectional encoding, replace the causal visibility matrix with an all-visible matrix; for autoregressive decoding, you must ensure future positions remain invisible.

    1. import math
    2. import torch
    3. def causal_visible(seq_len: int, device=None):
    4. """Generate the causal visibility matrix: True means visible."""
    5. return torch.tril(torch.ones(seq_len, seq_len, dtype=torch.bool, device=device))
    6. def dsa_attention(q, k, v, W_idx, k_top=256):
    7. """
    8. Illustrative: query-side index scoring + Top-k selection + attention only over visible keys.
    9. q, k, v: [B, H, S, D]
    10. W_idx: matrix projecting q/k into a low-dimensional index space [D, d_idx]
    11. k_top: maximum number of keys kept per query
    12. Note: this expresses the logic with broadcasting and gather; a production
    13. implementation typically needs a custom kernel to avoid intermediate-tensor memory blowup.
    14. """
    15. B, H, S, D = q.shape
    16. # Project queries and keys into the low-dimensional index space
    17. idx_q = q @ W_idx
    18. idx_k = k @ W_idx
    19. # Lightweight relevance scores
    20. scores = (idx_q @ idx_k.transpose(-2, -1)) / math.sqrt(idx_q.shape[-1])
    21. # Causal visibility matrix; for bidirectional encoding, replace with an all-True matrix
    22. vis = causal_visible(S, q.device)[None, None, :, :].expand(B, H, S, S)
    23. # Mask out future positions
    24. scores = scores.masked_fill(~vis, float("-inf"))
    25. # Keep the top-k highest-scoring keys per query; if S < k_top, take S
    26. kk = min(k_top, S)
    27. topk_idx = scores.topk(kk, dim=-1).indices # [B, H, S, kk]
    28. # Keep only visible positions among selected keys; early in the sequence, when too few
    29. # keys are visible, invalid entries will be masked in the following steps
    30. sel_vis = vis.gather(2, topk_idx)
    31. # gather: pull the corresponding key-values from the original k/v by index
    32. idx = topk_idx.unsqueeze(-1).expand(-1, -1, -1, D) # [B, H, S, kk, D]
    33. k_expanded = k.unsqueeze(2).expand(-1, -1, S, -1, -1) # [B, H, S, S, D]
    34. v_expanded = v.unsqueeze(2).expand(-1, -1, S, -1, -1)
    35. k_sel = k_expanded.gather(2, idx) # [B, H, S, kk, D]
    36. v_sel = v_expanded.gather(2, idx) # [B, H, S, kk, D]
    37. # Compute attention over the selected key-values
    38. q_exp = q.unsqueeze(2) # [B, H, S, 1, D]
    39. logits = (q_exp * k_sel).sum(-1) / math.sqrt(D) # [B, H, S, kk]
    40. # Mask out invisible keys again
    41. logits = logits.masked_fill(~sel_vis, float("-inf"))
    42. attn = logits.softmax(dim=-1)
    43. out = (attn.unsqueeze(-1) * v_sel).sum(dim=2) # [B, H, S, D]
    44. return out

    4.3 The Benefit: About 1/50 the Tokens to Match Dense Training

    In the controlled experiment for the long-context extension phase described in the report, a sparse-attention model trained with only about 20B tokens already matched a dense baseline trained with about 944B tokens. That is a token ratio of roughly (20 / 944 \approx 2.1\%), or about 1/50.

    The scope of this result must be clear: it is a controlled comparison from a specific phase and specific task in the report. The task focuses on long-context capability, and the comparison target is a dense-attention baseline. The specific context length, hardware configuration, evaluation metrics, and optimization details follow the original report, and this cannot simply be extrapolated into “sparse attention is always better at any scale.”

    The significance of this result is that sparse attention is not only an inference-side “cost-saving measure”; it can also improve data efficiency on the training side. The model no longer needs to waste compute on large numbers of low-relevance key-value interactions and instead concentrates capacity on the context segments that truly matter.

    4.4 The Cost: It Depends on Low-Level Kernel Support

    The benefits of sparse attention must convert into real hardware efficiency to truly materialize. Index scoring and Top-k selection introduce extra overhead; only when (k) is far smaller than the context length and a matching custom kernel is available will end-to-end latency and throughput improve.

    For this reason, such designs are usually tightly coupled with the underlying training and inference infrastructure. If you only do Top-k at the framework level without fused kernels, without optimizing memory-access patterns, and without handling variable-length sequences, the actual speedup can fall far short of the theoretical gain.

    5. Fully Asynchronous Agentic RL: Solving the Synchronous-Training Bottleneck

    5.1 The Post-Training Paradigm Shift: From “Answering Correctly” to “Getting It Done”

    LLM post-training is shifting from reinforcement learning from human feedback toward reinforcement learning centered on verifiable rewards. Tasks like math and code, where answers can be judged automatically, have become the main training grounds. For agent tasks, the objective also shifts from “is a single answer appropriate” to “can multi-step interaction ultimately achieve the goal,” including whether tool calls are accurate, whether intermediate decisions are reasonable, and whether the task succeeds end to end.

    5.2 The Triple Efficiency Bottleneck of Synchronous RL

    In the traditional approach, sampling (generation) and training strictly alternate, and the parameters are only updated after all data in a batch has been generated. This creates three bottlenecks:

    • Length variance: generation lengths differ greatly across questions, so each round must wait for the longest one to finish.
    • Interaction blocking: agent trajectories include multiple rounds of tool calls and interaction with the external environment, with hard-to-predict latency.
    • Resource mismatch: in the sampling phase, memory and compute demand concentrates on the generation side, while in the training phase it concentrates on the usage side, making the two kinds of resources hard to reuse efficiently.

    The contrast between synchronous and fully asynchronous can be summarized as:

    Dimension Synchronous RL Fully Asynchronous RL
    Data policy Usually requires generation from the current policy Allows use of stale policies from several steps back
    Pipeline Sampling and training phases run in series Sampling and training are pipelined in parallel
    Long tail Wait for the longest sample Length-aware scheduling, bucketing, and continuous consumption
    Main risk Resource waiting, low throughput Policy lag, requiring off-policy correction
    Engineering complexity Relatively simple Weight sync, sample replay, and fault tolerance are more demanding

    5.3 Fully Asynchronous: Decoupling, Pipelines, and Off-Policy Correction

    The fully asynchronous framework’s approach is:

    • Split sampling and training into two independent pipelines: the sampling side continuously produces trajectories, and the training side continuously consumes them.
    • Use length-aware scheduling to ease the long-tail problem.
    • Allow training data to come from stale policies several steps back.
    • For “policy lag,” typically apply off-policy correction via clipped objectives and truncated importance sampling.

    Taking GRPO-style methods as an example, the objective can be abstracted as:

    1. J(θ) = E[ 1/|o| · Σ_t ( min( r_t(θ)·A_t, clip(r_t(θ), 1-ε, 1+ε)·A_t )
    2. - β · KL(π_θ(·|q) || π_ref(·|q)) ) ]
    3. where r_t(θ) = π_θ(o_t | q, o_<t) / π_old(o_t | q, o_<t)
    4. A_t is the group-relative advantage: A_t = (R - mean(R_group)) / std(R_group)

    When the sampling policy lags the training policy, (r_t(θ)) deviates from 1:

    • The clipping mechanism limits the magnitude of a single update.
    • Truncated importance sampling further suppresses the impact of extreme ratios.
    • KL regularization keeps the policy from straying too far from the reference model.

    The illustrative code is:

    1. import torch
    2. # Illustrative: training loop for fully asynchronous agentic RL
    3. async def trainer(train_queue, step):
    4. batch = await train_queue.get() # Block until available, with no round alignment
    5. logp_new = policy.log_probs(batch) # Current policy log-probabilities
    6. ratios = (logp_new - batch.logp_old).exp()
    7. adv = batch.advantage # Group-relative advantage
    8. loss = -torch.min(
    9. ratios * adv,
    10. ratios.clamp(1 - eps, 1 + eps) * adv
    11. ).mean()
    12. loss = loss + beta * kl_penalty(policy, ref, batch)
    13. loss.backward()
    14. optimizer.step()
    15. publish_weights(policy) # Broadcast new weights; the sampling side pulls them asynchronously
    16. # Illustrative: an independent sampling coroutine that continuously produces trajectories
    17. async def rollout(engine, prompt_queue, train_queue):
    18. while True:
    19. prompt = await prompt_queue.get()
    20. traj = await agent.run(engine, prompt) # Multi-round tool calls and interaction
    21. reward = verifier.score(traj) # Verifiable reward
    22. logp_old = engine.log_probs(traj) # Record the policy's probabilities at sampling time
    23. await train_queue.put(Batch(traj, reward, logp_old))

    Its essence is to replace “phase-serial” execution with “pipelined parallel” execution, keeping the GPU busy between sampling and training. Note that “fully asynchronous” is a trade-off at the engineering-efficiency level. Whether the off-policy correction is sufficient directly determines training stability. If the policy lag is too large and the clipping, importance sampling, and KL constraints do not keep up, training may show reward collapse or policy drift.

    6. RL Infrastructure: Sampling, Weight Sync, and Fault Tolerance

    Supporting the paradigm above is a training infrastructure oriented toward reinforcement learning. RL training has a “generate-score-replay” loop on top of supervised pretraining, so its engineering complexity rises significantly.

    Stage Main Challenge Common Design Points
    Sampling execution Large trajectory-length variance, high interaction latency High-throughput inference engine, continuous batching, length-bucketed scheduling
    Weight sync Frequent parameter updates, large distribution overhead Fast broadcast to all sampling instances after each update; sampling side pulls asynchronously
    Samples and replay Quality control of off-policy data Sample storage and version tagging; sampling and filtering based on policy lag
    Training execution Large memory and compute demands Combinations of data/tensor/pipeline/expert parallelism; memory optimization
    Reward and verification Reward signals need to be reliable and verifiable Rule checks, unit tests, reproducible execution environments
    Fault tolerance and observability Stability of long-cycle tasks Checkpoint-based resume, task-isolated retries, fine-grained metrics and tracing

    Among these, weight sync and sample replay are the key steps that distinguish RL training from pretraining: the former determines how quickly the sampling side can start using the latest policy, and the latter determines whether stale-policy data is used appropriately. Without version tagging and lag control, asynchronous training can easily turn an “efficient pipeline” into an “unstable noise source.”

    7. Engineering Deployment: How to Choose, How to Deploy, and What to Watch

    7.1 Capability Status: Strong Overall, but Specific Dimensions Still Need Validation

    By the report’s figures, representative models in the open-source camp have entered a high-score range, and the gap with top closed-source models has narrowed clearly. Their overall capability is already enough to support many enterprise-grade applications. But in scenarios like ultra-long context, complex tool calls, multi-step reasoning, and long-tail domain knowledge, closed-source flagships usually still hold an advantage.

    The specific gap needs to be confirmed against your business evaluation, especially for:

    • Long-document understanding and cross-paragraph reasoning.
    • Multi-round tool calls and failure recovery.
    • Code generation, repair, and repository-level understanding.
    • Latency and throughput under high concurrency.
    • Capability stability after private-data fine-tuning.

    7.2 Three Paths: Deployment, Training, and Evaluation

    Deployment Side

    A MoE model’s memory demand is determined mainly by total params, not just by activated params.

    A simplified weight-memory estimate is:

    1. weight memory ≈ total param count × bytes per param

    With 744B total params:

    • BF16, 2 bytes per param: about (744 \times 2 = 1488) GB, roughly 1.5TB.
    • FP8, 1 byte per param: about (744 \times 1 = 744) GB, roughly 0.75TB.

    This means that even if the per-token activated params are only 40B, 744B total params still require multi-card, multi-machine hosting. The All-to-All communication introduced by expert routing becomes a scaling bottleneck, and the parallelization strategy should prioritize balanced expert distribution.

    In long-context scenarios, KV-cache usage grows with concurrency and context length. With MLA, the per-layer, per-token cache can be estimated as (c + r/2). A general formula is:

    1. per-request per-token KV cache bytes ≈ layers × (c + r/2) × bytes per element

    For example, assuming (D=128), (c=4D), (r=D/2), 80 layers, and BF16:

    1. c + r/2 = 4D + D/4 = 4.25D = 544
    2. per-layer per-token bytes = 544 × 2 = 1088 bytes
    3. all-layers per-token bytes = 1088 × 80 = 87040 bytes
    4. ≈ 85 KiB / token / request

    This is just an illustrative estimate; the actual numbers depend on the model’s real dimensions, number of layers, and precision. Engineering-wise, you can also combine techniques such as low-precision caching, chunked prefill, KV quantization, and request-level rate limiting.

    Training Side

    When switching from a dense model to MLA, MTP, or sparse-attention structures, the optimizer, loss functions, and low-level kernels need to be adapted in step. Key tuning points include:

    • How a Muon-style orthogonalized optimizer matches the attention structure.
    • Whether a per-head/per-subspace split strategy like Muon Split is used.
    • The auxiliary-loss weight for multi-token prediction.
    • The index mechanism, Top-k strategy, and kernel implementation for sparse attention.
    • Whether the training-phase and inference-phase structures are consistent.
    • Precision alignment and long-context degradation.

    Evaluation Side

    It is best to first build a verifiable task set that fits your own business, covering long-context understanding, multi-round tool calls, code generation and repair, math reasoning, and domain Q&A. For sampling-based tasks, set a fixed random seed to ensure evaluation reproducibility. For asynchronous RL training, also monitor:

    • The policy-lag distribution.
    • The importance-sampling ratio distribution.
    • Changes in KL divergence.
    • Reward variance.
    • The sampling-length distribution.
    • Whether training throughput matches sampling throughput.

    7.3 Risk Checklist: Questions to Answer Before Deciding

    • License compliance: confirm the model license’s constraints on commercial use, distribution, derivative models, and output content.
    • Deployment cost: total-param scale sets the memory floor; small-scale deployment needs to evaluate quantization, distributed inference, and expert-parallelism options.
    • Capability boundary: run controlled evaluations in long-tail, high-value business scenarios to confirm whether the gap with closed-source flagships is acceptable.
    • Architecture compatibility: when transferring sparse attention, MLA, or MTP to your own model, verify kernel support, precision alignment, long-context degradation, and training stability in step.
    • Asynchronous RL stability: if adopting fully asynchronous agentic RL, you must evaluate policy lag, off-policy correction, reward verification, and fault-tolerance mechanisms.

    Conclusion

    Reviewing this technical report block by block, three main threads emerge:

    • Sparsification: MoE sparsifies parameters, MLA compresses the KV cache, and DSA sparsifies attention interactions. At their core, they all “concentrate limited compute and memory on the most critical parts.”
    • Asynchronization: fully asynchronous RL decouples sampling and training into a pipeline, using system efficiency to compensate for the limits of single-card compute and synchronous rounds.
    • Structure–optimization co-design: the MLA + Muon Split combination shows that the model structure, optimizer, and low-level kernels must be designed as a whole.

    For developers, the most valuable takeaway is not a specific number, but this methodology of “trading system efficiency for model capability.” It is both the underlying driver of open-source models continuing to catch up with flagships and an engineering approach worth borrowing when building cost-effective AI applications.