Sparse Attention and Asynchronous Reinforcement Learning: A Layer-by-Layer Breakdown of a Large-Scale MoE Technical Report

    This article breaks down a 40-page technical report on a large-scale Mixture-of-Experts (MoE) base model with 744B total parameters and 40B activated parameters per inference. It explains the engineering choices behind the base configuration, multi-head latent attention, matrix-orthogonalization optimizer compatibility, multi-token prediction, dynamic sparse attention, fully asynchronous agent reinforcement learning, and custom training infrastructure. It also clarifies the trade-offs between memory, latency, compute efficiency, data efficiency, and rollout staleness, offering practical guidance for teams building or deploying large MoE models.

    1. Background: the gap between open and closed models is narrowing fast

    One of the most visible trends in the industry is that the strongest closed-source models still hold the position of being the “smartest,” but the gap between them and the best open-weight models has narrowed from a generational lead to a perceptible score difference. Independent leaderboards and human preference arenas show a similar conclusion: open models have entered the same discussion range as frontier systems in composite intelligence, conversational ability, and code generation.

    This convergence is not the result of a single technology. It reflects three maturing trends:

    1. Sparsification through MoE decouples model capacity from inference cost.
    2. Improved data efficiency allows stronger models to be trained under a limited token budget.
    3. Post-training has shifted from supervised fine-tuning to large-scale reinforcement learning, producing larger gains on long-horizon tasks than parameter scaling alone.

    A recently published 40-page technical report documents the engineering details behind these trends. This article breaks it down in the following order: base configuration, three architectural changes, asynchronous reinforcement learning, and infrastructure.

    2. Base configuration: the trade-off logic of 744B total parameters and 40B activated parameters

    The base model described in the report has:

    • 744B total parameters
    • 40B activated parameters per inference
    • 256 routed experts
    • 80 layers

    Compared with the previous generation, the total parameter count roughly doubled from the 355B range to 744B, activated parameters increased from 32B to 40B, and the pre-training corpus expanded from 23T tokens to 28.5T tokens, including 27T tokens for pre-training and 1.5T tokens for mid-training.

    The most important number here is not the total parameter count, but the activation ratio:

    1. 744B total parameters → 40B activated sparsity ≈ 18x
    2. 355B previous gen → 32B activated sparsity ≈ 11x

    The sparsity ratio increased from about 11x to about 18x. This means that model capacity — the knowledge and patterns the model can store — expands significantly, while the per-token multiply-accumulate overhead increases by only about 25% (from 32B to 40B). This is a classic case of “trading memory capacity for compute efficiency”: all MoE weights must either reside in memory or be streamed efficiently at high throughput, but each token only touches a small subset of them.

    This creates three direct engineering constraints:

    1. Memory budget is determined by total parameters; latency budget is determined by activated parameters.
      Deploying a 744B-class model requires memory planning based on the full 744B parameters, but tokens/s is closer to that of a 40B dense model. These two aspects must be planned with different metrics.

    2. Expert parallelism is mandatory, not optional.
      With 256 experts, expert parallelism across multiple devices is almost inevitable. This introduces all-to-all communication, which can easily become a ceiling for training throughput.

    3. Mid-training is an independent stage.
      The split between 27T pre-training tokens and 1.5T mid-training tokens shows that the team treats “adjusting high-value data mixtures such as long context, code, and reasoning” as a separate budget rather than mixing it into pre-training. This is now a common practice among leading models.

    On evaluation, the report states that the model achieved a composite intelligence score of 50, ranking first among open models. It also ranked first among open models in both the text arena and code arena, placing it in the same overall range as the most advanced closed models.

    3. Architectural change one: latent attention and the optimizer incompatibility problem

    3.1 What multi-head latent attention solves

    Standard multi-head attention requires caching complete K and V tensors for each layer. Memory usage grows linearly with:

    1. layers × heads × head_dim × sequence_length

    During long-context inference, the KV cache can easily exceed the model weights themselves, becoming a major consumer of both memory and bandwidth.

    Multi-head latent attention (MLA) addresses this with low-rank compression of the cache. Instead of directly caching K and V, it caches a low-dimensional latent vector and projects it back when needed.

    1. # Illustration: MLA write and read paths
    2. c_kv = x @ W_dkv # [B, T, d_c], d_c << n_head * head_dim
    3. k_nope = c_kv @ W_uk # reconstruct the content part of K
    4. v = c_kv @ W_uv # reconstruct V
    5. q = (x @ W_dq) @ W_uq # Q also goes through low-rank projection
    6. scores = (q @ k_nope.T) * scale + rope_bias # positional encoding applies only to a few dimensions

    The key point is that the latent dimension d_c is much smaller than n_head × head_dim. As a result, cache size drops by an order of magnitude, throughput for long-context processing improves significantly, and the memory curve for long context becomes smoother. MLA is closely related to the DeepSeek-V3 line and is already a mature, widely validated approach in the open-source community.

    3.2 The conflict: matrix-orthogonalization optimizers and reduced MLA performance

    The team encountered a counterintuitive result during training: when MLA was paired with a matrix-orthogonalization optimizer, such as Muon-style optimizers, it underperformed a simpler alternative — standard grouped-query attention with only 8 GQA groups.

    The root cause is worth examining because it has broader value for anyone building training systems.

    The core operation of a matrix-orthogonalization optimizer is to approximately project the momentum update of each weight matrix onto an orthogonal matrix, usually using Newton–Schulz iterations for a low-cost approximation. The goal is to balance update magnitudes across different singular directions. This works best when the optimizer is operating on a complete, semantically aligned two-dimensional matrix — for example, the full projection matrix of an attention head, whose row space and column space correspond to meaningful feature subspaces.

    In MLA, however, matrices such as W_dkv are artificially compressed bottleneck matrices. Applying orthogonalization to the whole matrix is equivalent to forcing the compressed latent space and the output space to update in a balanced way, which damages the semantic structure that the low-rank representation itself carries. Orthogonalization places different subspaces under the same scale, preventing the model from learning a clean compressed representation.

    3.3 The solution: per-head orthogonalization with Muon Split

    The team’s fix can be summarized in one sentence: replace whole-matrix orthogonalization with per-head orthogonalization. The original report calls this Muon Split. In essence, it is a semantic alignment fix — it makes the granularity of orthogonalization match the semantic granularity of the model.

    1. def muon_split(W, momentum, lr, heads):
    2. # W: [heads * head_dim, in_features]
    3. buf = momentum.accumulate(W) # accumulate momentum
    4. W_view = buf.view(heads, -1, W.shape[1]) # split by head
    5. for h in range(heads):
    6. O = newton_schulz_orthogonalize(W_view[h]) # orthogonalize each head separately
    7. W_view[h] = O
    8. return W.data.add_(W_view.reshape_as(W), alpha=-lr)

    An important detail is that per-head orthogonalization is implemented as a loop. Across 80 layers and multiple head-level matrices per layer, this introduces non-negligible serial overhead. Such an approach must therefore be fused at the kernel level — for example, by batching orthogonalization iterations for many small matrices into a single operation — otherwise the optimizer itself will consume training throughput. This is one of the practical reasons why teams often need custom RL and training infrastructure.

    A more general lesson is this: when the mathematical assumptions of an optimizer conflict with the structural priors of a network, prioritize changing the optimizer’s granularity rather than retreating to a simpler architecture. Here, giving up MLA and returning to 8-group GQA might work in the short term, but it would permanently lose the memory advantage for long-context inference.

    4. Architectural change two: multi-token prediction (MTP)

    Multi-token prediction is not a new concept, but its value in modern training stacks has changed.

    During training, MTP attaches several lightweight prediction heads on top of the main model, allowing the model to predict the next 1, 2, …, n tokens at each position.

    1. # Illustration: MTP training loss
    2. h = backbone(x)
    3. loss_main = CE(head_main(h), tokens[1:])
    4. h_mtp = mtp_head(h.detach() if detach else h) # lightweight module
    5. loss_mtp = sum(CE(mtp_heads[i](h_mtp), tokens[2+i]) for i in range(n))
    6. loss = loss_main + lambda_mtp * loss_mtp / n

    MTP brings two types of benefits, and they matter differently.

    Benefit one during training: denser supervision.
    Standard next-token prediction provides only one supervision point per position. Multi-token prediction provides n points. For tasks that require multi-step reasoning, code structure, or long-range dependencies, this effectively increases the gradient information density of the same data. It is especially helpful for planning-oriented capability — deciding where the next few steps should go — which is naturally weak under a pure next-token objective.

    Benefit two during inference: reusable draft heads.
    A trained MTP head can be used directly for speculative decoding: the lightweight draft head proposes several candidate tokens, and the main model verifies them in parallel in a single pass. Accepted tokens are adopted directly. This is a pure engineering gain: it does not change model quality and only reduces latency. This is also why mainstream solutions almost always train MTP heads, even when they do not fully rely on them for inference acceleration.

    There are two implementation pitfalls:

    • Do not detach too early. Otherwise, the MTP head degenerates into an independent small model that cannot learn from the backbone representation, and speculative acceptance rates will be low.
    • The MTP loss weight λ should be annealed. Keeping it too high in later training can crowd out representation capacity for the main task.

    5. Architectural change three: dynamic sparse attention and “20B tokens matching 943.7B tokens”

    This is the point in the report most worth discussing separately.

    5.1 Mechanism: changing attention from full to selective

    The core of dynamic sparse attention (DSA) is a lightweight learnable indexer, often called a lightning indexer. First, a low-cost scoring function estimates the relevance between each query and historical tokens. Only the top-k tokens are selected for exact attention computation; the rest are skipped.

    1. # Illustration: two-stage sparse attention
    2. # Stage 1: lightweight scoring (low-dimensional, few heads, FP8 allowed)
    3. idx_score = indexer(q_lowdim, k_cache_lowdim) # [B, H, T, S]
    4. topk_idx = idx_score.topk(k, dim=-1) # keep only k tokens, e.g., 2048
    5. # Stage 2: precise attention only on selected tokens
    6. attn_out = precise_attn(q, k_cache, v_cache, mask=gather(topk_idx, ...))

    The complexity comparison is intuitive:

    1. Dense attention: O(L^2 · d)
    2. Sparse attention: O(L^2 · d_idx) + O(L · k · d)
    3. where d_idx << d and k << L

    When sequence length reaches 128K or even 1M tokens, the L^2 term dominates. Sparse attention is not just “a little faster” — it changes the situation from not trainable to trainable. Because the indexer is learnable, the model itself decides where to look. This is much more flexible than static sparse patterns based on fixed windows or fixed strides.

    5.2 Causal explanation: why this improves data efficiency

    The report states that a model using DSA reached the performance of 943.7B tokens using only 20B tokens — roughly a 47x improvement in data efficiency. The number looks dramatic, but the logic chain is clear.

    • Sparse attention does not make each token “smarter.” Instead, it allows the model to see many more tokens under the same compute budget. With dense attention on long sequences, almost all compute is spent on the attention matrix, leaving less budget for the corpus itself.
    • Therefore, the correct interpretation of “20B vs. 943.7B” is: under a fixed compute budget, sparse attention improves effective throughput by roughly an order of magnitude, allowing the model to train on more data and converge to a comparable level.
    • This is also a typical case where an architectural choice is a compute-budget choice. Sparsification is not a free quality improvement; it moves compute from the quadratic term to the linear term, thereby freeing up budget.

    Conversely, this also provides a standard for evaluating any claim of “X times data efficiency”: first check whether the comparison was made under equal compute, and second check whether long-context capability has degraded.

    6. The reinforcement learning stage: why “fully asynchronous” is essential

    After the base model is trained, the capability ceiling is largely determined by post-training. The report describes a fully asynchronous agent reinforcement learning training framework. The phrase “fully asynchronous” is almost a prescription for the most painful point in existing RL frameworks.

    6.1 The fatal bottleneck of synchronous RL: long-tail trajectories

    In synchronous RLHF/RL, one step follows this standard flow:

    1. sample a batch of prompts → complete all rollouts → compute reward → update policy

    The problem lies in “complete all rollouts.”

    In agent settings, a single trajectory is not one sentence. It is a multi-step interaction involving dozens of tool calls: reading files, running commands, writing code, running tests, inspecting errors, and making fixes. The duration distribution of such trajectories has a very heavy tail:

    1. average 30s, P50 20s, but P99 can reach 8 minutes (one compilation plus one full test run)

    Synchronous frameworks must wait for the slowest trajectory. Suppose a batch contains 512 trajectories. If one of 512 workers takes 8 minutes, the other 511 workers simply wait for 8 minutes. Compute utilization is locked by the long tail, which is catastrophic waste at thousand-GPU scale.

    6.2 Asynchronous architecture: removing waiting entirely

    The fully asynchronous approach is to decouple rollout and training, allowing both sides to run as fast as possible:

    1. # Illustration: asynchronous RL main loop (pseudocode)
    2. rollout_producer = start_rollout_workers() # continuously produce trajectories, never wait
    3. train_consumer = start_trainer()
    4. buffer = ReplayBuffer(max_size=N)
    5. # Producer side: whichever finishes first submits first; no waiting for slow workers
    6. for worker in rollout_workers:
    7. async def produce(w):
    8. traj = await w.generate_task() # multi-step tool-call trajectory
    9. traj.reward = await reward_fn(traj)
    10. buffer.push(traj)
    11. spawn(produce(worker))
    12. # Consumer side: train once enough samples are collected
    13. while not converged:
    14. batch = buffer.sample(batch_size)
    15. loss = compute_loss(batch, policy) # with off-policy correction
    16. policy.step(loss)
    17. if step % K == 0:
    18. broadcast_weights(rollout_workers) # asynchronous weight sync

    Four supporting mechanisms are required:

    1. Partial rollout.
      A long trajectory should be allowed to resume after weight updates rather than discarding dozens of completed steps. This is key to flattening the cost of long-tail trajectories.

    2. Asynchronous weight synchronization.
      New weights are broadcast to the rollout side, allowing different workers to operate under different policy versions.

    3. Off-policy correction.
      Asynchrony necessarily introduces data staleness: the policy used for sampling is not the same as the policy used for training. Importance sampling ratios must be applied, with an upper bound on staleness.

    4. Explicit staleness control.
      A common practice is to limit how many policy versions the rollout side can lead the training side, or to use near-real-time weight updates to shorten the lag window.

    1. # Illustration: importance-weighted loss with staleness clipping
    2. rho = pi_new(a|s) / pi_old(a|s) # off-policy correction
    3. rho = rho.clamp(max=RHO_CLIP) # clip to prevent explosion
    4. loss = -min(rho * A, rho.clamp(max=1+eps) * A) # PPO-style clipping

    The engineering trade-off must be stated clearly: asynchronous training brings throughput, but at the cost of off-policy bias. This is not “free acceleration.” If the correction is poorly implemented, the policy can collapse. When the report emphasizes “fully asynchronous,” the implicit premise is that they have made this correction and fault-tolerance layer solid.

    6.3 Three differences between agent RL and classic RLHF

    • Reward is sparser and arrives later.
      For a task with dozens of steps, only the final step — whether the test passes — provides a signal. Advantage estimation must propagate long-horizon credit backward.

    • Environment-side cost is extremely high.
      Rollout is not just token generation. It also requires sandboxes, containers, command execution, and tool calls. The resource model on the rollout side is closer to “serverless compute plus container orchestration” than pure GPU inference.

    • Failure rate is not noise; it is signal.
      Compilation failures, timeouts, and abnormal tool returns are precisely where learning signals are densest. In engineering terms, rollout failures should not be swallowed as exceptions. They should be returned as valid trajectories, even if their reward is 0.

    7. Training infrastructure: custom frameworks solve problems that general frameworks have not solved

    The report mentions a self-developed RL training framework, comparable in spirit to approaches such as slime. Why do large and leading teams almost always build their own instead of directly using mature open-source frameworks?

    Because all of the architectural changes above will leak when applied to existing general-purpose frameworks:

    Architectural change Framework requirement Current state of general frameworks
    744B MoE with expert parallelism Expert parallelism, all-to-all optimization, memory streaming EP support varies widely
    Per-head orthogonalization optimizer Custom optimizer kernels supporting batched small-matrix iterations Usually only standard per-parameter optimizers
    Dynamic sparse attention Custom attention kernels with indexer forward/backward Standard attention interfaces cannot express it
    Asynchronous agent RL Decoupled rollout/train, partial trajectories, staleness control Mostly synchronous designs

    Any one of these four items requires substantial modification in a general framework. Moreover, after modification, upgrading upstream creates additional merge costs. As a result, “building in-house” is often not a technology preference but the inevitable outcome when changes exceed the abstraction boundary of the framework.

    If a team does not yet have the capability to build its own framework, a practical compromise is:

    • Start with a general framework and establish a baseline with dense attention and standard optimizers.
    • Make sparse attention and custom optimizers pluggable modules.
    • For the RL stage, prioritize asynchronous rollout, even if training remains synchronous. The largest gains often come from here.

    8. A practical checklist for engineering teams

    If you are building training or deployment systems at a similar scale, the following checklist is actionable.

    Before training

    • Clearly distinguish that memory budget follows total parameters, while latency budget follows activated parameters; plan expert parallelism degree and batch size separately.
    • Evaluate whether sparse attention is necessary: once the target context length exceeds 64K, the quadratic term becomes the dominant cost.
    • Confirm whether the optimizer is compatible with the attention structure. If using low-rank compressed attention, prioritize checking whether per-head orthogonalization is required.
    • Schedule mid-training separately, including adjustments to long-context, code, and reasoning data mixtures. Do not fold it into pre-training.

    Reinforcement learning stage

    • First establish a synchronous baseline and quantify the long tail, such as the P99/P50 latency ratio. This number determines the upper bound of asynchronous gains.
    • After going asynchronous, add off-policy correction and staleness limits; otherwise the policy may degrade silently.
    • Implement partial rollout to avoid discarding entire long trajectories.
    • Include rollout failures in training data rather than filtering them out as exceptions.

    Deployment and operations

    • Plan memory based on total parameters, including expert weights; plan QPS based on activated parameters.
    • MTP heads can be used for speculative decoding to reduce latency, but acceptance rates must be measured in practice.
    • Build token-level cost dashboards: in long-context scenarios, KV cache cost often exceeds weight cost.

    When deploying this kind of training and inference workload in the cloud, several general capabilities are typically required: object storage for corpora and checkpoints, container platforms for elastic training and rollout sandboxes, and logging plus monitoring for long-tail and throughput attribution. Baidu AI Cloud offers corresponding products in these general categories; refer to the official documentation for regional availability, specifications, and quotas.

    9. Three reminders

    1. “47x data efficiency” does not mean “47x cheaper.”
      It is a convergence comparison under equal compute. The actual total cost must also include engineering investment and tuning cost.

    2. Asynchrony is not a universal cure.
      It frees compute utilization from long-tail effects, but it introduces off-policy bias. If the correction is inadequate, throughput may rise while effectiveness falls, making the system more expensive overall.

    3. Architectural choices are budget choices.
      Sparse attention, MLA, and MoE sparsity are all answers to the same question: where should my compute be spent in the quadratic terms? Once you see this, the standard for reading any model technical report changes from “do the numbers look good?” to “what are the constraints, and what is the cost?”

    In one sentence, the value of this report is that it presents a complete paradigm: using architecture to exchange for data efficiency, and using asynchrony to exchange for compute utilization. The combination includes a 744B-total / 40B-activated sparse base, latent attention with a per-head orthogonalization optimizer, multi-token prediction, dynamic sparse attention, and fully asynchronous agent RL. The core logic remains consistent throughout: do not brute-force compute in any single stage; instead, find and eliminate waste in every stage.