Deep Dive Into an Open-Source LLM Technical Report: MoE Scaling, MLA, Muon Split, MTP, Sparse Attention, and Asynchronous Agent RL

    Over the past six months, the gap between open-source and closed-source large models has kept narrowing: leading open-source models now score close to, and in some scenarios level with, the top proprietary models in third-party composite intelligence benchmarks. This article walks through a recently published ~40-page technical report from a Chinese team that open-sourced its new-generation LLM (referred to below as "the new open-source model"), focusing on the engineering tradeoffs it documents. It covers the 744B-total / 40B-active MoE design, why MLA broke the assumptions of the Muon optimizer and how Muon Split fixed it, how multi-token prediction serves both training and inference, how learnable sparse attention matched dense attention with roughly 1/47 of the token budget, and how a fully asynchronous agent RL framework addresses the "generation is slower than training" bottleneck in long-horizon agent rollouts. A reusable engineering checklist is included at the end.

    Over the past six months, the gap between open-source models and closed-source models has continued to narrow. The top open-source models now score close to — and in some scenarios level with — the strongest proprietary models in third-party composite intelligence evaluations. The technical drivers behind this trend are worth taking seriously.

    A Chinese team recently published the complete technical report for its new-generation large language model (referred to below as “the new open-source model”). The report runs to roughly 40 pages and quickly became a page-by-page reading exercise in developer communities.

    The value of this report goes beyond “another model release.” It lays out several key tradeoffs across the training and inference pipeline:

    • How the base architecture evolved to 744B total parameters with 40B activated;
    • How attention and optimizer were co-designed, and the detours the team took along the way;
    • How multi-token prediction serves both training efficiency and inference acceleration;
    • How sparse attention approaches dense-attention quality on roughly 1/47 of the token budget;
    • How a fully asynchronous RL framework tackles the fundamental “generation is slower than training” bottleneck in agent scenarios with hour-long trajectories.

    Using this report as the primary source and adding general technical background and engineering practice, this article walks through the design logic, root causes, and practical reference value block by block, and closes with a reusable technical checklist.

    1. Fundamentals: 744B Total Parameters, 40B Activated — MoE Remains the Best Value Option

    The overall shape of the new open-source model can be summarized in three numbers:

    • Total parameters: 744B
    • Activated parameters: 40B
    • Structure: 80 layers, 256 experts (MoE, i.e. Mixture of Experts)

    Compared with the previous generation (355B total parameters, 32B activated), total parameters increased by about 1.1x and activated parameters by about 25%. Training data grew from 23T tokens to 28.5T tokens, split into 27T for pre-training and 1.5T for mid-training.

    Behind these numbers lies the core logic of the MoE architecture: decoupling capacity from computation. In a dense model, every parameter participates in computing every generated token. With MoE, a gating network routes each token to only a small subset of experts, so the compute cost scales with the activated parameter count rather than the total parameter count. The new model has 2.1x the total parameters of the previous generation while adding only 8B activated parameters, which means the capacity increase is absorbed almost entirely by memory and communication costs, not by inference-time floating-point operations. For an open-source model that needs to be both “stronger” and “affordable,” this tradeoff is close to inevitable.

    MoE is not free of engineering costs: the All-to-All communication overhead introduced by expert parallelism, the “bucket effect” caused by load imbalance, and the gap between memory footprint and effective compute are all unavoidable in large-scale MoE training and deployment. When teams reproduce or fine-tune similar architectures, they often underestimate this part of the cost. So judging whether an MoE model is “strong” cannot rest on total parameter count alone. It also requires looking at:

    • Its quality/cost ratio under a fixed inference budget;
    • Whether it is usable in 128K+ long-context scenarios (usually a better signal of real engineering capability than short-text leaderboards);
    • The stability of structured output and tool calling.

    2. Leaderboards and Arena Rankings: 50 Points, First Among Open Source — But Read the Score Carefully

    On a composite intelligence evaluation such as the Artificial Analysis Intelligence Index v4.0, the new open-source model scores 50, placing first among open-source models and behind only the strongest proprietary models. In the LMArena (formerly Chatbot Arena) text arena and code arena, it also ranks first among open-source models, performing at the same tier as the leading proprietary models.

    Still, these scores deserve a clear-eyed reading. A composite index is typically a weighted aggregation across multiple benchmarks, and leaderboard rank reflects competitiveness on the average, while real business needs usually care about specific capability dimensions — long-document understanding, repository-level code completion, multi-turn tool-call stability, or performance in a particular language. A more sensible approach: treat the composite index as a threshold signal for “entering the first tier,” then validate actual performance in your target scenario with your own evaluation set.

    3. Architecture Change No. 1: MLA + Muon Split

    Compared with the previous generation, the new open-source model has three clear architectural changes:

    • MLA (Multi-head Latent Attention) + Muon Split;
    • Multi-token prediction (MTP);
    • A learnable sparse attention mechanism (DSA).

    3.1 Why MLA Is Worth Using: Compressing the KV Cache Into a Low-Rank Space

    The core idea of MLA is to project each token’s Key/Value into a low-dimensional “latent” space for caching, then up-project on demand during inference. Compared with traditional multi-head attention, which caches a full copy of K and V for every head, this substantially reduces the memory footprint of the KV cache while retaining strong representational power. This is also why the “MLA route” has been widely studied since DeepSeek-V3.

    Early in training, however, the team hit a counterintuitive result: pairing a Muon-family optimizer with MLA could not match the simpler GQA-8 scheme (grouped-query attention with 8 KV heads). The phenomenon is worth unpacking.

    3.2 Why Muon Misfits: Projection-Matrix Coupling and Gradient-Scale Differences

    Muon is a family of orthogonalization-based optimizers: it orthogonalizes the gradient momentum matrix (typically approximated via Newton-Schulz iterations) so that the singular values of the update are more uniform, improving the conditioning of large-model pre-training. Its effectiveness on large-scale dense models has been repeatedly validated.

    The problem lies in matrix shape. In GQA, the Query/Key/Value projection matrices usually have a regular “n_heads x head_dim” structure, and orthogonalization is applied to one whole matrix. In MLA, to compress the KV cache, the team used compression/decompression projections shared across heads, which changed both the shape and the semantics of the projection matrices. Whole-matrix orthogonalization on this kind of structure forcibly flattens the updates of different heads and different semantic subspaces, and its effect is not comparable to GQA. This is fundamentally an optimizer-architecture adaptation problem: Muon’s assumptions rest on regular projection matrices, and MLA breaks that premise.

    3.3 Muon Split: Orthogonalizing Per Head So the Optimizer Fits Again

    The team’s solution is straightforward: change “one orthogonalization over the whole projection matrix” into “separate orthogonalization per attention head,” i.e. Muon Split. Illustrative code:

    1. # Original approach: orthogonalize the entire projection matrix at once
    2. def muon_step(W, lr):
    3. G = momentum_buffer(W) # gradient momentum
    4. O = newton_schulz(G) # orthogonalize the whole matrix
    5. return W - lr * O
    6. # Muon Split: orthogonalize each attention head separately
    7. def muon_split_step(W, n_heads, lr):
    8. G = momentum_buffer(W)
    9. G_heads = G.view(n_heads, -1, G.shape[-1])
    10. O_heads = torch.stack([newton_schulz(g) for g in G_heads])
    11. O = O_heads.reshape_as(G)
    12. return W - lr * O

    The point of this change is that it aligns the optimizer’s “orthogonalization granularity” with the “multi-head semantic structure” of attention. Each head gets an independent update direction, heads no longer squeeze each other, and MLA’s compressed structure regains support at the optimizer level. The lesson for engineering teams is direct: when a new architecture does not play well with an established optimizer, first check whether the optimizer’s assumed granularity matches the structural granularity, rather than rushing to overturn the architecture itself.

    4. Architecture Change No. 2: Multi-Token Prediction (MTP)

    The second change is multi-token prediction. Conventional autoregressive training predicts only the next token at each position, whereas MTP introduces multiple prediction heads on the output side to predict several future tokens at once.

    Its benefits come in two layers:

    • Training side: a denser supervision signal, effectively adding a “planning-oriented” auxiliary loss. This helps mitigate representation collapse and improve sample efficiency, with particularly visible gains on high proportions of code and structured data;
    • Inference side: the MTP heads can act as a draft model for speculative decoding, generating several candidate tokens in a single forward pass that the main model then verifies in parallel, pushing up generation throughput.

    The costs need attention too: MTP consumes extra parameters and memory, the draft hit rate depends on task entropy (code and structured output typically hit more often than open-ended writing), and it requires the inference framework to explicitly support a “multi-token head + verification” pipeline — a plain batched inference implementation cannot benefit from it directly.

    5. Architecture Change No. 3: Learnable Sparse Attention (DSA)

    The third change is also the most striking point in the report: the team presents a comparison showing that with sparse attention, only about 20B training tokens were enough to match the dense-attention baseline trained on roughly 940B tokens — a token-efficiency difference of nearly 47x.

    The motivation for sparse attention is clear: the cost of attention computation and the KV cache grows quadratically with sequence length, which is the main bottleneck in long-context and long-CoT scenarios. The idea behind DSA is not simply “truncate the window,” but to introduce a learnable lightweight scoring module that dynamically selects the Top-k most relevant Key/Value positions for each query and performs exact attention only over those positions. Illustrative code:

    1. # Lightweight scoring -> dynamic Top-k -> exact attention only on selected K/V
    2. score = indexer(q, k_cache) # score candidate keys for each query
    3. idx = score.topk(k, dim=-1).indices # select k tokens
    4. k_sel, v_sel = gather(k_cache, v_cache, idx)
    5. out = softmax(q @ k_sel.T / sqrt(d)) @ v_sel

    It is complementary to MLA: MLA compresses the dimension cached per token, while DSA cuts down the number of tokens that participate in computation. Stacked together, both the memory and compute curves of long context drop noticeably.

    Three pitfalls to watch for in engineering:

    1. High kernel complexity: dynamic Top-k means irregular memory access patterns, so dedicated kernels (fusing gather with attention) are required to saturate compute; otherwise the theoretical speedup shrinks substantially;
    2. Training stability: the scoring module itself needs gradient signal and tends early on to degrade into an approximation of local-window attention, which requires training strategies to guide it (for example, transitioning gradually from dense attention, or retaining a small number of global tokens);
    3. Evaluation methodology: conclusions like “20B tokens matches 940B tokens” are drawn under a specific baseline and specific data mixture, so reproduce them only after fixing the comparison conditions first.

    6. A Fully Asynchronous Agent RL Training Framework

    One of the most discussed engineering points in the report is the “fully asynchronous Agent RL training framework.” Traditional RLHF/RFT pipelines are strictly synchronous: generate a batch of rollouts -> compute rewards -> update the policy -> generate the next batch. The problem is that in agent scenarios a rollout often includes multiple rounds of tool calls, long CoT, and code execution; a single trajectory can take anywhere from tens of seconds to tens of minutes, with an extremely heavy long tail in the length distribution. Under a synchronous framework, the throughput of the entire cluster is held back by the slowest trajectory, and GPU utilization runs chronically low.

    The fully asynchronous approach completely decouples “generation” from “training”:

    • The inference side continuously produces trajectories, tagged with version numbers;
    • The training side consumes whatever samples are available at its own pace, no longer waiting for a whole round to complete;
    • Weights are published to the inference side asynchronously, producing near on-policy training that is “slightly stale but extremely high throughput.”
    1. # Synchronous: training stalls until the whole batch finishes
    2. while not done:
    3. batch = generate(policy, prompts) # long-tail trajectories slow everything down
    4. update(policy, batch)
    5. # Fully asynchronous: generation and training advance independently
    6. futures = []
    7. for step in range(T):
    8. futures.append(pool.submit(generate, snapshot(), prompts))
    9. batch = collect_finished(futures, min_size=N) # do not wait for the long tail
    10. batch = importance_ratio_fix(batch, current_policy, snapshot()) # correct off-policy bias
    11. update(policy, batch)
    12. publish_weights(step) # publish new weights asynchronously

    The core cost of asynchrony is off-policy bias: the consumed samples may have been generated by an older version of the policy. Practical mitigations include correcting with importance-sampling ratios, capping the maximum “staleness” of samples, introducing partial rollouts (cutting a long trajectory and resuming it later rather than discarding it), and debiasing on the reward side. The report notes that the team built its own RL infrastructure (a slime-style approach), and its focus areas are essentially these: connecting a high-throughput inference engine with the training framework, trajectory caching and resume-from-checkpoint, consistency checks for weight synchronization, and fault tolerance plus elastic scheduling at large cluster scale. This is the part of the report with the most reference value for teams that want to build their own RL infrastructure.

    7. A Practical Checklist for Developers

    Abstracted into actionable lessons, the report boils down to roughly these points:

    • Choose MoE models on quality/cost ratio, not total parameter count; validate with your own long-context and tool-calling evaluation sets first.
    • When a new architecture and an established optimizer do not match, align granularity first: small changes like per-head orthogonalization (Muon Split) are often a better deal than switching optimizers.
    • MTP pays off in both training and inference, but only if the inference framework supports the draft-and-verify pipeline; otherwise the benefit never materializes.
    • Sparse attention needs kernel engineering to go with it: the irregular memory access of dynamic Top-k means it can only land through dedicated operators.
    • The bottleneck in Agent RL is on the generation side: asynchronous execution + staleness control + partial rollouts are the three essentials for raising GPU utilization.
    • Prioritize reproducibility: any “X tokens matches Y tokens” conclusion requires fixing the baseline, data mixture, and evaluation methodology first.

    8. Conclusion

    What deserves the most attention in this 40-page report is not the “first among open source” ranking, but the clear engineering path it demonstrates: use MoE to decouple capacity from compute, use MLA to shrink the KV cache, use Muon Split to repair the optimizer-architecture mismatch, use MTP to raise both supervision density and inference throughput, use DSA to bring down the quadratic complexity of long context, and finally use a fully asynchronous agent RL framework to maximize compute utilization. None of these pieces is an isolated trick; each is a tradeoff that interlocks with the others. For developers, reading this as a “reusable architecture decision record” yields far more than reading it as a leaderboard result.