Open-Source LLM Technical Report Deep Dive: Trainable Sparse Attention and Fully Asynchronous Agent RL

    This article offers a close reading of a recent 40-page open-source large language model (LLM) technical report, focusing on the engineering trade-offs behind its most-discussed technical choices. It covers the model's fundamentals (744B total parameters with 40B activated parameters in a mixture-of-experts design), three key architecture changes — MLA paired with head-level optimizer splitting (Muon Split), multi-token prediction (MTP), and trainable sparse attention (DSA) — and its fully asynchronous agent reinforcement learning (RL) framework built on in-house RL infrastructure. Beyond the architecture itself, the article explains the reasoning behind each decision, including the decoupling of memory and compute in MoE, KV-cache compression, the data efficiency of sparse attention, and strategies for controlling bias in asynchronous training, and closes with practical takeaways for enterprise adoption.

    1. Introduction: As the Gap Narrows, How Open-Source Models Make Trade-offs

    Several international research firms have argued in recent market analyses that the top closed-source models remain the “smartest,” but that the gap between them and the strongest open-source models has narrowed considerably. Based on this, some overseas venture investors have described a leading Chinese open-source model as the “best open-source model.”

    This is the first time the open-source camp has sat on the same level as leading closed-source models across multiple public benchmarks. In this context, the value of a complete open-source LLM technical report is no longer just about topping scoreboards. It also shows how a team systematically makes technical trade-offs under constraints on compute, data, and engineering resources.

    A recently released open-source LLM technical report runs 40 pages. The most discussed technical points include a trainable native sparse attention mechanism, a fully asynchronous agent RL training framework, and the team’s in-house RL training infrastructure. This article reorganizes the content while staying faithful to the report’s facts, and expands on the engineering principles behind each technique.

    2. Fundamentals: 744B Total Parameters, 40B Activated Parameters

    2.1 A Set of Key Numbers

    Dimension Previous Generation Current Generation
    Model architecture Mixture of Experts (MoE) Mixture of Experts (MoE)
    Total parameters ~355B 744B
    Activated parameters per inference ~32B 40B
    Experts / layers Not disclosed 256 / 80
    Training tokens ~23T ~28.5T (~27T pretraining, ~1.5T mid-training)

    Performance in public evaluations:

    • Composite intelligence index: In a third-party evaluation agency’s intelligence index (v4.0), the model scored 50, ranking first among open-source models and second only to the top overseas closed-source model.
    • Community arena: In a crowdsourced voting arena organized by an open-source community, the model ranked first among open-source models on both the text and code leaderboards, placing it in the same tier as leading overseas closed-source models.

    2.2 Reading the “Total Parameters” vs. “Activated Parameters” in MoE

    The core idea of the Mixture of Experts (MoE) architecture is to place a large number of parallel “expert” sub-networks inside the model, with a routing mechanism selecting a small subset of experts to participate in computation for each token. As a result, the model’s knowledge capacity is determined by total parameters, while the compute for a single token depends only on activated parameters.

    In engineering terms, this decouples memory from compute:

    • During inference, all weights (or weights grouped by expert) must be loaded into memory, so memory usage is tied to total parameters.
    • The matrix-multiplication size for a single token depends only on activated parameters, so decoding throughput is governed more by the activated scale and by the memory-bandwidth pressure of reading weights.

    So “744B total parameters” does not mean “744B worth of inference cost.” But MoE is not without its costs:

    • Expert parallelism requires frequent token exchange between nodes, placing heavy demands on network bandwidth.
    • When routing load is uneven, the busiest expert becomes the weak link, and capacity factors, auxiliary losses, or dynamic batching are needed to mitigate this.
    • In long-context scenarios, memory pressure often comes from the KV cache rather than the weights themselves.

    2.3 Design Intent Behind the Parameter Allocation

    Compared with the previous generation, the current model does three things:

    1. Total parameters double from ~355B to 744B, with the total number of experts expanded to 256.
    2. Activated parameters increase from 32B to 40B, far below the growth rate of total parameters.
    3. Training tokens increase from 23T to 28.5T, of which ~1.5T is dedicated to mid-training.

    There is a clear design intent behind this:

    • Scale capacity, not compute: Doubling total parameters buys stronger knowledge capacity, while the 40B activated scale keeps inference compute within a reasonable range.
    • Mid-training to fill capability gaps: Adding a targeted mid-training phase after pretraining is a common industry practice for strengthening specialized capabilities such as reasoning and code, at a cost far lower than retraining from scratch.

    3. Three Key Architecture Changes

    Compared with the previous generation, the current model has three notable architectural changes: MLA with coordinated optimizer handling, multi-token prediction, and sparse attention. Together they can be summarized as “three savings”:

    • Save memory: Low-rank KV compression and sparse attention cut long-context overhead.
    • Save training cost: Letting the model learn sparse attention directly during pretraining.
    • Save inference time: Multi-token prediction heads combined with speculative decoding speed up generation.

    3.1 Change One: MLA + Head-Level Optimizer Split

    Background: Why the KV Cache Dominates Long-Context Cost

    In standard attention, every time a new token is generated, the model must reference the Key and Value of all previous tokens. These tensors are cached in memory as the KV cache. Its size is roughly proportional to “number of layers × number of KV heads × head dimension × sequence length” and grows linearly with context length.

    Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) compress the cache by having multiple query heads share a small number of KV heads. The MLA (Multi-head Latent Attention) adopted by the current model goes further:

    • The input is first compressed via low-rank projection into a latent vector whose dimension is far smaller than “number of heads × head dimension,” and only this latent vector is cached.
    • When computing attention, the Key and Value are reconstructed separately via “up-projection.”

    With a carefully chosen low-rank dimension, MLA can noticeably reduce the KV cache while maintaining comparable or even better representational power. The cost is that reconstruction matrix multiplications must be performed at every inference step, so highly optimized kernels are needed to turn the memory advantage into real speed.

    Problem: The Optimizer Is “Incompatible” with the Attention Structure

    The training team discovered a counterintuitive phenomenon: combining MLA with matrix-orthogonalization optimizers actually performed worse than a simpler GQA baseline with 8 KV heads. The typical approach of such optimizers (e.g., Muon) is to avoid using gradients directly, and instead approximately orthogonalize the gradient matrix before updating the weights. The goal is to keep updates as balanced as possible across all directions, thereby accelerating convergence.

    One plausible explanation is that the conflict lies in the “update granularity.” MLA compresses multi-head KV into a single low-rank latent vector, which by itself emphasizes sharing across heads. If orthogonalization is applied to the entire projection matrix, each attention head’s internal subspace lacks independent, balanced update directions, which limits representational power. By contrast, GQA-8 explicitly retains a grouped structure, making it more compatible with block-wise orthogonal updates.

    Solution: Muon Split (Head-Level Orthogonalization)

    The report’s solution changes “whole-matrix orthogonalization” into “per-head orthogonalization,” which the paper calls Muon Split: first slice the projection matrix into sub-matrices that correspond one-to-one with the attention heads, then orthogonalize each sub-matrix separately.

    The illustrative code below shows the difference between the two approaches:

    1. # Illustrative code: core idea of Muon Split
    2. # W: projection weight matrix, shape [out_features, in_features]
    3. # out_features = num_heads * head_dim
    4. # Shows the update idea only, without engineering implementation details
    5. # Approach A (conventional): orthogonalize the whole matrix at once
    6. # def orthogonalize(G):
    7. # return G @ inv(sqrt(G.T @ G + eps)) # simplified representation
    8. def split_update(W, grad, lr, num_heads, orthogonalize):
    9. head_dim = W.shape[0] // num_heads
    10. W_new = W.clone()
    11. for h in range(num_heads):
    12. lo, hi = h * head_dim, (h + 1) * head_dim
    13. # Approach B (Muon Split): orthogonalize only the sub-matrix of this head
    14. W_new[lo:hi] -= lr * orthogonalize(grad[lo:hi])
    15. return W_new

    Per-head splitting brings two benefits:

    1. Aligned update structure: The update direction matches the model’s inherent “per-head” organization, so each head receives independent and balanced updates.
    2. Parallel-friendly: The orthogonalization of each head is independent and can be processed in batch parallel. On models with hundreds of attention heads, the overhead is negligible.

    Practical takeaway: The optimizer is not a plug-and-play component. After changing the attention structure, the optimizer’s behavior also changes, so the combination must be re-validated with small-scale ablation experiments before large-scale pretraining.

    3.2 Change Two: Multi-Token Prediction (MTP)

    Principle: Predicting Multiple Future Tokens at Once

    A conventional language model predicts only the next token at each position. Multi-Token Prediction (MTP) has the model predict several future tokens at each position simultaneously: typically, a few lightweight prediction modules are attached after the output layer of the backbone.

    Taking a two-token prediction as an example, it can be expressed with the following pseudocode:

    1. # Illustrative code: multi-token prediction structure (training phase)
    2. # h: hidden state from the last layer of the backbone
    3. h1 = head_1(h) # predict position t+1
    4. h2 = head_2(transformer_block(cat(h, embed(y_next)))) # predict position t+2
    5. loss = ce(h1, y_next) + ce(h2, y_nextnext)

    The training loss consists of two parts:

    • The standard cross-entropy for the backbone’s “next-token prediction.”
    • The cross-entropy corresponding to each auxiliary prediction head.

    Benefits: Training Signal and Inference Speedup

    From a training perspective, multi-token prediction forces the model to “look ahead” over a longer horizon at each position, so the hidden state must encode more future information. This effectively adds a denser training signal, helping the model learn longer-range dependencies and improving coherence when generating code and structured text.

    From an inference perspective, the trained multi-token prediction heads can act as a “draft model”:

    1. The draft heads quickly produce candidate tokens for several future positions.
    2. The main model verifies these candidates in a single forward pass, in parallel.
    3. Accepted portions are taken directly, and sampling resumes at the first mismatched position.

    This is the basic idea of speculative decoding. Its speedup depends on the acceptance rate:

    • For highly structured text such as code and JSON, where continuation patterns are relatively predictable, the gains are usually more pronounced.
    • For creative tasks with high randomness, the gains may be limited.

    Practical notes:

    • Auxiliary heads add a small amount of training and memory overhead, usually negligible relative to the backbone.
    • The number and structure of auxiliary heads should be determined through ablation experiments; too many can crowd out the backbone’s capacity.
    • Whether MTP can actually be used for inference speedup also depends on whether the inference engine supports speculative decoding.

    3.3 Change Three: DSA Sparse Attention

    Problem: The Quadratic Complexity of Dense Attention

    In standard attention, each token must compute relevance against all previous tokens, so the computation grows quadratically with sequence length. When context expands from a few thousand tokens to hundreds of thousands or even millions, the attention portion becomes the main bottleneck in both training and inference.

    From Dense to Sparse: Where the Key Difference Lies

    A common community practice is to truncate attention at inference time, such as sliding windows or local attention:

    • This reduces inference cost, but the model has never learned how to work with such a restriction, so long-range information may simply be discarded.
    • As a result, it amounts to “bolt-on” sparsity.

    The DSA (DeepSeek Sparse Attention, a trainable sparse attention) adopted by the current model puts sparsity into pretraining:

    • A lightweight indexer first scores each query token, assessing the relevance of all historical tokens.
    • Based on this, it selects the top-k “truly important” historical tokens for each token.
    • Attention is computed only over these k selected tokens.

    Its essence is to let the model learn “where to look” during training. No matter where a key token appears in the long context, it has a chance to be selected and to participate in computation.

    Results: 20B Tokens Matching 900B+ Tokens

    The most closely watched figure in the report is this: in the attention pretraining stage, the model needed only about 20 billion (20B) tokens to match the effect previously achieved with dense training on roughly 943.7 billion (943.7B) tokens. This represents roughly a 47x improvement in data efficiency.

    More importantly, this report moves sparsity to the front of the pretraining stage rather than applying it as an after-the-fact fix:

    • The training cost of attention layers drops substantially as sequence length grows.
    • During long-context inference, each token only attends to a fixed k tokens, so attention cost no longer expands linearly with context.
    • Pretraining on massive long-sequence corpora only becomes economically viable when done this way.

    Principle Illustration and Engineering Notes

    The illustrative code below shows the basic flow of sparse attention:

    1. # Illustrative code: core steps of sparse attention (simplified)
    2. # indexer: lightweight scoring module; k: number of history tokens kept per query
    3. # 1. Score each history token to get relevance scores
    4. index_scores = indexer(q, k_cache) # [batch, heads, seq_len]
    5. # 2. Select the k most relevant history tokens per query
    6. topk_idx = topk(index_scores, k=k) # [batch, heads, k]
    7. # 3. Compute attention only on the selected tokens
    8. attn_out = sparse_attention(q, k_cache, v_cache, topk_idx)

    Three points to note in practice:

    • Scoring and memory-access overhead: The indexer is lightweight, but top-k selection and sparse memory access need dedicated optimization on GPUs; otherwise the real speedup can fall far short of the theoretical value.
    • Information recall: Sparse selection is essentially an approximate retrieval. For tasks where key information is scattered across a long text, such as multi-hop reasoning and cross-document retrieval, recall must be validated.
    • Low precision and memory: Attention is often computed at low precision and low memory footprint, and combined with KV-cache compression, only then can the cost of long context be truly brought down.

    4. Agent RL: From Synchronous Training to a Fully Asynchronous Framework

    4.1 Why Agent RL Needs a New Framework

    The most closely watched part of this generation’s training is the fully asynchronous agent RL framework and the team’s in-house RL infrastructure built for it. The core problem to solve is: the “long tail” of complex agent tasks is eating away GPU utilization.

    The “synchronous” dilemma of conventional RL

    The classic synchronous RL loop is “sample a batch, take one training step”:

    1. Use the current policy to generate a batch of trajectories.
    2. Wait for the entire batch of trajectories to finish.
    3. Compute rewards and update parameters.
    4. Use the updated policy to generate the next batch.

    This flow is mathematically clean, and each step uses data collected from the same policy version, making it strictly on-policy training. But in agent scenarios, a single trajectory may contain dozens of interaction rounds, along with time-consuming operations such as code execution, tool calls, and retrieval. Trajectory lengths vary enormously: simple tasks finish in a few steps, while complex ones may take dozens. The whole batch must wait for the slowest trajectory to finish, so fast samplers can only sit idle, wasting compute severely.

    The bias problem introduced by asynchrony

    After decoupling sampling and training, samplers can keep generating data with an old policy while the trainer is already doing gradient updates with an updated policy. This introduces two new problems:

    • Policy lag: Data comes from an “outdated” policy, deviating from the strict on-policy assumption.
    • Distribution drift: The old weights held by the samplers differ from the trainer’s latest weights, so the data distribution may drift away from the current policy.

    4.2 Core Design of the Fully Asynchronous Framework

    The fully asynchronous framework adopted in the report solves both the efficiency and bias problems simultaneously through three mechanisms.

    (1) Complete decoupling of sampling and training

    Sampling and training no longer wait on each other in turns, but each runs continuously:

    • The sampling side continuously produces trajectories, feeding them into the training queue as they are generated.
    • The training side consumes data in batches and updates parameters.
    • New parameters are then asynchronously synced back to the sampling side.
    1. # Illustrative code: fully asynchronous RL training loop
    2. # Sampling, scoring, and training run concurrently without blocking each other
    3. rollout_pool.submit(rollout_worker, policy_snapshot) # continuous sampling
    4. while not converged:
    5. trajs = rollout_pool.collect_ready(min_size=BATCH) # train on whatever is ready
    6. trajs = reward_service.score(trajs) # compute rewards
    7. loss = rl_objective.compute(trajs) # compute objective
    8. trainer.step(loss) # train while sampling continues
    9. weight_sync.publish(trainer.latest_weights()) # push new weights asynchronously

    (2) Explicit sample weighting and version management

    Each trajectory carries the parameter version number from when it was collected. During training, the framework combines the importance ratio of old-to-new policy probabilities and applies a KL constraint against the reference policy to keep bias within a controllable range. This approach is close to “asynchronous proximal policy optimization,” which has been widely validated in large-scale distributed training.

    (3) In-house RL infrastructure

    Supporting the framework above requires an RL infrastructure that connects the sampling engine, training engine, weight synchronization, and reward service. Its core components typically include:

    • Sampling engine: A high-throughput inference service supporting multi-turn agent interactions, tool calls, sandboxed code execution, and automatic retries.
    • Training engine: Support for large-scale parallelism and efficient training of long-sequence and mixture-of-experts models.
    • Weight synchronization channel: Ensures the sampling side can switch to the latest parameters within seconds.
    • Reward service: Unifies reward sources such as rule-based checks, unit tests, and model review, ensuring stable and reproducible scoring.
    • Observability system: Records key metrics such as trajectory length, reward distribution, and old-to-new policy probability ratios.

    4.3 Common Problems and Debugging Approaches in Asynchronous Training

    Problem Typical Symptom Debugging Approach
    Reward hacking The model learns to “cheat the reward function,” e.g., producing code that passes tests but violates intent Keep human review and adversarial samples; combine rule-based and model-based review for rewards
    Policy staleness Unstable training, increased variance, heavily long-tailed old-to-new probability ratios Limit the version gap between sampling and training; use probability-ratio clipping and KL regularization
    Distribution drift The sampling distribution drifts from the current policy, slowing convergence Monitor divergence between old and new policies; discard overly stale samples or lower the learning rate when needed
    Memory peaks When sampling and training share GPUs, weight switching maxes out memory Plan the timing of weight loading and release; allocate memory dynamically by phase
    Long-tail throughput drag Individual trajectories take too long to finish, blocking the sampling queue Set per-trajectory timeouts and truncation, combined with dynamic batching and priority scheduling

    5. Practical Takeaways for Enterprise Adoption

    1. Plan capacity by distinguishing “total parameters” from “activated parameters”
      When evaluating an open-source MoE model, estimate memory by total parameters and compute by activated parameters, then run stress tests on your own business datasets to validate throughput, first-token latency, and long-context performance.

    2. Architecture changes must be paired with optimizer-adaptation experiments
      The incompatibility between MLA and the optimizer shows that after changing the attention structure, you need to re-compare the convergence speed and stability of different optimizers in small-scale ablations, avoiding costly trial-and-error at large training scale.

    3. For long-context scenarios, prioritize the “sparse + compression” combination
      Sparse attention turns “where to look” into a learnable decision, while KV-cache compression lowers the cost of “how much to look at.” Together they can significantly relieve the memory and latency pressure of long texts and code-base-level understanding.

    4. Agent capabilities require coordination between a “training framework” and an “inference system”
      Asynchronous sampling, dynamic batching, tool sandboxes, and weight synchronization are all indispensable. When evaluating an open-source model with agent capabilities, beyond benchmark scores, pay more attention to tool-call stability, multi-turn task completion rates, and observability.

    5. Build a continuous evaluation and regression system
      Public leaderboard scores can only serve as an initial screening criterion. Enterprises should build a tiered evaluation set based on their business (fundamental capabilities, domain knowledge, tool use, long context), pair it with logging services and monitoring alerts, run regression tests after every model update, and make staged rollouts with automatic rollback part of the standard workflow.

    6. Conclusion

    The value of this technical report lies not only in topping leaderboards, but in presenting a complete engineering paradigm:

    • At the architecture level: MLA reduces the KV cache, Muon Split realigns the optimizer with the attention structure, multi-token prediction strengthens the training signal and supports speculative decoding, and DSA makes sparsity part of the model itself.
    • At the scale level: The design of 744B total parameters and 40B activated parameters decouples knowledge capacity from inference cost, while 28.5T training tokens balance breadth and specialized capability.
    • At the training level: The fully asynchronous agent RL framework, together with the supporting in-house RL infrastructure, directly confronts the erosion of compute by long-tail tasks.

    The gap between open and closed source is narrowing, but what truly determines application results is often how you plug these capabilities into your own data, tools, and evaluation systems. Understanding the design logic behind technical reports like this one offers more long-term value than remembering any single benchmark score.