Next-Generation Open-Source LLMs: A Deep Dive into Sparse Attention, Asynchronous Reinforcement Learning, and MoE Evolution

    A recent open-source large language model report details a 744B-parameter Mixture-of-Experts design with 40B active parameters, along with MLA, Muon Split, sparse attention, and a fully asynchronous Agent reinforcement learning framework. This article examines the model’s foundation, architecture, training methods, and infrastructure, and outlines the broader lessons for building efficient long-context and Agent systems.

    A roughly 40-page technical report on an open-source large language model has recently attracted significant attention. Evaluations from third-party organizations suggest that leading proprietary models still hold a slight advantage in overall intelligence, but the gap between top-tier open-source models and closed-source flagships has narrowed substantially.

    The report discloses a wide range of engineering details. The most discussed topics include the efficiency of sparse attention during large-scale pretraining, a fully asynchronous reinforcement learning framework for Agents, and custom-built RL infrastructure. This article examines the key ideas in four layers: the base model, architecture, training, and infrastructure.

    1. Model Fundamentals: A 744B-Parameter MoE with 40B Active Parameters

    The model uses a Mixture-of-Experts (MoE) architecture. The central idea behind MoE is to activate only a small subset of the model’s parameters during inference. This allows the model to expand its total capacity while keeping computational costs under control.

    The report specifies the following configuration:

    • 744B total parameters, with 40B parameters activated per inference step
    • 256 experts and 80 layers
    • 28.5T total pretraining tokens, including 27T tokens for pretraining and 1.5T tokens for mid-training

    Compared with the previous-generation model in the same series, total parameters increased from 355B to 744B, while active parameters grew more moderately from 32B to 40B. The pretraining dataset expanded from 23T to 28.5T tokens.

    This reflects a clear capacity strategy: substantially increase the number of experts to expand the model’s knowledge capacity, while increasing active parameters more gradually so that inference costs remain relatively predictable and close to linear.

    According to publicly reported evaluations, the model achieved the highest scores among open-source models on major intelligence benchmarks. It also ranked first among open-source models in text and code arenas, placing it broadly in the same tier as leading proprietary models from overseas. These results suggest that a high-capacity, low-activation MoE design, combined with high-quality data, can bring open-source models close to the performance of closed-source flagships.

    2. Major Architectural Changes: MLA and Muon Split

    2.1 The Core Value of MLA

    The model adopts Multi-head Latent Attention (MLA). MLA significantly reduces memory consumption by applying low-rank compression to the key-value cache.

    In conventional multi-head attention, inference requires caching complete Key and Value tensors for every attention head. As context length grows, the KV cache can become a major memory bottleneck. MLA projects KV representations into a compressed latent space. During inference, the system stores only the compressed representations and reconstructs them when needed, improving throughput for long-context workloads.

    This approach is aligned with widely used methods in the open-source community and represents a relatively mature and validated direction.

    2.2 The Compatibility Issue Between Muon and MLA

    During training, the team observed an unexpected result: combining the Muon optimizer with MLA performed worse than a simpler grouped-query attention design with eight query groups.

    Muon is an optimization method that applies orthogonalized updates to weight matrices. The result highlights an important point: optimizers and attention architectures can be tightly coupled. Combining advanced components does not necessarily produce a positive outcome when their underlying assumptions are misaligned.

    2.3 The Solution: Muon Split

    The team introduced a method called Muon Split. In the original approach, orthogonalization was applied uniformly to the entire projection matrix. Muon Split instead performs orthogonalization independently for each attention head.

    The intuition is that orthogonalizing the full matrix imposes shared constraints on the update directions of different heads. At the same time, MLA’s latent-space compression means that the effective degrees of freedom are not evenly distributed across heads. Splitting the operation by head allows each head’s update to follow its own geometric structure, bringing the optimization signal back into alignment with the architecture.

    The broader value of this change is methodological rather than merely implementation-specific. When an optimizer and an architecture do not match well, optimization operations should be applied at the architecture’s natural structural granularity.

    3. Sparse Attention: Improving Long-Context Efficiency

    Another widely discussed part of the report is sparse attention, referred to as DSA. The approach reduces computation by filtering out tokens that have low relevance to the current token.

    Its reported benefit can be illustrated with a comparison: the method matched the results previously achieved by similar approaches using several hundred billion training tokens, but did so with roughly 20B tokens. In other words, sparsification not only reduces the computation required for each step, but also improves the efficiency with which training signals are used.

    The underlying principle is consistent with the statistical behavior of attention. For most token pairs, attention weights are close to zero, meaning that a large amount of dense floating-point computation contributes very little to the final result. Sparse attention uses learnable or heuristic top-k selection to concentrate computation on the tokens that are actually relevant.

    For applications involving long documents, code repositories, and multi-turn Agent histories, this type of mechanism is increasingly important and may become a practical requirement.

    4. A Fully Asynchronous Agent RL Training Framework

    The reinforcement learning stage contains some of the report’s most substantial engineering work. Agent-oriented RL introduces three major challenges:

    • Environment interactions have unpredictable latency.
    • Trajectory lengths can vary significantly.
    • Rollout and training processes can block one another.

    In a traditional synchronous pipeline, the system collects a batch of trajectories, computes advantages, updates the model, and then begins the next collection phase. When the system is waiting for the environment to respond, GPUs may remain idle for significant periods.

    The reported solution is a fully asynchronous framework. Rollout and training processes are decoupled, policy parameters are synchronized asynchronously, and trajectories continuously flow into the training side through a pipeline.

    This design hides environment wait time behind parameter updates and substantially improves sampling throughput. Asynchronous execution is particularly well suited to Agent workloads, where environment response times can range from hundreds of milliseconds to tens of seconds. In a synchronous framework, these long-tail delays can severely reduce utilization.

    The custom RL infrastructure supports experience buffering, advantage estimation, trajectory reconstruction, and replay. Together, these components automate the loop connecting sampling, training, and evaluation.

    For teams planning to deploy Agent RL on their own workloads, several design principles are especially relevant:

    • Fully decouple rollout workers from trainers.
    • Implement an asynchronous parameter-broadcast mechanism.
    • Explicitly manage variance in trajectory lengths.

    5. Lessons for Practitioners and Enterprises

    Several broadly applicable conclusions can be drawn from the report:

    1. The capacity economics of MoE are becoming increasingly convincing. Expanding total parameters substantially while increasing active parameters more gradually offers an effective path for open-source models to approach closed-source performance.

    2. Component combinations require joint optimization. Optimizers and attention mechanisms are not independent choices. Orthogonalization and other optimization operations should respect the architecture’s natural structural divisions.

    3. Sparse attention is a major differentiator for long-context workloads. It can reduce computational cost while also lowering data requirements, giving it a strong efficiency-to-cost ratio.

    4. Asynchronous RL infrastructure can determine the upper bound of Agent capabilities. Beyond model quality, the engineering maturity of the training system is itself a competitive advantage.

    The gap between open-source and closed-source models is shifting from a generational difference toward a version-level difference. For developers, keeping up with publicly available architecture and training details—and combining them with general cloud GPU resources, container platforms, object storage, and other foundational services—offers a practical way to turn open-source progress into production capabilities.