From Sparse Attention to Asynchronous Agent Training: A Technical Review of an Open-Source Large Model

    This technical review examines the system design behind a large open-source model, including its mixture-of-experts architecture, latent attention, multi-token prediction, dynamic sparse attention, and fully asynchronous reinforcement learning for AI agents. It explains how these techniques improve effective compute efficiency, long-context inference, training signal density, and infrastructure utilization, while also highlighting their engineering trade-offs and implications for real-world model selection.

    1. From Parameter Count to Effective Compute Efficiency

    In recent years, competition among large language models has shifted from simply increasing parameter counts to improving the amount of useful capability delivered per unit of compute. A model with hundreds of billions of parameters can still create significant memory, communication, and latency pressure if all parameters must be activated during inference. As a result, many leading architectures use a mixture-of-experts (MoE) design, in which only a small number of experts process each input token.

    According to the technical report, the model uses a mixture-of-experts architecture with approximately 744 billion total parameters, while activating about 40 billion parameters for a single inference step. Internally, it contains 256 experts and 80 layers. This design has several implications:

    1. Total parameters determine the upper limit of model capacity
      A large parameter space can store more linguistic knowledge, code patterns, reasoning experience, and domain information.

    2. Activated parameters determine the cost of each computation step
      By invoking only a subset of experts during inference, the model can retain the capacity of a much larger model while reducing the amount of matrix computation.

    3. Expert routing affects actual model quality
      The routing module must assign different types of tasks to the experts best suited to handle them. Unstable routing can result in uneven expert loads, insufficient training for some experts, or routing fluctuations.

    Compared with the previous-generation model, the total parameter count increased from approximately 355 billion to 744 billion. Activated parameters increased from about 32 billion to 40 billion, while the training corpus grew from roughly 23 trillion tokens to approximately 28.5 trillion tokens. The latter includes about 27 trillion tokens of pretraining data and 1.5 trillion tokens used for mid-training.

    Parameter count and data volume should not be viewed as having a simple linear relationship. Model quality is also influenced by data quality, training mixtures, optimizer design, parallelism strategies, context length, post-training methods, and evaluation-contamination controls.

    2. Latent Attention: Reducing Cache Pressure in Long-Context Inference

    Traditional attention mechanisms must preserve large key-value caches during inference. As context length increases, the key-value cache can consume a substantial amount of GPU memory. This is especially problematic for batched requests and multi-turn conversations, where the cache often becomes a primary bottleneck.

    The report describes a latent-variable attention mechanism. Instead of storing complete key and value representations directly, the model first compresses relevant information into lower-dimensional latent representations and then restores or transforms that information when attention computation is required. This can reduce the size of the key-value cache and provide several benefits:

    • Lower memory consumption during long-context inference;
    • Larger batch sizes on the same hardware;
    • More stable throughput for multi-turn conversations and long-document tasks;
    • Reduced cache-transfer pressure in distributed inference.

    Latent attention also introduces trade-offs. If the compression dimension is too small, information may be lost. The restoration process can also add computational overhead. In practice, the system must balance cache size, attention accuracy, and additional computation.

    Optimizer Compatibility with Latent Attention

    The technical report also notes that directly applying a particular orthogonalization-based optimizer to the latent attention structure initially performed worse than a more conventional grouped-query attention approach. Possible reasons include:

    • Attention projection matrices have specialized block structures;
    • Different attention heads do not necessarily perform identical semantic functions;
    • Applying the same orthogonalization operation to an entire matrix may weaken functional differences between attention heads;
    • The parameter update direction may not align well with the geometry of the latent space.

    To address this issue, the research team proposed a head-wise optimization strategy. Instead of applying a unified transformation to the complete projection matrix, the matrix is divided by attention head, and orthogonalization or parameter updates are performed separately. This reduces interference between heads and makes optimization better aligned with the attention structure.

    This experience demonstrates that optimizers cannot be evaluated independently of network architecture. An optimization method that works well on a standard Transformer may not remain effective when transferred to low-rank compression, mixture-of-experts, or specialized attention structures.

    3. Multi-Token Prediction: Increasing Training Signal Density and Inference Efficiency

    Traditional autoregressive language models generally predict one next token at a time. Multi-token prediction allows the model to learn several future positions simultaneously. Given the current context, the model may predict multiple subsequent tokens rather than focusing only on the immediate next token.

    This approach offers two primary benefits.

    1. More Supervision During Training

    For a given input position, the model learns not only what the next token should be, but also what may appear farther in the sequence. This increases the density of supervision provided by each training example and can help the model learn longer-range dependencies.

    2. More Efficient Generation Paths During Inference

    If the multi-token prediction module is designed appropriately, the model can propose several candidate tokens during inference and then use a verification mechanism to accept some of them. This may reduce the serial delay associated with generating one token at a time.

    However, multi-token prediction also introduces several challenges:

    • Uncertainty increases as the prediction distance becomes greater;
    • Predictions for different positions may conflict;
    • Training objectives must prevent the model from relying too heavily on unreliable future information;
    • Inference requires additional candidate-verification logic.

    Therefore, multi-token prediction is not simply a matter of adding several output heads. It requires coordinated design across training objectives, decoding strategies, and error-propagation control.

    4. Dynamic Sparse Attention: Focusing Computation on Relevant Long-Context Information

    In long-context tasks, full attention between every pair of tokens typically results in computational complexity that grows quadratically with sequence length. Dynamic sparse attention addresses this problem by allowing each position to focus on only the most relevant parts of the context, selected according to the content.

    Common sparse attention patterns include:

    • Local window attention: Prioritizes nearby tokens;
    • Global anchor attention: Preserves a limited number of globally important information nodes;
    • Block attention: Divides long text into multiple regions;
    • Content-routed attention: Selects important segments based on semantic relevance;
    • Cross-layer hybrid patterns: Uses different sparse strategies at different layers.

    One of the report’s most notable conclusions is that, after dynamic sparse attention training, the model can achieve some results comparable to those of a traditional dense approach trained on approximately 943.7 billion tokens, while using about 20 billion tokens. This should not be interpreted as evidence that sparse attention can unconditionally replace massive training datasets. Rather, it suggests that an appropriate structural inductive bias can significantly improve data efficiency.

    Sparse attention should be evaluated across several dimensions:

    1. Recall capability: Can important information be selected correctly?
    2. Long-range dependency modeling: Can the model connect information across chapters and documents?
    3. Training stability: Does the sparse pattern introduce discontinuities in gradient flow?
    4. Hardware compatibility: Can sparse computation translate into actual hardware acceleration?
    5. Task coverage: Do code, mathematics, retrieval, and long-document summarization all benefit?

    A reduction in theoretical complexity does not necessarily produce lower real-world latency. If sparse indices are organized inefficiently, or if the hardware cannot execute irregular memory access effectively, the practical gains may be smaller than expected.

    5. Fully Asynchronous Reinforcement Learning for AI Agents

    For models designed to handle complex tasks, static question-and-answer data alone cannot adequately represent the multi-step decision-making processes found in real environments. Agent reinforcement learning typically requires the model to interact with an environment, call tools, plan actions, correct errors, and verify results.

    Traditional training pipelines are often synchronous: the model generates a batch of trajectories, the environments execute them, and the system waits for every result before calculating rewards and updating parameters. This approach is relatively straightforward to implement, but it can waste substantial resources:

    • Different tasks may require very different execution times;
    • Slow tasks can delay the entire training batch;
    • Tool calls and environment interactions involve considerable waiting;
    • Generation, execution, evaluation, and parameter updates cannot be fully parallelized.

    Fully asynchronous training divides the system into multiple independently operating components:

    1. The sampling module generates task trajectories;
    2. The environment module executes tools and external operations;
    3. The evaluation module assesses task completion quality;
    4. The experience module stores and filters training samples;
    5. The update module calculates gradients and refreshes model parameters.

    These components exchange data through queues or buffers rather than waiting for all tasks to complete synchronously. This can improve hardware utilization and allow long-running and short-running tasks to progress in parallel.

    The central challenge in asynchronous reinforcement learning is data freshness. If sampling uses older parameters while the update module has already completed multiple training rounds, the trajectory distribution can gradually diverge from the current policy. This may reduce training stability. Systems therefore generally need to control:

    • Differences between parameter versions;
    • The maximum allowable trajectory delay;
    • The proportion of stale data in the experience pool;
    • The weights assigned to samples from different sources;
    • Filtering rules for failed trajectories and abnormal tool calls.

    6. The Engineering Value of Custom Reinforcement Learning Infrastructure

    Large-scale reinforcement learning is not only an algorithmic problem; it is also a systems-engineering problem. A complete infrastructure stack typically needs to support:

    • Distributed task scheduling;
    • Model inference services;
    • Trajectory collection and persistence;
    • Reward calculation;
    • Sample deduplication and quality filtering;
    • Parameter synchronization;
    • Training monitoring and fault recovery;
    • GPU, network, and storage resource management.

    Agent tasks make observability especially important. A single trajectory may contain multiple model calls and environment interactions. The system must record the input, output, tool results, execution time, error state, and final reward for every step. Without this information, it is difficult to determine why training failed.

    From an engineering perspective, the value of custom infrastructure is not limited to faster training. It also makes experiments reproducible, observable, and reversible. For enterprise users, these capabilities are similarly relevant to model training, inference services, log analysis, and monitoring and alerting systems on Baidu AI Cloud.

    7. How to Interpret Claims of Leading Open-Source Model Capability

    The evaluation results in the technical report indicate that the model achieved strong performance on several public capability benchmarks. For example, it received a score of 50 on a comprehensive intelligence index and ranked among the leading open-source models in text- and code-oriented competitive evaluations.

    Such results demonstrate the model’s performance on specific standardized tests, but they should not be treated as proof that it will be superior for every real-world business workload.

    When selecting a model for enterprise use, organizations should conduct additional evaluations across the following dimensions:

    • Compatibility with business data;
    • Stability on long-context tasks;
    • Accuracy of structured outputs;
    • Tool-calling success rate;
    • The percentage of generated code that runs successfully;
    • Latency and throughput under high concurrency;
    • GPU memory and network requirements for deployment;
    • The difficulty of fine-tuning, quantization, and inference-service integration;
    • Data security, access isolation, and audit capabilities.

    8. Conclusion: Large Model Competition Is Entering an Era of System-Level Coordination

    The most important aspect of this technical report is not any single benchmark result or architectural innovation. It is the coordination among multiple design directions:

    • Mixture-of-experts architecture increases model capacity while improving compute efficiency;
    • Latent attention reduces cache pressure for long-context inference;
    • Head-wise optimization improves training compatibility with specialized attention structures;
    • Multi-token prediction increases supervision density and explores faster generation;
    • Dynamic sparse attention reduces unnecessary computation;
    • Asynchronous agent reinforcement learning improves training efficiency for complex tasks;
    • Dedicated infrastructure supports stable execution of large-scale experiments.

    The future of large-model systems will be determined less by which model has the most parameters and more by which system can achieve reliable reasoning at a lower cost. For developers and enterprise users, the key consideration is the coordination among model architecture, training data, inference systems, agent frameworks, and cloud engineering capabilities.