Open-Source LLM Technical Report: MoE Architecture, Sparse Attention, and Asynchronous Agent RL

    This article reviews an open-source large model technical report that discloses architecture, pretraining, and reinforcement learning details. It examines how the model balances total parameter scale with activated parameters per inference, how attention compression and sparsification reduce long-context costs, and how asynchronous Agent RL organizes tool calls, environment interaction, and training updates into a sustainable engineering pipeline. Readers get a neutral, section-by-section interpretation covering base configuration, attention and optimizer design, multi-token prediction, dynamic sparse attention, asynchronous Agent RL, infrastructure, and engineering recommendations.

    A recently released technical report on an open-source large model discloses relatively complete details about its architecture, pretraining, and reinforcement learning training. Rather than focusing only on parameter scale or leaderboard scores, developers may find it more useful to study three questions: how the model controls activated parameters per inference while expanding total parameters, how attention compression and sparsification reduce long-context costs, and how asynchronous Agent RL organizes tool calls, environment interaction, and training updates into a sustainable engineering pipeline. Based on the information disclosed in the technical report, this article provides a neutral interpretation in the following order: base configuration, attention and optimizer, multi-token prediction, sparse attention, asynchronous Agent RL, infrastructure, and engineering recommendations.

    1. Base Configuration and Effective Compute: Why MoE Scaling Is Not Just About Total Parameters

    The model in the report follows a Mixture of Experts (MoE) architecture. The basic idea is that the model as a whole has a large parameter capacity, but each inference activates only a small subset of experts, striking a trade-off between capacity and compute cost. The report discloses an example configuration with about 744B total parameters, about 40B activated parameters per inference, 256 experts, and 80 layers. Compared with the previous generation, total parameters increased from about 355B to about 744B, activated parameters increased from about 32B to about 40B, and pretraining data increased from about 23T tokens to about 28.5T tokens, including about 27T for pretraining and about 1.5T for mid-training.

    These numbers send a clear signal: total parameters nearly doubled, but activated parameters increased only modestly, indicating that scaling focused on capacity rather than compute per forward pass. For engineering teams, the difficulty of MoE is not just the number of experts, but routing stability, expert load balancing, communication overhead, and training throughput. With more experts, token routing is more likely to concentrate on a few experts, leaving some experts undertrained. With deeper layers, pipeline parallelism and communication scheduling become more complex. Therefore, evaluating MoE should not rely only on total parameters; it should also consider activated parameters, expert parallelism strategy, routing loss, load-balancing metrics, and per-token training cost.

    From the perspective of effective compute, the goal of MoE is to improve the amount of effective learning per unit of compute. If activated parameters grow slowly while training data increases significantly, the model may rely more on capacity to absorb knowledge. If routing quality is poor, additional experts may not translate into capability. The ratio of total parameters to activated parameters in the report is better analyzed through a capacity/compute ratio than through simple horizontal comparisons of total parameter counts.

    2. Attention Compression and Optimizer Co-Design: Low-Rank KV Cache and Per-Head Orthogonalization

    For attention, the report takes a low-rank KV compression route, compressing the KV cache dimension to reduce memory usage and improve inference efficiency in long-text scenarios. The core benefit of this approach is that during long-context decoding, the KV cache often becomes a memory and bandwidth bottleneck. With compressed dimensions, the per-token cache is smaller, which can support longer contexts or higher concurrency.

    However, the report mentions a training issue worth noting: when low-rank KV compressed attention was paired with an optimizer based on matrix orthogonalization, performance initially lagged behind a simpler grouped-query attention approach. The team’s solution was to perform orthogonalization separately for each attention head, rather than applying uniform orthogonalization to the entire projection matrix. This detail shows that attention structure and optimizer cannot be selected independently. Low-rank compression changes the spectral structure and gradient distribution of parameter matrices. Uniform orthogonalization may ignore scale differences across attention heads. With per-head splitting, each head’s projection matrix receives more uniform update constraints, improving training stability and convergence.

    In engineering terms, this can be abstracted into a lesson: when using low-rank compression, grouped-query attention, or multi-head structures, the optimizer’s parameter grouping strategy should align with the physical structure of attention heads. Otherwise, global statistics may mask abnormal gradients in local heads. In practice, teams should monitor per-head gradient norm, weight norm, activation variance, and output entropy to prevent a few degraded heads from being hidden by global averaging.

    3. Multi-Token Prediction: From Single-Step Supervision to Multi-Step Planning

    The second architectural change in the report is multi-token prediction. Traditional autoregressive training usually predicts the next token at each position. Multi-token prediction lets the model predict multiple future tokens in one forward pass, or uses auxiliary heads to supervise multiple subsequent positions. Potential benefits include better training-sample utilization, stronger planning ability for short-term future structure, and possibly improved coherence during generation.

    However, multi-token prediction is not simply a matter of adding a loss term. First, different prediction steps have different difficulty levels, and farther tokens have higher uncertainty, so loss weights need careful design. Second, multi-token heads introduce additional parameters and computation; if the activated-parameter budget is fixed, they must be traded off against backbone capacity. Finally, changes in the training objective may affect how the model models long-range dependencies. In engineering deployment, multi-token prediction should be treated as an auxiliary training objective. Teams should focus on validation perplexity, long-text consistency, code-completion pass rate, and instruction-following stability, rather than only on training loss reduction.

    4. Dynamic Sparse Attention: Reducing Long-Context Cost with Selection

    Another point discussed extensively in the report is dynamic sparse attention. Standard attention computes all token pairs, and complexity grows rapidly with context length. Sparse attention retains only some key token pairs or blocks, reducing compute and memory overhead. According to the report, the sparse route reached results close to large-scale dense attention training with significantly fewer training tokens. This suggests that sparsification is not only an inference acceleration technique, but may also be an architectural choice for improving efficiency during training.

    Pseudocode can express the basic logic:

    1. def dynamic_sparse_attention(q, k, v, topk):
    2. scores = q @ k.transpose(-1, -2)
    3. idx = scores.topk(topk, dim=-1).indices
    4. sparse_scores = gather(scores, idx)
    5. sparse_v = gather(v, idx)
    6. return softmax(sparse_scores) @ sparse_v

    A real system is far more complex than pseudocode: it requires block sparsity, routing networks, load balancing, mask stability, and backpropagation support. The risk of sparse attention is “wrong selection”: if key context is not selected, the model loses information; if selection patterns are too fixed, attention may degrade into local attention. Therefore, training should monitor sparsity rate, the distribution of selected tokens, selection entropy across layers, and long-context task metrics. For long-document question answering, code repository understanding, and multi-turn dialogue, sparse attention must be validated through end-to-end evaluation rather than relying only on theoretical complexity.

    5. Fully Asynchronous Agent RL Training Framework

    Another main thread in the report is Agent RL. Unlike traditional single-turn RL, Agent tasks usually involve multi-step tool calls, environment feedback, long trajectories, and uncertain latency. With synchronous training, the rollout phase may be blocked by slow environments, reducing GPU utilization. A fully asynchronous framework decouples trajectory sampling from parameter updates: sampling workers continuously generate trajectories, while the training side consumes experience in batches, improving throughput.

    However, asynchrony introduces policy staleness: sampled trajectories may have been generated by older parameter versions while training has already updated several times. In engineering, version gaps need to be controlled, using methods such as importance sampling, clipping, experience replay, and stale-sample filtering to reduce bias. This can be abstracted into the following loop:

    1. while training:
    2. batch = replay.sample()
    3. if max_version_gap(batch) > threshold:
    4. batch = filter_stale(batch)
    5. update = train_step(batch)
    6. replay.add(rollout_worker.collect())

    Key monitoring metrics for asynchronous Agent RL include trajectory length distribution, tool-call success rate, environment timeout rate, sample version gap, importance weight distribution, gradient norm, and reward distribution drift. If throughput is pursued while ignoring staleness, the policy may oscillate; if synchronization is excessive, the benefits of asynchrony are lost. A reasonable approach is to layer tasks by latency: short trajectories can be updated at high frequency, while long trajectories can use stricter version windows.

    6. RL Infrastructure: Treating the Training Framework as a Systems Problem

    The report mentions self-developed RL infrastructure. For Agent RL, infrastructure should at least include environment pools and tool-call sandboxes, trajectory storage, experience replay, parameter version management, distributed training, checkpointing and fault tolerance, and logging and monitoring. The difficulty is that the data pipeline is not steady-state: environment latency, tool failures, and differences in trajectory length all cause batch imbalance. Therefore, the process of sampling, filtering, training, and evaluation needs to be an observable pipeline, rather than only optimizing single-step training speed.

    In general cloud product categories, object storage, message queues, container platforms, logging services, and monitoring and alerting can all play corresponding roles, but specific choices should serve throughput, isolation, and reproducibility. In particular, Agent trajectories contain tool outputs and intermediate states and must be traceable; training parameter versions and trajectory versions must be bound; failed samples should not simply be discarded, or distribution bias may be introduced.

    7. Engineering Recommendations and Boundaries

    First, establish an “effective compute” perspective. Do not compare only total parameters. Record activated parameters, training tokens, expert load, communication share, and per-token cost at the same time. Second, tune attention and the optimizer jointly. Low-rank compression, grouped-query attention, and per-head orthogonalization splitting should be validated as a group of design choices. Third, multi-token prediction needs clear auxiliary-objective weights, and its benefits should be validated with long-text and code tasks. Fourth, sparse attention must go through end-to-end regression, with emphasis on key-information recall and long-context stability. Fifth, asynchronous Agent RL should set version-gap thresholds and stale-sample policies to avoid trading throughput gains for policy drift.

    Finally, it is important to emphasize that a technical report discloses specific configurations and training settings and cannot be directly equated with the optimal solution for all scenarios. MoE routing, sparse selection, and asynchronous RL all increase system complexity; leaderboard performance also cannot replace real business evaluation. For developers, the most valuable takeaway is not to replicate a particular parameter number, but to understand the trade-offs behind it: capacity for effective compute, compression for long context, sparsity for complexity, asynchrony for throughput, and monitoring and reproducibility to control risk.