How Open-Source LLMs Are Closing the Gap with Closed Models: Sparse Attention and Asynchronous Training
This article examines the architectural and training techniques described in an open-source model technical report, including Mixture of Experts, Multi-head Latent Attention, multi-token prediction, dynamic sparse attention, and fully asynchronous agent reinforcement learning. It explains the problems each technique addresses, the trade-offs involved, and the validation checks developers should perform before adopting similar designs.
Why Open-Source Models Are Rapidly Approaching Closed Models
The capabilities of open-source large language models have improved significantly over the past two years. On tasks such as code generation and multi-step reasoning, some open-source models now approach the performance of leading closed models. This progress does not come from a single change. It results from the combined effects of model architecture, training objectives, attention mechanisms, and post-training frameworks.
Following the structure of the technical report, this article explains the problems addressed by these design choices and the conditions developers should consider when applying them.
All techniques discussed here are based on the report’s descriptions. Specific figures and comparison results depend on the evaluation settings used in the original report and are not reproduced here. Readers who need to cite those results should consult the original publication directly.
1. Mixture of Experts: Separate Total Parameters from Active Parameters
Mixture of Experts (MoE) is a foundational architecture for this class of models. An MoE model contains many expert modules, but only a subset of them is activated for each inference step. As a result, the model can have a large total parameter count while keeping the computation required for each token within a manageable range.
The key distinction is between two figures:
- Total parameters determine the model’s potential capacity.
- Active parameters determine the actual computational cost for each token.
When comparing different models, both figures should be considered. Looking only at the total parameter count can lead to an overestimate of inference costs.
The number of experts, the number of layers, and the routing strategy also affect training stability and communication overhead. If routing is unbalanced, some experts may remain underused for long periods, wasting part of the model’s capacity. Load balancing therefore usually requires dedicated tuning during MoE training.
2. Adapting the Optimizer to Multi-head Latent Attention
Multi-head Latent Attention (MLA) reduces memory consumption by compressing the key-value cache. During long-context processing, the cache is often one of the largest consumers of GPU memory. Compression allows the same memory budget to support longer inputs or larger batches.
The report notes that when the Muon optimizer was used with this attention structure, its training results were weaker than those of a Grouped-Query Attention (GQA) configuration. The team addressed the issue by changing the granularity of orthogonalization.
Instead of orthogonalizing an entire projection matrix as a single block, the process was applied separately to each attention head. Orthogonalization can make weight updates more uniform. Adjusting its granularity reduced the conflict between the optimizer and the attention architecture.
This example shows that an optimizer and a model architecture must be evaluated as a combination. An optimizer that works well with a conventional architecture may not perform equally well with a newer attention design. When incompatibilities appear, a practical approach is to first identify where they occur, then decide whether to modify the optimizer or the architecture.
3. Multi-Token Prediction
In conventional language-model training, each position predicts only the next token. Multi-token prediction requires the model to predict several subsequent positions at the same time. This provides additional supervision and encourages the hidden states to retain information over longer distances.
This objective may be useful for tasks that depend on long-range relationships, such as long-form code completion and multi-step reasoning. The trade-off is a more complex training objective, while additional prediction heads introduce some extra computation. Whether to adopt this approach depends on the task and the available training budget.
4. Sparse Attention
The computational cost of standard attention grows quadratically with sequence length, making it a major bottleneck for long-text processing. Sparse attention computes attention weights only between selected positions, reducing the overall cost. The central challenge is deciding which positions should participate in the computation. Poor selection can discard critical information.
The report’s DSA sparse attention mechanism dynamically selects the positions that should receive attention based on the input content. This approach is more adaptive, but the selection process itself requires computation and makes the implementation more complex.
When evaluating sparse attention, the comparison conditions must be clearly defined. These include the task type, evaluation metrics, baseline model, and training stage. If a report presents only differences in training-data volume without explaining these conditions, its conclusions can easily be misunderstood.
Claims about data efficiency should therefore be treated as conditional. Their applicability depends on the evaluation settings specified in the original report.
5. Fully Asynchronous Agent Reinforcement Learning
Reinforcement learning is used during post-training to align models with human preferences and improve task performance. In synchronous training, all environment interactions must finish before the parameters are updated. The slowest environment can hold up the entire process.
Asynchronous training allows different agents to interact with their environments and update independently, which can improve hardware utilization.
However, asynchronous training introduces new challenges:
- Data freshness: Trajectories used for updates may have been generated by older policy versions.
- Gradient delay: The gradients used during an update may have been calculated from stale parameters.
- Inconsistent reward signals: Different environments may use reward scales that are not directly comparable. Mixing them without adjustment can affect training.
The report states that the team addressed these issues through scheduling strategies and buffering mechanisms. The specific implementation details should be verified against the original report.
6. A Checklist for Developers
- Verify parameter counts: Distinguish total parameters from active parameters, and confirm the sources of the layer and expert counts.
- Reproduce evaluations: Record the evaluation version, task set, metric definitions, and baseline configuration.
- Monitor stability: Track loss curves, expert load distribution, and gradient norms.
- Measure long-context costs: Test memory usage and inference latency at the target context length.
- Validate component combinations: After changing the optimizer or attention architecture, run small-scale comparative experiments first.
- Inspect asynchronous data: Record the policy version associated with each trajectory and define a staleness threshold.
Conclusion
The progress of open-source models is largely the result of continued accumulation of engineering improvements. Mixture of Experts, latent attention, multi-token prediction, sparse attention, and asynchronous reinforcement learning each address a specific problem.
For developers, the most valuable lesson is to understand the conditions and risks associated with each technique. Adoption decisions should then be based on the target task, available resources, and results from controlled validation.
