Open-Source LLM Technical Report, Decoded: A Complete Guide to Its Architecture and Training
An in-depth walkthrough of a recently published 40-page open-source large model technical report. The article covers the model's core specs (744B total parameters, 40B activated, 256 experts, 80 layers, 28.5T training tokens) and three architectural changes—MLA with Muon Split, multi-token prediction, and trainable sparse attention—along with a fully asynchronous Agent RL training framework and self-built RL infrastructure. It explains what each technique solves, how to interpret third-party benchmark rankings, and why open-source models are shifting from a scale-driven to an efficiency-driven design, with practical takeaways for developers and cloud-based training teams.
Introduction: The Gap Between Open and Closed Source Is Being Redefined
A well-known overseas investment firm recently published a chart-based weekly report with a notable judgment: a closed-source flagship model is still the “smartest” model available, but the gap between it and the second-ranked model has narrowed dramatically—and that second-place model is open source. The assessment comes from third-party evaluation data, yet it points to a deeper trend: the capability ceiling of open-source models is no longer determined by how many parameters you pile on, but increasingly by the compounding effect of system-level capabilities like attention efficiency, sparsification, and reinforcement learning (RL) engineering.
Recently, a leading open-source large model released its complete 40-page technical report on a public paper platform. The developer community quickly dove into a page-by-page reading, and three technical points drew the most discussion: sparse attention, a fully asynchronous Agent RL training framework, and self-developed RL infrastructure.
This article draws primarily on that report. To be clear, all parameters, data, and evaluation conclusions come from the public report; the architecture interpretation and cause analysis are the author’s own technical analysis. For ease of reference, the report is referred to below as “the model.”
1. At a Glance: What This Report Solves
First, let me lay out the technical thread, then expand each item:
| Dimension | Key Content | Core Problem Solved |
|---|---|---|
| Model fundamentals | 744B total params, 40B activated, 256 experts, 80 layers; 28.5T training tokens | Scale up model capacity while keeping inference cost controllable |
| Attention structure (1) | MLA + Muon Split | Shrink the KV cache while fixing regressions introduced by a new optimizer |
| Prediction target | Multi-token prediction | Increase training signal density and leave room for inference acceleration |
| Attention structure (2) | Sparse attention | Replace expensive dense-attention pre-training with data efficiency |
| Training paradigm | Fully asynchronous Agent RL | Eliminate the “long-tail drag” and idle compute in synchronous RL |
| Engineering foundation | Self-developed RL infrastructure | Support large-scale, long-horizon, multi-turn tool-calling RL training |
2. Fundamentals: What 744B Total / 40B Activated Actually Mean
2.1 Key Parameters: Scale Capacity, Not Compute
The model uses a MoE (Mixture of Experts) architecture, whose core idea is to split the parameters into a number of expert subnetworks, activating only a small subset for each token.
- Scale: 744B total parameters, 40B activated per inference, 256 experts, 80 layers.
- Generational comparison: versus the previous generation, total parameters grew from 355B to 744B, activated parameters rose only from 32B to 40B, and training tokens increased from 23T to 28.5T.
- Data composition: of the 28.5T tokens, roughly 27T are for pre-training and about 1.5T for mid-training.
This set of numbers reflects a clear set of tradeoffs:
- Total parameters doubled, but activated parameters grew only about 25%. The benefit of sparse MoE comes mainly from “capacity”: more experts mean the model can hold more knowledge and patterns. Inference cost, by contrast, is driven mainly by activated parameters, the KV cache, and context length. Enlarging capacity aggressively while growing compute only moderately is a typical approach to controlling per-token cost.
- A structure with relatively few layers and wide width. 80 layers combined with 744B total parameters means each layer has a considerable intermediate dimension and expert capacity, so the model relies more on width and routing for its expressiveness. Such structures usually require more refined load-balancing strategies and expert-parallel communication optimizations; implementation details are in the report’s appendix.
- Mid-training as a distinct stage. 1.5T tokens are dedicated to mid-training, which typically serves a transitional role in capability improvement—for example, extending context length, strengthening code and reasoning, and improving instruction following. This reflects a staged data strategy of “pre-training as the foundation, mid-training as the shaping.”
2.2 Reading the Evaluations: Beyond the Rankings
The report offers two third-party conclusions: on a comprehensive capability index (v4.0) published by an independent evaluation institution, the model scored 50 and ranked first among open-source models; on a crowdsourced model arena platform, it ranked first among open-source models in both text and code, with an overall level now in the same tier as top overseas closed-source models.
When interpreting such conclusions, keep three things in mind:
- A composite index is a weighted aggregation of several capability dimensions, and the weighting determines what “comprehensive” actually means.
- Arena rankings come from blind preference tests by real users and are close to lived experience, but they are influenced by the distribution of prompts.
- Leaderboards carry risks of data contamination and overfitting, and they generally do not directly reflect tool-calling and long-chain Agent tasks—capabilities that enterprises care about most.
So the more reliable approach is to use leaderboards as a screening threshold, run secondary validation with an evaluation set tailored to your own business, and simultaneously track engineering metrics such as success rate, average number of steps, token consumption, and P95 latency.
3. Architecture Change 1: MLA + Muon Split
The three architectural changes listed in the report correspond to three different layers: MLA belongs to the attention-structure layer, multi-token prediction belongs to the training-objective layer, and sparse attention changes both the training method and inference overhead. Let’s go through them one by one.
3.1 MLA: Cutting the KV Cache with Low-Rank Compression
MLA (Multi-head Latent Attention) compresses each token’s Key and Value into low-dimensional latent vectors. At inference time, it caches only the latent vectors plus a small number of vectors related to positional encoding, rather than the full K and V. The benefit is direct: the KV cache shrinks substantially, so memory usage and memory-bandwidth pressure in long-context scenarios drop, and decoding throughput improves.
The cost is a more complex structure: compression and decompression add computation, the latent dimension needs careful tuning, and existing attention kernels often cannot be reused directly.
3.2 The Problem: Muon Optimizer “Doesn’t Fit”
Muon is a class of optimizers designed for matrix-shaped parameters with an orthogonalization idea: it first performs iterative orthogonalization on the momentum matrix, then uses the result for parameter updates. On some vision and language model training tasks, it is considered able to speed up convergence at comparable compute.
The team quickly hit a problem: when Muon is paired with MLA, it fails to match a simpler approach—grouped-query attention with 8 KV heads (GQA-8). This is worth pondering: a more “advanced” optimizer paired with a more “advanced” attention structure performs worse than the combination in which both are more plain.
The author’s analysis suggests two possible reasons:
- A mismatch between orthogonalization and parameter structure. Orthogonalization applied to an entire matrix alters the relative scales among rows (output channels). In GQA, each KV group corresponds to a non-overlapping set of output channels, so whole-block orthogonalization does not change the relationships among channels within a group. MLA, by contrast, relies on cross-head sharing and decoupling in the latent space, and each channel’s contribution depends more on the distribution of the latent representation; whole-matrix orthogonalization most likely disturbed this fine-grained structure.
- The optimizer and the architecture had been co-adapted to each other. The team had already tuned a large number of hyperparameters around Muon, and those may not transfer to GQA-8. In addition, Muon and AdamW interact differently with weight decay and learning-rate scheduling, so a fair comparison requires separate tuning.
3.3 The Fix: Muon Split, Lowering the Orthogonalization Granularity to the Head Level
The idea behind Muon Split is this: instead of orthogonalizing the entire projection matrix, split the weights by attention head and orthogonalize each head’s sub-matrix separately. Taking the Query projection as an example, here is illustrative code (it only shows the splitting granularity and is not a real implementation):
def split_orthonormalize(W, num_heads, eps=1e-7):# W: [num_heads * head_dim, hidden], query projection weights split by attention headhead_dim = W.shape[0] // num_headsparts = []for h in range(num_heads):sub = W[h * head_dim:(h + 1) * head_dim, :]# Illustrative: Newton-Schulz iterative approximation of orthogonalization# (in practice, combine with momentum and damping)Q = sub @ sub.TU, _, V = torch.linalg.svd(Q + eps * torch.eye(head_dim, device=W.device))parts.append(U @ V @ sub)return torch.cat(parts, dim=0)
Confining orthogonalization within each head amounts to explicitly guaranteeing that the update subspaces of different heads do not interfere with each other, aligning the optimizer’s geometric properties with the attention structure.
The significance of this finding goes beyond a single model: new optimizers do not work in a vacuum—there is an implicit coupling between them and the model structure. The engineering recommendation is that whenever you introduce a new optimizer, run at least one set of compute-matched ablation experiments on both the target architecture and a plain baseline architecture, and simultaneously check gradient norms, update norms, and the weight distribution of each layer.
4. Architecture Change 2: Multi-Token Prediction
4.1 Mechanism
Multi-token prediction (MTP) is a training method proposed by Meta in 2024: in addition to predicting the next token, the model adds prediction modules that let it continue predicting several future tokens based on the current representation.
At inference time, these extra prediction heads can act as a “draft model”: generate several candidate tokens at once, then have the main model verify them in parallel. Accepted tokens are taken directly, and rejected ones are rolled back and resampled. Because the output distribution is strictly identical to the original model, this approach is usually called self-speculative decoding.
4.2 Value
- Training side: a single forward pass carries more supervision signal, effectively telling the model that “the current representation must prepare for multiple future positions,” pushing it to learn longer-range dependencies and planning abilities and improving the data efficiency per token.
- Inference side: mathematically equivalent to standard decoding, and in scenarios with strong predictability such as code completion and structured output, the acceptance rate is usually higher and end-to-end latency lower.
- Engineering side: the training objective and inference acceleration share the same set of weights, so there is no need to maintain a separate draft model.
Keep in mind that the MTP module itself consumes parameters and memory. Whether to enable it at inference and what draft length to use should both be measured per scenario.
5. Architecture Change 3: Sparse Attention
5.1 Motivation
In standard full attention, every token interacts with all previous tokens, so computation grows quadratically with sequence length. A seemingly paradoxical fact is that dense attention, after pre-training on tens of billions of tokens, has already trained far more attention capacity than needed; many attention heads heavily connect to “irrelevant” tokens that have nothing to do with the decoding task itself.
If attention could be constrained to focus only on “relevant” parts already during pre-training, it would surely reach the same effect with fewer tokens.
5.2 Method
The report’s approach is a kind of trainable sparsity: a lightweight selection module dynamically picks a small number of the most important keys for each query to compute attention over, and an auxiliary loss guides the model to learn “where to look.”
5.3 Effect and Interpretation
The report provides a comparison: sparse attention matched, with only about 20B tokens, the effect that a certain dense-attention baseline reached at 943.7B tokens—a gap of nearly 47x.
The key is to understand this conclusion fairly: the comparison is about “the training efficiency of the attention mechanism,” not the model’s overall capability; the baseline is a dense model trained under a specific token budget, while the overall model still needs 28.5T tokens to complete its full training.
The dense baseline’s disadvantage comes mainly from wasted computation: most of its compute is spent on keys unrelated to the current token. Sparse attention brings three benefits: higher information density per unit of compute; lower KV-cache and attention overhead in long-context scenarios; and fewer redundant connections, acting like regularization.
5.4 Engineering Implementation Points
- Kernel support is a prerequisite. You need to write efficient fused kernels for “select + sparse attention” and connect them with inference mechanisms such as KV-cache management and paged caching.
- Sparsity must be traded off per scenario. The higher the sparsity, the more long context you save, but the greater the risk of missing key information, so it usually must be set based on a recall-rate evaluation for the specific task.
- Pair it with long-context extension. In the late training stage, dedicated long-context data and positional-encoding extrapolation strategies are still required; sparse attention does not automatically solve length generalization.
6. Training Paradigm: A Fully Asynchronous Agent RL Training Framework
6.1 Synchronous RL: GPUs “Waiting for the Slowest Trajectory”
In the classic “generate → score → update” synchronous loop, each step must wait for all trajectories in a batch to finish generating before moving on to scoring and parameter updates.
For Agent RL, the problem is amplified further: within the same batch, some tasks need only one round of tool calls while others require multiple rounds of retrieval, coding, and execution, so trajectory lengths can differ by an order of magnitude; long-tail cases such as tool timeouts and retries are also unavoidable. As a result, most compute sits idle while waiting for the longest trajectory, and cluster utilization deteriorates rapidly as Agent task complexity rises.
6.2 Fully Asynchronous: Decoupling Generation from Training
The fully asynchronous idea is to let the generation side continuously produce trajectories while the training side continuously consumes them: once the samples needed for a training step are collected, parameters update immediately. This introduces a “sample staleness” problem—the trajectories used for an update may have been generated several parameter versions ago. This is similar to the “stale gradients” commonly seen when training parameter servers adopt asynchronous updates in distributed systems.
RL is far more sensitive to distribution shift than supervised learning, so engineering requires a full set of stabilization mechanisms:
| Mechanism | Role |
|---|---|
| Importance-sampling ratios and clipping | Limit how far the new policy drifts from the sampling policy, preventing a single update from going too far |
| Reference-model KL constraint | Keep the policy from drifting too far from the reference model, preventing capability collapse |
| Parameter version management | Record the weight version for each trajectory, weighting by version or discarding samples that are too old |
| Asynchronous reward scoring | Reward models or rule validators deployed independently, running in parallel with the generation and training pipelines |
| Observability | Monitor staleness distribution, clipping trigger ratio, entropy, and reward and KL curves |
Below is illustrative pseudo-code (it only describes the flow, not real interfaces):
# Pseudo-code: fully asynchronous Agent RL training loop# Generation side (rollout worker): continuously produces trajectoriesdef rollout_worker(inbox, outbox, weights):for task in inbox: # keep pulling tasks, don't wait for the rest of the batchtraj = agent.rollout(task, weights.latest(), max_steps=32)outbox.put(Experience(traj, task, policy_version=weights.version()))# Reward side: runs in parallel with generation and trainingdef verifier_worker(inbox, outbox):for exp in inbox:exp.reward = evaluate(exp.task, exp.traj) # rule check / reward modeloutbox.put(exp)# Training side: update as soon as a batch is readydef trainer_worker(inbox, weights):while not converged:batch = collect_min_batch(inbox) # start once min batch size is reachedbatch = filter_by_staleness(batch, max_version_gap=2)loss = ppo_loss_with_clip(batch) - beta * kl_penalty(batch)optimizer.step(loss)weights.publish(weights.version() + 1) # new version becomes visible to the generation side
What needs to be stressed is that asynchrony improves throughput at the cost of harder tuning. Parameters such as the clipping range, the KL coefficient, and the maximum stale version count must be tuned jointly: too loose leads to unstable training, too strict degrades to near-synchronous, and the throughput advantage disappears.
7. Engineering Foundation: Self-Developed RL Infrastructure
The report discloses that the team built its own RL infrastructure to support the training above. Why build their own? The fundamental reason is that RL training is the most engineering-complex part of the entire training pipeline—it simultaneously involves three kinds of workloads: distributed training, online inference, and reward services:
- Mismatched abstractions in general-purpose frameworks. Many general-purpose frameworks are designed for supervised training and struggle to natively express Agent-RL-specific flows such as “multi-turn tool calls, asynchronous rewards, and hot parameter updates.”
- Coupled optimization of training and inference. Efficient weight synchronization, memory reuse, and pipeline overlap are needed between the two engines, and off-the-shelf assemblies make global optimization difficult.
- Fault tolerance for long-horizon tasks. RL training often runs for weeks and must support fast fault recovery, automatic checkpointing, and the filtering and retrying of invalid trajectories.
The direct benefits of self-developed infrastructure are: the asynchronous strategy can run stably across the whole cluster, long-tail trajectories no longer block overall progress, and training compute is utilized more fully.
8. Extended Thinking: Building Efficient Training Systems in the Cloud
This technical approach offers plenty of insight for enterprises building large-model capabilities. If you want to implement it in the cloud, the following general-purpose capabilities are worth planning ahead of time:
- Heterogeneous compute pooling and elastic scaling: training, inference, and RL generation have completely different compute-demand profiles, so managing resources with distinct elastic policies for each can markedly improve overall utilization.
- High-performance parallel file and object storage: to support the throughput of massive training data and the fast reads and writes of checkpoints.
- Message queues and task scheduling: to carry task distribution, reward scoring, and result collection in asynchronous RL.
- An observability system: unified log services, metric monitoring, and alerting are the foundation for spotting training anomalies and locating performance bottlenecks.
- Security isolation and access control: when multiple teams share a cluster, do network isolation, quota management, and key management well.
9. Summary: From “Piling Up Parameters” to “Competing on Efficiency”
The core signal this technical report sends is clear: open-source large models are shifting from a scale-driven to an efficiency-driven design. Let’s distill the key lessons into four points:
- Attention efficiency is a system-level problem. MLA reduces the KV cache and sparse attention reduces training and inference costs, and the two can be combined; but every time you introduce a new structure, you must re-validate its compatibility with optimizers, kernel libraries, and inference engines.
- Optimizers and architecture must be co-tuned. The lesson of Muon Split shows that the geometric properties of parameter updates must align with the model structure; a simple “whole-matrix orthogonalization” is not necessarily optimal.
- The bottleneck of Agent RL is on the engineering side. The compute waste of synchronous frameworks amplifies sharply with task complexity, and asynchrony is a necessary direction—but it needs a full set of stabilization mechanisms as a backstop.
- Infrastructure sets the ceiling. Once model capability reaches a certain level, whether you can keep iterating depends more on the throughput, stability, and observability of your RL infrastructure.
For developers and enterprises, the pragmatic path is: use third-party evaluations as a reference and your own business evaluations as the standard, and gradually fold proven techniques such as sparse attention, multi-token prediction, and asynchronous RL into your own model R&D or continued-training pipeline. At the same time, lay the groundwork early in cloud infrastructure for compute scheduling, storage, and monitoring. Only then can the technological dividend represented by “the best open-source model” truly be converted into product competitiveness.
