Open-Source LLM Architecture Explained: MoE, MLA, Multi-Token Prediction, and Async RL Training
A detailed walkthrough of a new open-source large language model's technical report, organized around three transferable themes rather than any single benchmark score. It covers the model's Mixture-of-Experts design (744B total parameters with 40B activated per token), three architectural changes—Multi-head Latent Attention paired with Muon Split, multi-token prediction, and trainable sparse attention—and the fully asynchronous agent-RL training stack. It also includes a practical checklist for deployment, inference-framework support, evaluation, and post-training planning, aimed at developers and enterprises evaluating open-source models.
Introduction: Why This Technical Report Rewards a Close Read
A recent release from an open-source LLM series came with a complete technical report, and the community quickly fell into a round of “page-by-page close reading.” Earlier versions of the series have already been adopted widely by developers and enterprises, so every adjustment to the architecture and training method in a new release tends to be read as an industry signal. One well-known international investment firm even called it “the best open-source model available today.” Evaluations like that carry some subjectivity and are best combined with third-party benchmarks and your own business testing.
This article does not revolve around any specific product. Instead, it pulls out three main threads from the report that have broad, transferable value:
- Model architecture: balancing massive parameter scale against low inference activation, while using attention- and optimizer-level changes to control long-context cost;
- Training method: moving from dense pre-training to “pre-training + mid-training,” and then to reinforcement learning aimed at agent tasks;
- Training system: the asynchronous scheduling and in-house infrastructure that support the full RL pipeline.
For developers, understanding these three threads is more valuable than memorizing any single score.
1. Overall Specs: The Fundamentals of MoE Sparse Activation
According to public information, the new open-source model uses a Mixture of Experts (MoE) architecture:
- 744B total parameters, with only 40B activated per token, spread across 256 experts and 80 layers;
- roughly 28.5T training tokens, of which about 27T are pre-training and about 1.5T are mid-training.
The previous generation in the same series had 355B total parameters and 32B activated parameters. By comparison, this generation more than doubles the total parameters while adding only 8B activated parameters.
The core idea of MoE is straightforward: each layer holds multiple expert networks (typically the feed-forward sublayers), and a gating network selects a small number of experts to participate in the computation for each token. This produces two independent metrics:
- Total parameters determine the model’s knowledge capacity;
- Activated parameters determine the average computation per token, which intuitively approximates the inference cost of a dense model of equivalent scale.
The key trade-off in this generation is: gaining more than a doubling of total-parameter capacity for an activation overhead increase of about 1.125x (40B/32B). The math is not hard to follow. Let total parameters be P, activated parameters be A, and the number of experts be N; a rough estimate gives A ≈ P/N, so:
- with activated parameters fixed, parameter scale and expert count are roughly proportional;
- with the expert count fixed, doubling total parameters raises activated parameters by roughly the same proportion.
This generation follows the first approach more closely: it expands both total parameters and expert count while keeping the activation ratio around 5.4% (40B/744B).
The direct impact on the deployment side is a decoupling of memory and compute:
- all expert weights must remain resident in VRAM (or be stored quantized at lower precision), so the memory threshold does not drop meaningfully just because only 40B parameters are activated;
- actual compute consumption, however, stays close to that of a 40B-class dense model.
For enterprises that want to deploy locally, this shifts the focus of hardware selection from “is there enough compute?” to “will it fit in VRAM, and are expert routing and batch scheduling efficient?”
From public information, the model scored 50 on a third-party composite intelligence index, ranking first among open-source models and second overall only to the top overseas closed-source models. In a third-party LLM arena human blind test, it ranked first among open-source models on both the text and code tracks, placing it in the same tier as the leading overseas closed-source models overall. Two caveats are worth stressing:
- these leaderboards measure general capability and human preference;
- whether a leaderboard score translates into business value ultimately depends on performance against your own evaluation set.
On the data side, beyond the roughly 27T pre-training tokens, the report also schedules about 1.5T tokens of mid-training. This stage usually sits between pre-training and post-training and is used to reinforce specific capabilities, such as:
- long context;
- high-density reasoning tasks like code and math;
- high-quality curated corpora.
The methodology it signals is that competition among top models has shifted from “whose pre-training corpus is larger” to “who can apply high-quality data more precisely at the right stage.”
2. Architecture Evolution: Three Key Changes, One by One
Compared with the previous generation, this series introduces three architectural changes: MLA + Muon Split, multi-token prediction (MTP), and sparse attention. Their shared goal is to lower training and inference cost in long-sequence scenarios without sacrificing model capability.
2.1 MLA: Compressing the KV Cache with Low-Rank Latents
Multi-head Latent Attention (MLA) was first proposed and popularized by open-source models in the industry. Its core idea is to compress each token’s Key/Value into a low-dimensional latent vector, cache only these latent vectors, and reconstruct each head’s Key and Value through an up-projection matrix when needed.
Standard multi-head attention must cache the full K and V of all historical tokens during autoregressive decoding. That cache overhead is proportional to “layers × KV heads × head dimension × sequence length,” and in long-context, high-concurrency inference it is often the main consumer of VRAM and memory bandwidth.
MLA’s gains come from a low-rank assumption:
- the K/V of different attention heads are highly correlated in representation space;
- a single shared low-dimensional latent representation can approximate their reconstruction.
The resulting benefits include a much smaller KV cache, fewer VRAM reads during attention computation, higher long-context decoding throughput, and more concurrent sessions per card.
The team also honestly recorded a counterintuitive engineering finding. MLA and traditional Grouped-Query Attention (GQA) are not in a purely progressive relationship:
- during training, the team found that pairing a matrix-orthogonalization optimizer with MLA actually underperformed the structurally simpler 8-group GQA scheme;
- GQA lets multiple query heads share one set of K/V—an intuitive design with a mature inference kernel;
- MLA compresses more aggressively but has a coupling problem with the optimizer.
The lesson for anyone selecting an architecture: a mechanism’s benefit cannot be evaluated in isolation from the training algorithm. The match between model architecture and optimizer is itself a variable that must be investigated systematically during experimentation.
2.2 Muon Split: Aligning the Orthogonalization Optimizer with the Attention Structure
The optimizer mechanism. Recently, matrix-orthogonalization-based optimizers (commonly referred to as Muon-class optimizers) have drawn considerable attention in large-model pre-training. For a two-dimensional weight matrix, they:
- accumulate gradients following the momentum approach;
- approximately orthogonalize the momentum matrix—usually via Newton-iteration-like matrix operations—so that gains across directions trend toward being balanced;
- multiply by a scaling factor related to the matrix size before updating the weights.
Intuitively, if you treat the update as a “direction” in the row and column spaces of the matrix, orthogonalization makes the update magnitude along each direction roughly consistent, which improves the conditioning of the optimization. Compared with element-wise adaptive methods, it makes fuller use of the structural information in the weight matrix.
The name Muon does relate to momentum, but “splitting by head” has nothing to do with the optimizer’s name. Muon Split is a structural-level improvement: instead of orthogonalizing one whole projection matrix, it processes each attention head separately.
The problem. In MLA’s query, Key, and Value up-projections, each head’s projection sub-matrix is concatenated into one block for training. If that whole block is treated as a single object for orthogonalization, the sub-matrices are forced to share the same set of orthogonal directions:
- a head with a large gradient interferes with other heads along the shared directions;
- the “personalized” update directions each head forms through its semantic division of labor get flattened out.
The team therefore observed the earlier phenomenon where “MLA + optimizer” underperformed 8-group GQA.
The improvement. Muon Split cuts the overall projection matrix by attention head, orthogonalizes and updates the sub-matrices separately, then stitches them back into the full matrix:
- each head gets a relatively independent optimization geometry during the update;
- this matches the “parallel multi-subspace” structural assumption of multi-head attention;
- it requires no change to the forward structure, so inference is identical to ordinary MLA—zero cost on the deployment side.
def split_muon_update(proj, n_heads, head_dim, lr):"""Split the projection matrix by attention head and apply amatrix-orthogonalization update to each head separately.proj: weight of shape (n_heads * head_dim, in_dim), gradient includedNote: illustrative code; coefficients and iteration count must betuned to the matrix scale."""patches, weight = proj.grad, proj.datafor i in range(n_heads):s, e = i * head_dim, (i + 1) * head_dimX = patches[s:e] / (patches[s:e].norm() + 1e-7)for _ in range(5): # approximate orthogonalization: Newton iterationX = 1.5 * X - 0.5 * X @ X.T @ X# scale by row/column size to offset step-size differences from matrix shapescale = max(1.0, (X.shape[0] / X.shape[1]) ** 0.5)weight[s:e] -= lr * scale * X
Notably, changes that exist only during training usually carry the lowest cost. Because they do not touch the forward computation and only alter how parameters are updated, they add no inference overhead and do not affect compatibility with existing deployment tooling.
2.3 Multi-Token Prediction (MTP): A Denser Training Signal
The standard training objective for a language model is to predict the next token given the preceding text—one position at a time. Multi-token prediction adds auxiliary prediction heads on top of this, so the model predicts several future tokens at each position simultaneously and folds those auxiliary predictions into the training objective.
Its value shows up on three levels:
- A denser training signal: the supervision signal per position multiplies, which helps the model learn longer-range, more structured dependencies;
- Fuller representation learning: being forced to “plan” subsequent content ahead of time helps the model capture local patterns such as syntactic structure, code blocks, and reasoning steps;
- Reusable for inference acceleration: the trained MTP heads can act as a draft model for speculative decoding—generating multiple candidate tokens in one forward pass, then having the main model verify them in parallel. The actual benefit depends on how well the inference framework supports this.
A few cautions are worth noting:
- MTP heads should stay consistent with pre-training during fine-tuning to avoid capability degradation;
- if used for speculative decoding, watch the output-distribution consistency between the draft model and the main model.
2.4 Sparse Attention: The Efficiency Lever for Long Sequences
Standard attention’s computation and memory access grow quadratically with sequence length. In ultra-long context inputs, long chain-of-thought reasoning, and multi-turn agent interactions, this overhead becomes the primary bottleneck for throughput and cost. Sparse attention lets each query attend only to a carefully selected subset of historical positions—for example, nearby tokens within a local window plus a few key tokens chosen by scoring.
These changes can share the same parameter set as dense attention and switch directly via a sparse mask:
- Trainable sparse: compared with “dense pre-training followed by post-hoc sparsification,” trainable sparsity lets the model adapt to the sparse pattern from the start;
- Inference benefit: reductions in both attention FLOPs and KV reads.
According to public information, the series reached the effect of a dense baseline trained on roughly 943.7B tokens after only about 20B tokens of sparsification continued training. By that measure, the required training compute is roughly 1/47 of the dense baseline (a rough order-of-magnitude reference only; the actual figure depends on the experimental setup). A gap of that magnitude means sparsification is not merely an engineering optimization but a structural design that may reshape the model’s capability growth curve.
Of course, such comparisons depend heavily on specific experimental conditions. Teams making this choice should re-check the following dimensions:
- how much their own tasks depend on precise long-range retrieval;
- how sparse patterns perform on strongly structured corpora such as code;
- how well the inference framework supports sparse masks and the corresponding kernels.
3. Training System: Why Async Is a Required Course in the RL Era
Another highlight that the developer community returned to repeatedly is the fully asynchronous Agent RL training framework and the in-house RL infrastructure behind it. It is two sides of the same coin as the architectural changes above.
Sparse attention, MLA, and multi-token prediction mainly solve the problem of making each unit of compute “worth more”; the asynchronous training framework solves a different problem: whether precious GPU time is being wasted.
Why RL training inherently needs to be asynchronous. In pre-training, data is pre-prepared as fixed-length sequences, so training can stay highly synchronized and deterministic. Reinforcement learning is different: it alternates between two phases with very different rhythms:
- Sampling (rollout): agents interact with an environment over multiple turns to produce trajectories. The environment might be a code executor, unit tests, a terminal, or a browser, and a single task can take anywhere from a few seconds to several minutes, with very high variance;
- Training (optimization): computing the loss on the sampled results and updating parameters, with relatively predictable per-step durations.
Under a strict synchronous scheme—waiting for all sampling to finish before a unified update each round—the overall pace gets dragged down by the slowest sample. The more long-tail tasks there are, the more GPUs sit idle. Asynchrony lets sampling and training run continuously in parallel across separate resource pools, putting idle compute back to work.
# Illustration: a fully asynchronous RL training loop (networking and engineering details omitted)weight_version = 0queue = ExperienceQueue() # bounded experience queue: limits policy stalenessenvs = [CodeEnv(task) for task in task_pool] # environments: code execution, tests, etc.def rollout_worker(): # sampling side: inference cluster + environment clusterver = weight_versiontraj = envs.sample().run(policy=lambda s: infer(s, weights[ver]))queue.put(Trajectory(traj, policy_version=ver))def trainer(): # training side: training clusterglobal weight_versionwhile not done():batch = queue.get()loss = ppo_loss(batch, clip=True) # importance sampling + clipping to constrain new/old policy deviationloss.backward()optimizer.step()if step % k == 0:weight_version += 1broadcast_weights(weight_version) # asynchronously push new weights
This fully asynchronous design introduces two new engineering challenges.
First is the data staleness (off-policy) problem. A sampled trajectory may come from a policy several versions old. Using it directly for the policy gradient introduces bias, so techniques such as truncated importance sampling and clipping are typically used to constrain how far the new and old policies diverge. Common approaches include:
- setting a maximum allowed policy version gap;
- defining a capacity and eviction policy for the bounded experience queue.
Second is system reliability and observability. Sampling involves huge numbers of environment instances and tool calls, so you must solve failure retries, timeout control, and sandbox isolation. The training side needs:
- high-frequency metric monitoring and alerting;
- fine-grained logging to help locate reward anomalies or training crashes.
Large-scale training jobs often run for weeks or even months, which raises the bar for task scheduling, fault tolerance, and checkpoint-and-resume capability. In such settings, general-purpose infrastructure—containerized scheduling, object storage, message queues, logging services, and monitoring/alerting—can significantly reduce the operational complexity of building a training system in-house.
As for why teams choose to build their own RL infrastructure: the prevailing industry experience is that most post-training frameworks for large models are designed around pre-training or supervised fine-tuning, and when facing an RL scenario of “high-concurrency long-tail sampling + high-frequency parameter sync + multi-role resource coordination,” they often need substantial customization. The core value of a self-built RL infrastructure is to:
- decouple the inference engine, the training engine, and the scheduling layer;
- enable asynchronous parameter delivery and elastic scaling;
- deliver end-to-end observability.
For teams planning to build out post-training capabilities, this is a direction worth evaluating early.
4. A Practical Checklist for Developers
The value of a technical report ultimately lands on engineering decisions. The following checklist can be used for selection, deployment, and evaluation:
Do a “full” VRAM budget calculation first.
- MoE inference must load all expert weights;
- when converting weight footprint by precision, also reserve space for the KV cache, activations, and framework overhead;
- do not estimate hardware needs from activated parameters alone.
Confirm the inference framework supports MLA and MTP.
- MLA typically depends on custom support for latent-vector caching;
- MTP can enable speculative decoding but requires framework cooperation;
- without that support, the associated efficiency gains are hard to realize.
Re-check sparse-attention benefits against your own data.
- tasks such as long-document QA, repository-level code understanding, and multi-turn agent trajectories vary widely in their sensitivity to long-range dependencies;
- treat benchmark throughput figures as reference only.
Validate on your own evaluation set.
- leaderboard scores are not business metrics;
- cover dimensions such as code generation and repair, math reasoning, long-document understanding, and tool/function calling;
- follow the report’s human blind-test approach to reduce automated-evaluation bias.
Plan the post-training pipeline ahead of time.
- if you intend to run reinforcement-learning post-training, treating asynchronous sampling and training as separate should be a first-class concern in the architecture design;
- for tasks like coding agents, rewards mostly come from verifiable signals such as test cases, making reward reliability easier to control than pre-training corpus cleaning.
Pay attention to coordination between the training system and model architecture.
- this generation’s “optimizer split by attention head” approach shows that architecture changes, optimizer choice, and parallelism strategy must be evaluated together;
- the model, framework, and optimizer should be treated as one integrated system.
Use the report as a guide, and avoid over-inferring.
- an open-source technical report usually discloses methods and ideas; specific implementation details, hyperparameters, and recipes are not necessarily fully public;
- before reproducing or building on it, validate assumptions with small-scale experiments first;
- community feedback can help cross-confirm findings.
Conclusion: The Gap Between Open and Closed Source Is Being Redefined
Reading through this technical report, a clear line of evolution emerges:
- Parameter scale achieves a “decoupling of capacity and compute” through MoE sparse activation;
- coordinated changes to attention and the optimizer push long-sequence training and inference costs lower;
- multi-token prediction makes the training signal denser;
- trainable sparse attention reshapes the growth curve of long-context capability;
- the fully asynchronous reinforcement-learning framework and in-house infrastructure move a model that “can solve problems” toward an agent stage where it “can get things done.”
Together, these designs show that the competitiveness of a top open-source model is no longer just “bigger parameters and more data,” but the result of coordinated optimization across model architecture, training algorithms, and the training system.
For developers and enterprises, the rational stance is:
- neither to be blindly optimistic over an individual leaderboard ranking, nor to underestimate the open-source ecosystem’s pace of iteration because of the gap to closed-source models;
- to focus on architectural ideas and engineering methods, and to run independent evaluation and validation against your own business scenarios.
Only then can these technical dividends genuinely be turned into product competitiveness.
