Open-Source 700B-Class MoE Architecture Breakdown: Sparse Attention, MLA, MTP, and Asynchronous Agent RL

    This article breaks down the architecture of an open-source MoE foundation model reported with 744B total parameters and 40B active parameters. It explains how the model separates model capacity from per-token computation, and how multi-head latent attention, per-head Muon optimization, multi-token prediction, and dynamic sparse attention work together. It also covers fully asynchronous Agent RL, RL infrastructure, and practical guidance for model selection, deployment, long-context handling, and post-training.

    Open-Source MoE Foundation Model Breakdown: From Parameter Growth to Asynchronous Agent RL

    Introduction: Efficiency Becomes the Second Competitive Line

    Recent open-source foundation models have been getting closer to top closed-source models on public leaderboards. One open-source foundation model release included a technical report of roughly 40 pages covering MoE architecture, attention compression, multi-token prediction, sparse attention, and asynchronous Agent reinforcement learning.

    To avoid confusion in terminology, this article uses the following naming: the model is a 700B-total-parameter-class MoE model. According to the technical report, the total parameter count is 744B, and the number of activated parameters is 40B.

    The focus of this article is not simply to restate parameter scale. Instead, it examines a more practical engineering thread: when compute, GPU memory, and training time are all limited, how should limited resources be spent on computation that actually creates value?

    During training, dynamic sparse attention compresses ineffective computation. During post-training, an asynchronous framework removes idle waiting time. During inference, multi-head latent attention compresses the KV cache. Together, these choices point to a broader conclusion: efficiency itself is becoming part of model capability.

    This article covers architecture principles, training tradeoffs, and engineering practices. It does not go into low-level operator implementation, a clause-by-clause review of licenses, or service-level commitments for specific business use cases.

    1. Model Positioning: 700B Total Parameters, 40B Active Parameters

    The base model uses a Mixture of Experts, or MoE, architecture. During the forward pass, a router selects a small number of experts for each token, while the remaining experts stay inactive. Compared with a dense model, the core value of MoE is that it decouples “model capacity” from “compute per token.”

    Dimension Previous generation Current generation
    Total parameters 355B 744B, about 2.1x
    Activated parameters 32B 40B, about 1.25x
    Number of experts — 256
    Number of network layers — 80
    Training tokens 23T 28.5T: 27T pre-training + 1.5T mid-training

    These numbers show a key tradeoff: total parameters grow much faster than activated parameters. Total parameters determine the amount of knowledge the model can hold, while activated parameters determine the inference compute cost per token. Capacity expands, but the cost per token increases by only about one quarter. This is one of the main attractions of the MoE architecture.

    However, MoE is not free of cost. All expert weights still need to remain resident in memory during inference. Deployment therefore often relies on tensor parallelism and expert parallelism, splitting different experts across multiple machines and GPUs.

    This leads to two practical judgments:

    • When estimating compute cost, focus mainly on activated parameters.
    • When planning memory capacity and deployment, focus mainly on total parameters.

    For training data, the 28.5T tokens consist of about 27T tokens for pre-training and 1.5T tokens for mid-training.

    Pre-training emphasizes breadth: corpus scale, diversity, and quality filtering. Mid-training is more capability-oriented, for example with high-quality code, mathematical reasoning data, long-context corpora, and synthetic data. In engineering practice, mid-training can also be combined with context extension and sparse-attention adaptation.

    In other words, the data mixture often matters more than the raw token count when shaping the final capability profile of a model.

    For evaluation, at the time the report was published, the model scored 50 on a third-party composite intelligence index version 4.0, becoming the first open-source model under that index to pass the 50-point threshold. In text and code arenas based on anonymous pairwise comparisons and human preference votes, it also ranked first among open-source models and was in the same tier as leading closed-source models outside its home market.

    These conclusions are limited to the leaderboard versions at the time the report was published. Leaderboards are dynamic. They are more useful for initial screening of candidate models. The final decision should still return to the team’s own business evaluation sets.

    2. Architecture Changes: Key Designs and Their Interaction

    The architecture changes described in the report are concentrated in several areas: multi-head latent attention with per-head orthogonalized optimization, multi-token prediction, and dynamic sparse attention. They affect attention representation, optimization, prediction targets, and long-sequence computation. They are not isolated tricks.

    1. Multi-Head Latent Attention: Compressing the KV Cache

    Transformer inference is usually divided into two stages.

    In the prefill stage, the input is processed in parallel. This stage is compute-intensive. In the decode stage, tokens are generated one by one, and performance is more constrained by memory bandwidth. The key and value of every generated token need to be cached. As context length grows, the KV cache quickly consumes memory and slows down decoding.

    The core idea of multi-head latent attention, or MLA, is to project keys and values into a low-dimensional latent representation. During inference, the model mainly caches the compressed representation instead of caching the full multi-head key/value tensors.

    The idea can be illustrated as follows:

    1. # Illustrative pseudocode: KV compression idea in MLA
    2. # x: hidden state of the current token; h: hidden dimension; c: latent dimension
    3. c = down_proj(x) # Only c is cached during inference
    4. k_nope = c @ W_up_k # Key without positional information
    5. v = c @ W_up_v # Value
    6. k_rope = apply_rope(k_rope_proj(x)) # Position-dependent key is generated separately
    7. cache.append(c, k_rope) # Cache c and k_rope for each token

    The most important detail here is RoPE decoupling. Rotary position embeddings need to be applied to keys, while the latent vector is closer to a position-independent shared representation. If RoPE were applied directly to the latent vector, it would be difficult for each head to independently decide whether to carry positional information.

    Therefore, MLA separately generates and caches a position-dependent key. This preserves positional encoding capability and also leaves room for adding sparse attention later.

    MLA also has clear costs:

    • Each token incurs extra down-projection and up-projection computation.
    • Whether low-rank compression loses representational capacity must be compensated through training.

    MLA therefore cannot be understood only from the inference side. It must be designed together with the optimizer and training strategy.

    2. Muon Split: Per-Head Orthogonalized Optimization

    Current mainstream large-model training still often uses adaptive optimizers, which maintain first-order and second-order momentum for each parameter. Muon-type matrix orthogonalization optimizers take a different approach: they maintain momentum for weight matrices, orthogonalize the momentum matrix, make singular values more balanced, and then update the weights with a global scale. This can be understood as finding a more stable descent direction under a spectral-norm-like constraint.

    Directly performing SVD on the momentum matrix is expensive. In practice, a common approach is to approximate the orthogonalization result with five Newton-Schulz iterations. The process is mainly matrix multiplication, which is suitable for large-scale parallelism.

    However, combining this with MLA creates a conflict.

    In multi-head attention, the Q and K projections are often stored as large fused matrices containing all heads. If orthogonalization is applied to the entire matrix, a larger gradient from one head may dominate the update direction of the whole matrix. The low-rank structure of MLA can further amplify sensitivity differences among heads.

    In practice, applying Muon directly with MLA performed worse than the simpler GQA-8 baseline.

    The solution is Muon Split: split the fused projection matrices by attention head, maintain and orthogonalize momentum separately for each head, and then merge the updates.

    1. # Illustrative pseudocode: orthogonalizing separately after splitting by head
    2. # Preconditions: separate weight decay, gradient clipping, and learning-rate schedule;
    3. # Newton-Schulz iteration count can vary by training stage;
    4. # small per-head matrices can run in parallel to reduce synchronization overhead.
    5. def muon_split(W, n_heads, d_head, momentum, lr):
    6. upd = []
    7. for i in range(n_heads):
    8. W_i = W[i * d_head:(i + 1) * d_head, :]
    9. momentum[i] = beta * momentum[i] + (1 - beta) * W_i.grad
    10. upd.append(newton_schulz(momentum[i]))
    11. return W - lr * concat(upd, dim=0)

    In essence, this approach uses a structural prior to constrain optimization: attention uses heads as its basic computation unit, so optimization should also use heads as its unit.

    It also brings an engineering benefit: the number of Newton-Schulz iterations can be adjusted adaptively. Early in training, gradients are larger, so more iterations can be used. Later, updates become more stable, so fewer iterations can be used. This can reduce optimizer overhead with almost no negative impact on convergence.

    3. Multi-Token Prediction: Learning One Step Ahead at Every Position

    Standard language model training usually predicts only the next token, so each position provides one supervision signal. Multi-token prediction, or MTP, asks the model to predict the next second and third tokens as well, forcing hidden states to carry longer-range planning information.

    In a typical structure, the backbone model outputs hidden states. Several lightweight MTP modules are attached after it. Each module receives the backbone output or the output of the previous MTP module, as well as the embedding of the true next token. After normalization and a small number of Transformer blocks, it predicts a farther position. The training loss is usually a weighted sum of the main prediction loss and the MTP losses at different depths.

    1. # Illustrative pseudocode: one MTP module depth
    2. h_mtp = RMSNorm(concat(h_i, proj(t_next_emb)))
    3. for block in mtp_blocks:
    4. h_mtp = block(h_mtp)
    5. logits = shared_head(h_mtp)

    The benefit of MTP is higher training signal density. It is especially useful for long-context tasks, multi-step reasoning, and tasks that require early planning.

    The cost is increased compute per token during training. During inference, the backbone is usually retained while MTP modules can be discarded. If retained, they can also serve as a built-in draft model for speculative decoding, reducing generation latency.

    It is worth noting that MTP increases the attention computation burden at each position. The longer the context, the more noticeable this overhead becomes. This provides motivation for introducing dynamic sparse attention.

    The three changes are not isolated techniques. They pull on one another and form an integrated design.

    4. Dynamic Sparse Attention: Computing Only the Relevant Tokens

    Attention computation grows quadratically with context length, and this is also the main bottleneck in long-context inference.

    Static sparse attention patterns are fixed and simple to implement, but they struggle to cover dynamic dependencies such as “skip several rows to look back at a variable name.” Linear attention and state-space models have lower complexity, but they may lose accuracy in exact retrieval and long-range precise recall tasks.

    Dynamic sparse attention, or DSA, takes a different approach: it dynamically decides where to look. For each query token, the model first performs a lightweight relevance score over all historical tokens and selects only the top-k tokens to participate in attention computation.

    1. Query q_t
    2. ├─ Lightweight scorer: compute relevance against historical tokens
    3. ├─ Top-k selection: keep the k tokens with the highest scores
    4. └─ MLA attention: compute and output only over selected tokens

    There are four key design points:

    1. The scorer must be lightweight enough. For example, it can use a small number of index key dimensions, low-precision computation, or shared indexing results across multiple query heads. The scoring cost must be much lower than the attention cost it removes.

    2. Positional encoding cannot be dropped. Consistent with MLA, position-dependent keys need to be handled separately to avoid breaking RoPE logic after sparsification.

    3. The training strategy must support it. Typically, the model is first trained on shorter contexts, and sparse attention is introduced at the long-context stage so that it gradually adapts to inputs where only k tokens are attended to.

    4. Engineering requirements are high. Top-k selection has a data dependency with the subsequent attention computation. Custom operators may be needed; otherwise, scheduling overhead can offset the theoretical benefit.

    The report states that, in the experimental setup, DSA was trained with about 20B tokens and matched the performance of a dense approach using 943.7B tokens. More precisely, the report describes this as roughly a 47x reduction in the number of training tokens needed to achieve comparable quality.

    This number comes from the experimental setup described in the report and indicates a data-efficiency improvement, not a 47x improvement in inference speed. Actual inference speedup depends on the k value, context length, operator implementation, and hardware utilization.

    Theoretically, DSA can reduce attention computation from about O(n^2 · d) to about O(n · k · d). When n is much larger than k, this is close to linear.

    The KV cache still scales with sequence length, but after MLA compression, the cost per unit is significantly lower. In short, DSA reduces “how much is computed,” while MLA reduces “how much is stored.” They are natural partners.

    3. Synergy: Individual Optimizations Should Not Be Evaluated in Isolation

    Change Layer affected Main benefit Main cost
    MLA Attention representation Significantly reduces KV cache memory Extra down/up projection computation
    Muon Split Parameter optimization Compatible with MLA; more stable updates Requires per-head momentum maintenance
    MTP Prediction target Denser training signals; better for planning Increased training compute
    DSA Attention computation Long-sequence computation becomes closer to linear Depends on lightweight scorer and custom operators

    The implication of this set of designs is clear: foundation model architecture optimization is no longer a single-point trick. It is systems engineering that combines model structure, optimization algorithms, parallel strategies, and inference deployment.

    Replacing any single component without considering the whole system can easily cause incompatibility.

    4. Post-Training: Fully Asynchronous Agent RL Training

    Pre-training gives a base model general capabilities. However, reasoning, tool use, and multi-step tasks depend heavily on post-training, namely supervised fine-tuning and reinforcement learning.

    The compute cost of RL is rising quickly for three reasons:

    1. Every policy update requires a large amount of sampling, scored by a reward model or verifiable rewards.

    2. Agent tasks often involve multi-turn interaction and long-chain reasoning. A single trajectory may reach thousands or even tens of thousands of tokens.

    3. Trajectory lengths are highly uneven. Simple requests finish quickly, while difficult requests involve repeated trial and error, causing severe GPU utilization imbalance.

    Synchronous RL divides the process into a serial loop: generate a batch, train one step, then generate the next batch.

    Its bottlenecks are obvious:

    • The duration of a training step is often determined by the longest trajectory.
    • The training cluster is idle during generation, and the generation cluster is idle during training.
    • Truncated unfinished trajectories may also have to be discarded.

    The long-tail nature of Agent tasks amplifies these inefficiencies.

    A fully asynchronous framework decouples generation from training:

    • The generation side continuously produces trajectories and writes them to a buffer.
    • The training side samples data from the buffer to update parameters.
    • The training side periodically synchronizes the latest weights to the generation side.
    • The buffer may contain data generated by older policies. The framework uses truncated importance sampling to correct distribution shift by bounding the ratio between old-policy probabilities and new-policy probabilities.
    1. # Illustrative pseudocode: asynchronous Agent RL main loop
    2. async def rollout_worker(actor, buf):
    3. while not stop:
    4. weights = await weight_server.latest()
    5. actor.update(weights)
    6. traj = await run_agent(actor)
    7. buf.put(traj)
    8. async def trainer(actor, buf):
    9. step = 0
    10. while step < max_steps:
    11. batch = await buf.sample(min_batch)
    12. ratio = clip(actor.prob(batch) / batch.old_prob, lo, hi)
    13. loss = -ratio * batch.advantage
    14. loss.backward()
    15. opt.step()
    16. if step % sync_interval == 0:
    17. await weight_server.publish(actor.state_dict())
    18. step += 1

    This is highly similar to the producer-consumer model in distributed systems: a queue absorbs the long tail, and version control manages data freshness.

    For extremely long trajectories, common approaches include early truncation and partial continuation.

    Early truncation ends a trajectory when it reaches the maximum number of turns or maximum length, and assigns a value estimate to the unfinished part. Partial continuation saves intermediate state and resumes generation later with the latest policy.

    The two can be combined. The key is to balance data freshness against trajectory completeness.

    Asynchrony also introduces stability challenges:

    • A high proportion of old-policy data can cause training oscillation.
    • Multi-turn Agent rewards are sparse, so process rewards or value models may be needed for credit assignment.
    • Non-deterministic scheduling order can reduce reproducibility.

    Therefore, it is important to limit the maximum age of queued data, record weight synchronization versions, fix random seeds, and downweight stale samples.

    5. RL Infrastructure: Turning Engineering Bottlenecks into Reusable Capability

    The report emphasizes self-developed RL infrastructure. The reason is that general training frameworks usually assume that data is static and the model is static. RL training is the opposite: the inference engine continuously generates trajectories, the training framework continuously updates parameters, and high-frequency weight synchronization is needed between them.

    Component Responsibility Key challenge
    Inference engine High-throughput trajectory generation Long-sequence throughput, KV cache management, continuous batching
    Training framework Parameter updates and gradient synchronization Pipeline parallel bubbles, recomputation, optimizer state partitioning
    Weight synchronization Push new parameters to the inference side Synchronization frequency, blocking time, transfer precision
    Data flow Trajectory buffering and scheduling Multi-producer writes, scheduling by length or difficulty
    Reward service Rule validation or reward model scoring Throughput, latency, consistency
    Observability Metrics, logs, tracing Locating bottlenecks across generation, training, and synchronization

    Engineering priorities include:

    • Dynamically adjusting the compute ratio between the generation pool and training pool according to context length and sampling volume.
    • Transferring weights at low precision and overlapping transfer with computation.
    • Grouping long and short samples in fine-grained ways to reduce pipeline bubbles.
    • Scheduling GPU resources in a unified way.

    From a public cloud perspective, such a system can map to general capabilities such as elastic container platforms, object storage, message queues, logging services, and monitoring and alerting. This section only describes general engineering capability mapping and does not recommend specific vendors. Actual selection should depend on business evaluation, cost budget, and operational capability.

    6. Developer Practices: From “Can It Be Used?” to “Is It Used Well?”

    FAQ: Three Common Questions

    What scenarios are MoE models suitable for?

    MoE models are suitable for scenarios that require large capacity, high throughput, and can accept higher memory costs, such as code completion, document understanding, multi-task question answering, and Agent tool calling. They are not necessarily suitable for every low-latency scenario, because memory usage and expert parallelism increase deployment complexity.

    When is sparse attention not cost-effective?

    Sparse attention may not outperform dense attention in short-context tasks, tasks with strong global dependencies, or tasks requiring precise global retrieval. The k value also needs to be validated for each use case. The optimal k may differ for code completion, long-document question answering, and repository-level understanding.

    How can asynchronous RL control stale data?

    Limit the maximum age of data in the queue, downweight stale samples, use truncated importance sampling to control distribution shift, and record each weight synchronization version. If training becomes unstable, first check data freshness and reward estimation.

    1. Model Selection: Build Your Own Evaluation Loop

    Leaderboards are only for initial screening. It is recommended to sample 200 to 500 evaluation examples from real business scenarios. These examples should include at least the following fields:

    • Task type: code generation, long-document question answering, tool calling, structured output, and so on.
    • Input length: short inputs, long inputs, and multi-turn dialogues tracked separately.
    • Expected output: standard answers, executable scripts, JSON structures, or human evaluation points.
    • Scoring criteria: accuracy, pass rate, format compliance rate, human preference score.
    • Latency records: first-token latency, overall response time, concurrent throughput.
    • Cost records: tokens per call, unit price, retry rate, cache hit rate.

    A/B testing with candidate models is more reliable than relying only on public leaderboards.

    2. Deployment Planning: Separate Compute Cost from Memory Cost

    Weight memory can be roughly estimated as:

    1. Weight memory ≈ total parameters × bytes per parameter

    For example, if 744B parameters are stored in bf16, each parameter takes about 2 bytes, so the weight memory is about 1.5TB. This requires multi-machine, multi-GPU deployment.

    The KV cache can be roughly estimated as:

    1. KV cache ≈ 2 × layers × KV heads × head dimension × context length × batch size × bytes per element

    MLA reduces KV cache cost by compressing latent vectors.

    If the model provides MTP modules, inference can consider using them as a draft model together with speculative decoding to reduce latency. However, this only works if the draft acceptance rate is high enough; otherwise, the extra overhead may not be worthwhile.

    Tensor parallelism is suitable for splitting parameters and computation within a single layer. Expert parallelism is suitable for distributing different experts across different devices.

    Deployment should be planned jointly based on activated parameters, total parameters, context length, and throughput targets.

    3. Long Context: Watch Out for the Hidden Cost of Quadratic Complexity

    The benefit of sparse attention depends on implementation quality. Business validation should focus on first-token latency, throughput, and memory usage at the target context length.

    MLA’s memory benefit is relatively clear, but long-context models may still suffer from “middle forgetting.” In engineering practice, important instructions can be placed at the beginning and end of the input, and long documents can be processed with segmented retrieval or structured recall.

    4. Post-Training: Evaluate the Benefit Before Investing Compute

    RL training has high infrastructure requirements. Small and medium-sized teams can prioritize managed post-training services.

    If building in-house, first estimate “average trajectory length × number of sampled trajectories” to understand the compute demand on the generation side. Then decide the ratio between the generation pool and the training pool.

    Algorithmically, start with mature approaches such as PPO or GRPO, and prioritize engineering effort on data quality and reward design. In many cases, data and reward design determine final performance more than framework tuning.

    5. Data and Compliance: Open Source Does Not Mean Unrestricted

    Open weights lower the barrier to use, but different licenses have different requirements for commercial use, redistribution, attribution, and output safety.

    Before deployment, teams should complete license review, output safety evaluation, and log compliance design.

    Conclusion: Efficiency-Driven Architecture Evolution

    Three main threads can be extracted from this technical report:

    1. Sparsification: MLA compresses the KV cache, and DSA dynamically selects computation. Together, they reduce the cost of storing and computing long sequences.

    2. Structural synergy: Muon Split, MTP, and sparse attention influence one another. No single change can be evaluated outside the overall system.

    3. Asynchrony: post-training is moving from synchronous batch processing toward streaming pipelines. Engineering capability is becoming a direct component of model capability.

    As open-source models approach frontier closed-source models on public evaluations, developers’ focus should also shift: from “can it be used?” to “is it used well, is it affordable, and is it stable?”

    The breakdown and checklists in this article can serve as a reference map for model selection, deployment planning, and post-training practice.