Large-Scale Mixture-of-Experts Architecture and Agentic Reinforcement Learning: A Technical Report Breakdown
This article breaks down a new-generation large model technical report across four dimensions: why the architecture is designed the way it is, how the training strategy improves sample efficiency, how agentic reinforcement learning transfers into real task capability, and how the underlying infrastructure supports large-scale experiments. It covers the trade-offs of a 744B total parameter / 40B activated parameter MoE design with 256 experts and 80 layers, the comparison with the previous generation (~355B total, ~32B activated, 23T tokens vs. 28.5T tokens), multi-head latent attention (MLA) and per-head orthogonalization, multi-token prediction, and dynamic sparse attention. It then walks through fully asynchronous agentic RL, the platform capabilities required to run it, evaluation dimensions beyond benchmark scores, and deployment cost considerations.
1. Technical Reports Are Now Read as Systems Engineering Documents
Large model technical reports have shifted from plain model descriptions to systems engineering documents that cover architecture, data, training strategy, and infrastructure together. Early readers mostly looked at headline numbers: parameter count, context length, leaderboard scores. As model capabilities converge, however, the marginal return of any single parameter is falling. What actually determines performance is how well multiple stages work together:
- whether the attention mechanism stays efficient at long context lengths
- whether the mixture-of-experts (MoE) structure balances capacity against activation cost
- whether the training data mixture is tuned with enough granularity
- whether reinforcement learning can inject task capability without destabilizing training
- whether the training framework can support large-scale concurrent sampling
Reading a new-generation technical report therefore cannot stop at “the model got bigger” or “the benchmark scores went up.” It needs to be decomposed into four questions: why is the architecture designed this way; how does the training strategy improve sample efficiency; how does agentic reinforcement learning improve real task capability; and how does the training infrastructure support complex experiments. Only when those questions are connected does the engineering logic behind the report become visible.
At the industry level, closed-source frontier models still hold an overall capability advantage, but the gap to open-weight models is narrowing noticeably. In several public evaluations, open-weight models have reached the first tier on dimensions such as general intelligence, code generation, and text arena performance. The reason is not a single algorithmic breakthrough but the combined effect of MoE architectures, sparse attention, asynchronous reinforcement learning, training data engineering, and infrastructure platforms.
2. Base Architecture: The Trade-Off Behind 744B Total Parameters and 40B Activated Parameters
This generation uses a large-scale mixture-of-experts architecture. The core idea of MoE is to keep growing total parameter count while activating only a small fraction of those parameters per inference pass, so capacity and inference cost are decoupled. The reported baseline figures are: approximately 744B total parameters, approximately 40B activated per inference, 256 experts, and a network depth of 80 layers.
Compared with the previous generation, the key structural changes can be summarized as doubling total parameters, raising activated parameters modestly, and expanding training data further. The previous generation had roughly 355B total parameters and 32B activated; this generation moves to 744B total and 40B activated. Pre-training data also grew from about 23T tokens to about 28.5T tokens, split into roughly 27T for pre-training and 1.5T for mid-training.
These numbers reflect the prevailing approach to base model design: expand total parameters to increase knowledge capacity and representational space, while controlling activated parameters so inference cost does not balloon. The goal is no longer simply “bigger” but “affordable at greater capacity.”
From an engineering standpoint, MoE offers three main advantages:
- Parameter capacity is decoupled from computation. Total parameters can be very large, but the path each token actually takes is only a subset, so effective compute at inference time stays controllable.
- Expert specialization helps the model learn different modalities, domains, and task patterns. Code, math, natural language, and tool calling may activate different expert combinations.
- It suits large-scale distributed training. Expert parallelism, tensor parallelism, and data parallelism can distribute different experts across different devices.
MoE also introduces new challenges:
- Routing stability. If the routing distribution is too concentrated, a few experts get selected repeatedly and the load becomes unbalanced. If it is too diffuse, experts end up underutilized.
- Communication overhead. When experts live on different devices, tokens must be dispatched across devices, and communication efficiency directly affects training throughput.
- Inference cost. Activated parameters per pass are lower, but memory footprint, caching, scheduling, and batching strategy all influence the final serving cost.
Evaluating an MoE model on total or activated parameters alone is therefore insufficient. Routing policy, expert granularity, training stability, inference throughput, and cost per request all matter.
3. Architecture Change One: Multi-Head Latent Attention and Per-Head Orthogonalization
For attention, this generation uses Multi-Head Latent Attention (MLA). MLA’s core goal is to reduce the memory footprint and memory-access pressure of the key-value (KV) cache in long-context scenarios. Standard attention typically stores full keys and values for every head, so the cache grows quickly as context length increases. MLA applies low-rank compression, mapping key and value representations into a lower-dimensional latent space and thereby shrinking cache usage at inference time.
This matters for long documents, multi-turn conversations, and repository-level code understanding. The practical bottleneck for long-context models is usually not peak compute but memory bandwidth and cache capacity. An oversized KV cache reduces throughput even when compute is abundant, because memory access cannot keep up.
The report also notes an engineering problem the team hit during training: with a certain orthogonalization-based optimizer, the MLA structure underperformed the simpler grouped-query attention (GQA) structure. GQA reduces cache overhead by cutting the number of key-value heads; it is simple and well-established. The team found that applying uniform orthogonalization to the projection matrix as one whole block did not match the compressed structure of MLA well, and optimization quality suffered.
The response was per-head orthogonalization. Instead of orthogonalizing the entire projection matrix as a single unit, the team applies the orthogonalization constraint separately to each attention head. The intent is to make optimization better match the parameter distribution of each individual head and avoid interference between heads within the same block matrix.
The idea can be expressed in pseudocode:
# Original approach: orthogonalize the whole projection matrix as one blockW = orthogonalize(W_all_heads)# Per-head approach: orthogonalize each attention head separatelyfor head_id in range(num_heads):W_head = extract_head_weight(W_all_heads, head_id)W_head = orthogonalize(W_head)write_back_head_weight(W_all_heads, head_id, W_head)
The point of this change is to keep MLA’s long-context efficiency while restoring optimizer stability during training. In large model training, adjustments that look this granular often prove decisive. When training stability is insufficient, the mild outcome is slow loss reduction; the severe outcomes are gradient anomalies, expert routing collapse, or erratic final quality.
4. Architecture Change Two: Multi-Token Prediction for a Denser Training Signal
The second significant architectural change is multi-token prediction. A conventional autoregressive model predicts only the next token at each position, so the training signal unfolds one position at a time. Multi-token prediction lets the model predict not just the immediately following token but several future positions as well, or learn longer-range dependencies through auxiliary objectives.
The direct benefit is a denser training signal. In natural language, code, and math, a given position relates not only to the next token but also to many subsequent tokens, sentences, or even code blocks. Optimizing single-step prediction alone can push the model toward local statistical regularities. Multi-token prediction forces it to learn longer dependency structures, which helps planning and context understanding.
For training efficiency, multi-token prediction supplies richer supervision at the same data scale. This is especially useful when high-quality data is limited: denser objectives help the model learn language structure, code patterns, and reasoning chains faster.
At inference time, multi-token prediction can also combine with parallel decoding, draft verification, and speculative decoding. The model can generate several candidate tokens quickly and have the main model verify or correct them. If the hit rate is high, the main model needs fewer sequential generation steps; if it is low, decoding falls back to the standard path. Throughput improves without moving the quality ceiling.
There is a cost: higher training complexity. Predicting multiple future positions requires a more elaborate loss design, weighting decisions across different prediction distances, and safeguards against the model attending only to easy short-range tokens while neglecting long-range dependencies. Common mitigations include weighted losses, curriculum-style training, and staged objective switching, letting the model stabilize short-range prediction first and then strengthen long-range prediction.
5. Architecture Change Three: Dynamic Sparse Attention and Long-Context Efficiency
The third key architecture is dynamic sparse attention (DSA). In long-context modeling, full attention computes the relationship between every pair of tokens, so complexity grows quadratically with context length. As context expands from a few thousand tokens to hundreds of thousands or more, the compute and memory cost of full attention rises sharply.
Dynamic sparse attention has each position attend only to the most important tokens rather than treating all tokens equally. Typical implementations include local window attention, global key-token attention, block sparse attention, and routing-based sparse attention. The goal is to retain the context most critical to the task while cutting irrelevant computation.
In pseudocode, sparse attention can be abstracted as:
scores = compute_attention_scores(Q, K)# Keep only the most important keys for each queryindices = topk_select(scores, k)# Build a sparse mask so attention runs only on the selected keysmask = create_sparse_mask(scores.shape, indices)attn_output = sparse_softmax_attention(Q, K, V, mask)
The value of this design is a shift from full dependence to key dependence for long-context training and inference. In code comprehension, the model does not necessarily need every token in the repository; it needs to identify the current change site, the relevant functions, the dependency context, and the test snippets. In long-document question answering, it does not necessarily need to spend equal compute on every paragraph; it needs to locate key facts, causal chains, and summary content.
Comparisons in the report show that under certain training configurations, roughly 20B tokens of training approached the results of a much larger full-attention training run of about 940B tokens. That result needs to be read alongside the specific tasks, data quality, and evaluation methodology, but it is enough to demonstrate the potential of sparse attention and efficient training strategies for sample efficiency.
For engineering implementation, dynamic sparse attention is not simply “compute less.” The real difficulties are:
- Selecting key tokens well, so important information is not dropped.
- Keeping training stable, since frequently changing sparsity patterns can cause gradient oscillation.
- Building efficient kernels, so sparse computation actually translates into speed gains.
- Remaining compatible at inference time with cache reuse, batching, and variable context lengths.
6. Agentic Reinforcement Learning: From Static Training to Dynamic Environments
If the base architecture determines knowledge capacity and inference efficiency, agentic reinforcement learning determines whether the model can complete multi-step operations in real tasks. Pre-training mostly learns “what the next token is.” Agentic reinforcement learning in post-training instead asks “given a task objective, which tools should the model call, which actions should it execute, and which feedback should it use to correct itself.”
Agentic tasks typically involve tool calling, code execution, retrieval augmentation, multi-round planning, environmental feedback, and result verification. Unlike single-turn question answering, they feature multi-round interaction, changing state, sparse rewards, and delayed feedback. The model must not only produce correct content but also make consecutive decisions inside a complex environment.
A fully asynchronous reinforcement learning training framework is aimed precisely at the training and sampling efficiency of these tasks. In synchronous training, sampling, environment execution, reward computation, and parameter updates all wait for the same batch to finish. If one sample calls a slow tool or one environment returns with high latency, the entire batch slows down. An asynchronous framework decouples sampling, environment interaction, reward evaluation, and model updates so the modules can advance in parallel.
A simplified sketch of asynchronous agentic RL:
while training:# The rollout service keeps generating trajectoriestrajectories = rollout_service.generate_async_batch(model_snapshot)# Environment execution and reward evaluation proceed in parallelrewards = reward_service.evaluate_async(trajectories)# Write trajectories to the replay buffer for the training stagereplay_buffer.add(trajectories, rewards)# The training stage consumes samples from the latest model versionbatch = replay_buffer.sample()loss = policy_loss(model, batch)optimizer.step()
The gains from asynchronous training show up in three areas:
- Higher hardware utilization. Inference sampling, environment execution, and training updates overlap, reducing idle waiting.
- Higher experiment throughput. Many environment interactions can run concurrently, which suits the long-chain sampling common in agentic tasks.
- Support for larger-scale exploration. The framework keeps producing new trajectories, exposing the model to a wider range of task states.
Asynchronous training also brings new challenges:
- Data freshness. Training trajectories may come from an older version of the model, and large distribution shifts can hurt training stability.
- Reward quality. Rewards in agentic tasks may come from rule-based checks, execution results, human feedback, or model evaluation, and noise differs across these sources.
- Long-tail environments. Some tasks rarely trigger, skewing the training sample distribution.
The usual response combines several strategies: curriculum learning to raise task difficulty gradually; reward shaping to convert sparse feedback into denser intermediate signals; experience replay to increase sample reuse; filtering of low-quality trajectories; weighted sampling for critical tasks; and a thorough monitoring setup covering reward curves, policy entropy, task success rate, trajectory length, and abnormal interruption rate.
7. Training Infrastructure: From Algorithm Experiments to Platform Capability
The training infrastructure discussed repeatedly in the report reflects a broader shift: large model training has moved from isolated algorithmic optimization into platform-level competition. Agentic reinforcement learning depends on infrastructure more heavily than most training regimes, because it simultaneously involves inference sampling, environment execution, reward computation, data storage, task scheduling, and model training.
A complete training infrastructure generally needs to cover five capabilities.
Resource scheduling. Large-scale training manages hundreds to thousands of accelerators and involves allocation, queueing, isolation, failure recovery, and elastic scaling. Training jobs and environment jobs have different resource profiles, so scheduling needs to differ by task type.
Trajectory storage. Agentic training produces large volumes of multi-turn conversations, tool call records, execution results, and reward signals. This data must not only be stored but also support versioning, retrieval and replay, quality filtering, and later analysis.
Task orchestration. An asynchronous framework has to coordinate samplers, environment pools, reward services, and trainers. Any component that blocks can drag down overall throughput, so orchestration must support concurrency control, failure retries, and state tracking.
Observability. Large model training failures rarely have a single cause; they usually result from several metrics drifting together. Monitoring should cover loss, reward, gradient norms, routing distribution, expert load, memory usage, communication latency, sampling throughput, and environment success rate.
Storage and messaging decoupling. In cloud-based training systems, object storage holds training data, model checkpoints, and trajectory files; message queues decouple the sampling, reward, and training modules; container platforms provide a uniform deployment target for training, inference, and environment services; and log and alerting services track the state of long-chain jobs. These general-purpose capabilities combined are what keep complex reinforcement learning training stable.
Infrastructure, in other words, is not an appendage to the algorithm; it is the precondition for validating the algorithm at scale. Without stable asynchronous scheduling, trajectory management, and monitoring, reinforcement learning experiments easily end up in a state where they run but cannot be reproduced.
8. Evaluation and Engineering Deployment: From Leaderboard Capability to Real Business Value
For anyone adopting a model, the technical report ultimately has to answer one question: do these architectural and training optimizations translate into real business gains? Answering it requires a more dimensional evaluation system.
General capability evaluation covers language understanding, knowledge question answering, logical reasoning, arithmetic, long-document summarization, and code generation. This kind of evaluation is useful for observing broad capability, though it should be read with caution rather than treated as a single leaderboard.
Code and agent evaluation. Real development work is rarely single-file completion; it involves repository-level understanding, defect localization, patch generation, test execution, dependency analysis, and multi-round repair. Agent evaluations need to look at tool call success rate, task completion rate, average number of interaction rounds, error recovery, and final code quality.
Long-context evaluation. Now that long context is standard, the focus should not stop at whether a long document fits. It should examine whether the model retrieves key information accurately, understands cross-paragraph relationships, performs reasoning over long documents, and produces structured summaries.
Cost and latency evaluation. Capability gains have to be read together with cost per token, time to first token, throughput, concurrency, and cache hit rate. For enterprise use, a high-quality model whose cost is too high is not necessarily a better fit for scale than a slightly weaker model with controllable cost.
During engineering deployment, inference strategy also needs attention. KV cache compression reduces long-context memory pressure; sparse attention speeds up long-document inference; batching raises throughput; quantization lowers memory usage; and elastic scaling absorbs traffic fluctuation. Online evaluation and gradual rollout mechanisms should combine offline metrics, in-production experience, and cost indicators.
9. Conclusion: Competition Now Turns on Architecture, Training, and Infrastructure Together
Breaking down this generation’s technical report reveals several clear trends in large model development.
Architecture design now emphasizes a balance between efficiency and capacity. MoE raises the capability ceiling by expanding total parameters while containing inference cost by limiting activated parameters; MLA improves long-context efficiency by compressing the KV cache; dynamic sparse attention reduces the long-context compute burden by selecting key tokens. All of these point to the same objective: higher effective intelligence under a fixed compute budget.
Training objectives are moving from single-step prediction to multi-step decision-making. Multi-token prediction increases training signal density, and agentic reinforcement learning keeps improving the model on real tasks such as tool calling, code execution, and environmental feedback. The model is no longer merely something that “generates text”; it is progressively becoming something that “completes tasks.”
Asynchronous training frameworks are central to agentic capability. Synchronous training struggles to support high-concurrency, multi-environment, long-chain sampling. Asynchronous architectures decouple sampling, reward, replay, and training, raising throughput and enabling larger-scale exploration.
Training infrastructure is becoming a platform. Large-scale model training is no longer a single-algorithm problem but a combination of resource scheduling, storage, messaging, monitoring, observability, and experiment management. Algorithmic innovation can only be validated and amplified repeatedly on a stable infrastructure base.
Overall, the value of this report lies less in disclosing a specific set of parameters and more in showing a representative path for large model competition: MoE carries capacity, efficient attention mechanisms lower long-context cost, multi-token prediction improves sample efficiency, asynchronous agentic reinforcement learning strengthens task capability, and training infrastructure sustains iteration at scale. For developers and enterprises, understanding these mechanisms is more useful than tracking any single model score.
