Large-Scale MoE Model Training: A Deep Technical Breakdown of a New Open-Source Foundation Model

    Open-source models are now iterating fast enough to match, and in some tasks overtake, the top overseas closed-source models. A recent open-source team released a full 40-page technical report on its new-generation foundation model, with sparse attention, a fully asynchronous reinforcement learning training framework, and in-house training infrastructure drawing the most attention. The real value of reports like this is not "another model launch" — it is that they put the actual engineering trade-offs of large-scale MoE training on the table, which is directly useful for teams planning training, fine-tuning, or inference deployment. This article walks through the model's fundamentals, benchmark results, three notable architecture changes, the asynchronous Agent RL training framework, the case for custom training infrastructure, and the engineering practices worth carrying away.

    1. Model Fundamentals: Reading “Total Parameters” vs. “Activated Parameters”

    The model uses a Mixture of Experts (MoE) architecture. The core idea: the network contains many parallel “expert” subnetworks, and a router/gating mechanism activates only a small fraction of them at inference time, achieving “very large parameter count, controllable compute per pass.”

    The key numbers are:

    • 744B total parameters, 40B activated per inference;
    • 256 experts, 80 layers;
    • Versus the previous generation: total parameters doubled from 355B to 744B, activated parameters rose from 32B to 40B;
    • Pretraining data grew from 23T tokens to 28.5T tokens (27T pretraining, 1.5T mid-training).

    Understanding these figures comes down to separating capacity from compute. Total parameters determine the model’s knowledge capacity and expressive ceiling; activated parameters determine the compute and GPU memory cost of a single forward pass. This is exactly what MoE decouples: a large total parameter count to hold knowledge, a small activated count to keep inference cost controllable.

    1. # Simplified MoE routing illustration (pseudocode)
    2. logits = router(x) # score every token against every expert
    3. topk_idx, topk_w = top_k(logits, k=2) # keep only the 2 highest-scoring experts
    4. y = 0
    5. for i, expert_id in enumerate(topk_idx):
    6. y = y + topk_w[i] * experts[expert_id](x) # compute only the selected experts

    One caveat: while MoE reduces compute per pass, GPU memory must still hold all expert weights. Memory optimization for inference deployment — quantization, expert parallelism, cache reuse — is therefore just as critical.

    2. Benchmark Performance: Reading the Leaderboard Rationally

    At release, the model scored 50 on a widely cited general intelligence benchmark, ranking first among open-source models and behind only the strongest overseas closed-source model. It also placed first among open-source models on two public human-preference arenas — a general text arena and a code arena — putting it in the same tier as the top closed-source models overall.

    It is worth emphasizing that a single benchmark score is not equivalent to real business performance. Evaluation sets carry inherent limitations: data contamination risk, shifts in question distribution, and subjectivity in scoring criteria. The rational approach is to treat leaderboards as a reference coordinate for horizontal comparison, not as a conclusion about a capability ceiling. Actual model selection should come back to A/B comparison on your own business evaluation set.

    3. Architecture Evolution: Three Changes Worth Watching

    Compared with the previous generation, this model has three clear architectural changes: the interaction between the attention mechanism and the optimizer, multi-token prediction, and sparse attention. Together they point at one goal — reducing training and inference cost while maintaining or even improving capability.

    3.1 Attention Mechanism + Optimizer Co-Tuning

    The model uses an attention mechanism that saves GPU memory by compressing the KV cache dimension (in the same family as mainstream low-rank KV compression approaches), which uses less memory and runs faster than conventional designs in long-context scenarios.

    The engineering team, however, hit a counterintuitive result during training: pairing a class of matrix-orthogonalization-based optimizers directly with this compressed attention actually underperformed the more plain-vanilla grouped-query attention setup. The team’s eventual solution can be summarized as “split orthogonalization”: instead of applying unified orthogonalization to the whole projection matrix, orthogonalize each attention head separately.

    The underlying logic is that across different attention heads, the row/column scales of the projection matrices vary widely, and global orthogonalization effectively “averages out” those differences, hurting expressive power. Splitting per head gives each subspace a better-matched update scale. This is a reminder that the optimal combination in a paper is not necessarily optimal in your own reproduction — an optimizer must be tuned together with the specific architecture.

    3.2 Multi-Token Prediction (MTP)

    Multi-token prediction is a training-objective enhancement: rather than predicting only the next token, the model uses extra prediction heads to predict several tokens further ahead. It serves two purposes. First, it provides a denser supervision signal, improving training efficiency and representation quality. Second, those extra prediction heads can be reused at inference time for speculative decoding, accelerating generation. Two levels should be kept distinct: during training it helps the model learn better; at inference it can optionally be used for speedup. The two do not conflict.

    3.3 Sparse Attention

    Sparse attention is the most-discussed change in this generation. Its core point is not “compute less” but “compute selectively”: through a learnable mechanism, it dynamically selects the small set of tokens each query actually needs to attend to, reducing attention’s computational complexity from roughly quadratic in sequence length to nearly linear.

    The result reported is compelling: with only about 20B tokens of continued training, it matched the effect of a well-known industry approach using about 943.7B tokens. The logic behind the conversion is that once attention moves from dense to sparse, the model extracts more information per token on the same data — equivalent to capturing, with very few incremental tokens, the benefit others obtained from nearly a trillion tokens. For long-context training, that is a real cost saving.

    4. Training Framework: Fully Asynchronous Agent Reinforcement Learning

    If architecture determines per-step efficiency, the training framework determines overall throughput. The other focus of this report is a “fully asynchronous Agent RL training framework.”

    In conventional RL training, “sampling (rollout)” and “training (update)” are usually serial: a batch of samples must be generated and scored before parameters can be updated once. The Agent setting makes this worse — one sample may require multiple rounds of tool calls and environment interaction, making trajectories long and uneven in duration. A serial framework leaves compute idle while it waits.

    The “fully asynchronous” approach decouples generation and training into two continuously running pipelines: the inference side keeps producing experience and writing it into a buffer, while the training side keeps pulling data from the buffer to update the policy, so neither waits on the other. The benefits include:

    • Much higher compute utilization, with long-tail samples no longer blocking overall progress;
    • A natural fit for Agent-style multi-turn, variable-length trajectories;
    • Easier independent horizontal scaling of the inference side and the training side.

    The cost is the introduction of “stale samples (on-policy drift)”: training data may come from a slightly outdated policy. In practice this requires controlling the degree of asynchrony, applying importance-sampling corrections, or setting a data-freshness window to balance throughput against stability.

    5. Training Infrastructure: The Value of Building Your Own

    The report also mentions in-house reinforcement learning training infrastructure. The rationale is practical: general-purpose training frameworks are mostly optimized for the supervised “forward + backward” flow, while RL/Agent training must additionally manage environment interaction, experience buffers, reward computation, and orchestration of multiple roles (generation / training / scoring). Stitching these pieces into a high-throughput, elastically scalable system usually requires customization. The point of in-house infrastructure is not to reinvent the wheel, but to specifically optimize the bottleneck areas unique to RL: asynchronous scheduling, experience management, fault recovery, and compute orchestration.

    6. Engineering Takeaways

    From a general engineering perspective, the following capability combination is worth attention when building such training/deployment systems:

    • Use a container platform for elastic resource orchestration, splitting generation and training into independently scalable roles;
    • Use object storage and message queues to buffer and decouple experience and trajectory data;
    • Use logging plus monitoring and alerting to establish end-to-end observability — asynchronous systems especially require watching “data freshness” and “throughput gap between the two sides”;
    • Use an independent business evaluation set with an automated pipeline for regression, avoiding reliance on public leaderboards alone.

    Conclusion

    Reading this report thoroughly yields more than a model’s parameter table — it yields a complete engineering methodology for how to train large-scale MoE models both well and cheaply: MoE that decouples capacity from compute, co-tuning of attention and optimizer, the data efficiency that sparse attention brings, the dual training-and-inference payoff of multi-token prediction, fully asynchronous Agent RL, and the infrastructure that supports it. For teams doing training and deployment, the single most valuable principle is this: architecture, optimizer, and training framework must be co-designed as one whole, because a break at any point becomes the bottleneck for overall efficiency.