Sparse MoE Architecture and Post-Training RL for Large Models

1. The Problem: Where the Focus of Architecture Iteration Is Shifting

As large-parameter models continue to evolve, a frequently discussed question is: once the underlying skeleton of a model stabilizes, where does the room for capability improvement come from? Recent industry practice points to a fairly consistent answer — architecture determines the capability ceiling and cost structure, while post-training determines how much of that capability is actually realized. This article takes a sparse MoE model with approximately 2.8T total parameters as an analytical subject, discussing its architectural inheritance, attention mechanism combinations, methods for balancing long context and throughput, and the effects of large-scale post-training and cross-domain joint reinforcement learning, along with transferable engineering insights.

It should be noted that this article does not evaluate or recommend any specific commercial product. Instead, it abstracts the relevant content into general technical questions: how to fully extract capability from an already-validated sparse architecture skeleton through post-training and reinforcement learning.

2. Architectural Inheritance: Stable Skeleton, Improvements Concentrated in Post-Training

From publicly available technical descriptions, the new generation model and its predecessor are essentially consistent at the underlying skeleton level, with differences mainly concentrated in the post-training stage and cross-domain reinforcement learning. This “skeleton inheritance + post-training enhancement” approach has several engineering rationales:

1. Reduced architectural risk. Issues such as routing stability, load balancing, and communication overhead in sparse MoE have been thoroughly validated in the previous generation. Direct inheritance avoids repeating known pitfalls.
2. Shorter iteration cycles. An unchanged architecture means the training framework, parallelism strategy, and inference engine can largely be reused, allowing teams to focus on data composition, reward modeling, and RL strategy.
3. Easier attribution. When the architecture is fixed, capability changes are more easily attributed to post-training data and RL strategy, enabling reusable tuning experience.

This approach also highlights a common misconception: many teams first think of “changing the architecture” when model performance is unsatisfactory. However, when the architecture is already sufficiently general, data quality, reward signal design, and training stability are often more efficient areas of investment.

3. Sparse MoE Parameter Structure and Cost Model

The model adopts a sparse MoE structure with approximately 2.8T total parameters, supports a context window of millions of tokens, stores weights in a native low-precision format (MXFP4), and has a measured weight size of approximately 1.56TB, split into 96 shards.

Several technical points deserve elaboration:

Total parameters vs. activated parameters. The core idea of sparse MoE is “large parameter capacity, small per-inference compute.” Total parameters determine the model’s memory capacity and knowledge coverage, while the expert parameters actually activated per token account for only a small fraction. Therefore, the per-token compute cost is far lower than a dense model with equivalent total parameters. This is why a 2.8T-level model remains engineering-feasible.

Low-precision native weights. Training and storing natively in low-precision formats such as MXFP4 can significantly compress weight size and memory bandwidth pressure. The 1.56TB of weights split into 96 shards is essentially preparation for distributed inference and on-demand loading — a single machine cannot hold all weights, so shard parallelism and pipeline scheduling are required.

The cost of million-token context windows. Extending the context window is not “free.” Its cost is mainly reflected in KV Cache memory usage and attention computation complexity. Therefore, long-context capability must be designed in coordination with the attention mechanism, which is the subject of the next section.

A pseudocode snippet can summarize the cost structure of sparse MoE forward computation:

# Pseudocode: per-token cost illustration for sparse MoE forward pass
def moe_forward(x, experts, router, top_k):
    # 1. Routing scores: all experts participate in scoring, but computation is minimal
    scores = router(x)                  # shape: [num_experts]
    # 2. Only top_k experts are activated; computation scales with top_k
    selected = top_k_indices(scores, k=top_k)
    outputs = [experts[i](x) for i in selected]
    # 3. Weighted merge
    return weighted_sum(outputs, scores[selected])
# Conclusion: total params ~ num_experts * per_expert_params
#             per-token compute ~ top_k * per_expert_params

4. Hybrid Attention: Combining Linear and Sparse Attention

At the attention level, the model employs a hybrid of two mechanism types: linear attention (a linear attention variant represented by KDA) and sparse attention (DSA-type approaches), with support for multi-token prediction (MTP).

The value of linear attention lies in complexity. Standard attention has O(N²) computational complexity for sequence length N, while linear attention reduces it to approximately O(N), which is critical for million-token contexts. The trade-off is that linear attention is typically weaker than full attention in precise retrieval and long-range dependency modeling.

The value of sparse attention lies in selectivity. Sparse attention does not compute relationships between all token pairs. Instead, it selects connections based on certain rules (such as local windows, global tokens, or block sparsity), compressing computation while retaining key information.

The engineering logic of combining the two is: use linear attention for the bulk of long-range information aggregation, and sparse attention to supplement precise modeling at critical positions, thereby balancing memory usage and throughput. This also explains why long-context capability must be designed together with the attention architecture — simply adding memory cannot solve the O(N²) computational bottleneck.

Multi-token prediction (MTP) improves efficiency from another angle: predicting multiple subsequent tokens in a single forward pass increases training signal density and, during inference, can work with speculative sampling techniques to improve effective throughput. Its essence is trading additional prediction heads for higher output per unit of compute.

5. Post-Training and Cross-Domain Joint Reinforcement Learning

Beyond architecture, post-training is what truly creates the gap. Related practice concentrates compute investment in large-scale post-training and cross-domain joint reinforcement learning, and reconstructs the full-stack engineering from training and inference to sandbox evaluation. This deserves to be examined in parts:

Data engineering for large-scale post-training. The core of the post-training stage is not data volume, but the “task coverage” and “verifiability” of the data. For tasks such as code, terminal operations, and tool calling, data naturally has executable and verifiable properties, providing high-quality reward sources for reinforcement learning.

Cross-domain joint reinforcement learning. RL in a single domain easily leads to capability imbalance or reward hacking. The value of cross-domain joint training is that tasks from different domains share underlying reasoning and planning capabilities. Joint optimization can improve generalization and reduce overfitting to specific task formats.

Sandbox evaluation loop. Incorporating sandbox evaluation into the full training and inference stack means model outputs can be actually executed in an isolated environment and results collected, forming a “generate—execute—feedback” loop. This is key infrastructure for improving terminal operation and code capabilities.

Performance-level reference. Based on public information, this type of model achieves approximately 65.7% on terminal task benchmarks, improves from approximately 55.95% to 61.2% on coding benchmarks, and simultaneously releases a high-speed version with 2–3x speedup. The significance of these numbers is not absolute ranking, but rather demonstrating that with an unchanged architecture, investment in post-training and RL can deliver measurable capability gains, while the release of a high-speed version indicates parallel engineering investment in inference optimization.

It should be emphasized that benchmark results are heavily influenced by evaluation environment, task sampling, and execution sandbox configuration. When making horizontal comparisons, the consistency of evaluation protocols should be the focus rather than simply comparing numbers.

6. Engineering Practice Recommendations

Based on the above analysis, several practice recommendations can be distilled for developers and enterprise users:

1. Stabilize the architecture first, then optimize post-training. If the architecture is already validated, prioritize resources on data composition, reward modeling, and training stability rather than frequently modifying the skeleton.
2. Long-context capability requires matching attention design. When extending the context window, simultaneously evaluate linear attention, sparse attention, and KV Cache compression strategies to avoid both memory and throughput spiraling out of control.
3. Use executable tasks to construct RL signals. Verifiable tasks such as code, terminal operations, and tool calling are natural sources of high-quality rewards and are suitable starting points for reinforcement learning.
4. Build a sandbox evaluation loop. Incorporating execution environments into the training and evaluation pipeline can significantly reduce the problem of “the model says it correctly but it won’t run.”
5. Reserve a high-speed version path on the inference side. Through low-precision weights, sharded loading, multi-token prediction, and speculative sampling, trade for throughput gains without significant capability degradation.

7. Summary

The evolution of ultra-large sparse MoE models is shifting from “architecture innovation-driven” to “architecture stability + post-training-driven.” The 2.8T total-parameter sparse structure, million-token context, low-precision native weights with sharded deployment, and hybrid linear-sparse attention with multi-token prediction together form the balancing foundation between capability and cost. Large-scale post-training and cross-domain joint reinforcement learning are the key steps that actually realize capability on this foundation. For engineering teams, understanding this layered logic has more long-term value than chasing single-point metrics.