Trillion-Parameter MoE Models: Three Architecture Breakthroughs and Engineering Practices

1. Introduction: The Real Challenges of Trillion-Parameter MoE Models

In recent years, the open-source community has seen the emergence of sparse large models featuring total parameters at the trillion scale, activated parameters at the hundred-billion scale, native multimodality, and support for million-token contexts. Rather than just benchmark scores, the engineering details disclosed in their technical reports offer immense value to developers. At this scale, the bottleneck is no longer about “having enough parameters” but rather three systemic challenges:

1. Computational efficiency of ultra-long contexts: When the context window expands to a million tokens, the quadratic complexity of standard self-attention becomes unmanageable for both training and inference. Sub-quadratic attention variants must be introduced without sacrificing hardware utilization.
2. Training stability with massive experts: When routing experts reach the hundreds and each token activates only a tiny fraction, load imbalance can lead to routing collapse and expert under-training. Traditional auxiliary-loss-based balancing contaminates the main gradient and degrades model quality.
3. Long-horizon agentic reasoning: Multi-step tool calling, environmental interaction, and error recovery require planning, memory, and self-correction capabilities far beyond simple “next-token prediction,” necessitating specialized post-training pipelines.

A recent systematic solution from an open-source technical report addresses exactly these three directions. Stripping away product-specific perspectives, this article dissects the three most reusable designs from a general technical standpoint and provides training paradigms and implementation recommendations.

2. Architecture Overview: Hybrid Attention + Stable Sparse MoE

Based on the report, the architecture of such models can be summarized in four key points:

1. Hybrid Attention: The attention module combines Incremental Decay Attention (a linear attention based on delta update rules with natural decaying memory) and Gated Multi-Head Latent Attention. The former handles long-term memory decay with linear complexity, while the latter manages precise retrieval through KV latent space compression. A gating mechanism allows the model to dynamically weigh “fuzzy memory” against “precise retrieval” token by token.
2. Attention Residual (AttnRes): It shares attention information across layers to mitigate the representation bottleneck in deep networks while controlling memory and communication overhead.
3. Stable Latent Space MoE: Hundreds of routing experts are combined with an improved GLU activation variant and a quantile balancing mechanism to ensure training stability under extreme sparse activation.
4. Multi-stage Training: Pre-training progresses context length in four stages (8K → 64K → 256K → 1M), followed by agent RL post-training at the million-token level.

3. Deep Dive into Three High-Value Reusable Technologies

3.1 Lower-Bounded Decay Parameterization: Maximizing Matrix Multiplication Units for Linear Attention

Pain Point & Mechanism: Delta-rule-based linear attention typically uses chunked computation for parallel training. Within a chunk, intermediate results must be scaled by the inverse of the cumulative decay factor Γ. If log-decay uses unbounded Negative-Softplus parameterization, its value can approach negative infinity, causing Γ to approach zero. Under finite precision like BF16, 1/Γ will severely overflow or underflow. To avoid numerical risks, engineering implementations often force the diagonal tile to degenerate into an explicit point-by-point computation path—which fails to leverage hardware matrix multiplication units, drastically reducing operator throughput.

Solution: Change the parameterization of log-decay from unbounded to a lower-bounded scaled Sigmoid:

g_t = g_min · Sigmoid(w · z_t), where g_min is fixed at -5

Since the Sigmoid output falls in (0,1), the log-decay at any position is strictly bounded within (-5, 0). Consequently, within a fixed-size tile (e.g., 16 tokens), the inverse scaling range of the cumulative decay factor is also bounded and never exceeds the safe dynamic range of BF16. The intra-chunk computation can be fully matrixized and run entirely on tensor acceleration hardware.

Pseudocode:

def lower_bounded_log_decay(z, W, g_min=-5.0):
    # z: [T, D] input state; W: [D, H] learnable projection
    logits = linear(z, W)              # [T, H]
    return g_min * sigmoid(logits)    # Value range (g_min, 0), numerically safe

Engineering Takeaways: First, numerical stability should not rely on “patches” but should eliminate uncontrollable variables at the source through parameterization design. Second, “bounded activation + fixed tile size = statically deducible safe interval.” This approach of trading design constraints for operator efficiency can be transferred to other operators. This minor modification (replacing a single parameterization) significantly improves the implementation efficiency of linear attention operators on high-performance GPU stacks.

3.2 Quantile Load Balancing: Auxiliary-Loss-Free Dynamic MoE Routing Control

Pain Point & Mechanism: Traditional MoE load balancing relies on auxiliary losses—adding a balancing penalty alongside the main loss to “persuade” the router to distribute load via backpropagation. This causes three side effects: the balancing gradient interferes with the main task gradient, degrading model quality; the auxiliary loss weight is hard to tune (too weak is ineffective, too strong hurts accuracy); and it operates at the parameter level, meaning its response time is limited by the optimizer step size, unable to keep up with load fluctuations under extreme sparse activation.

Solution: Downgrade “balancing” from an optimization objective to a control problem at the routing layer. Dynamically adjust the bias of routing logits based on the real-time load quantile statistics of each expert—decrease the bias for overloaded experts and increase it for underloaded ones. Because this adjustment occurs during the forward routing phase rather than the backward gradient phase, it offers three characteristics:

• Near-zero overhead: Per-step cost is only on the order of the number of experts for statistics and updates.
• Fast response: Load fluctuations are perceived and corrected within the same step, naturally fitting ultra-large-scale expert scenarios.
• No gradient contamination: The main loss gradient is no longer diluted by the balancing objective, decoupling model quality from the balancing goal.

Engineering Takeaways: When introducing such mechanisms, incorporate the “expert load quantile curve” into the training monitoring dashboard rather than just looking at the final loss. Ablation studies for balancing strategies should be measured by model quality metrics, not mere balancing degrees.

3.3 Block-Level Attention Residual: Resolving Representation Bottlenecks in Deep Networks

Reusability: Medium-High, suitable for deep architectures with dozens of layers or more, especially models with a high proportion of attention layers.

Pain Point & Mechanism: The residual stream of Transformers has a classic issue: as depth increases, the relative proportion of attention sublayer outputs in the residual stream gradually dilutes. Deep attention “becomes quieter,” forming a representation bottleneck. A naive solution is to scale up the attention output or add bypasses, but this increases memory usage and inter-layer communication costs, which is prohibitively expensive in ultra-large-scale training.

Solution: Using “blocks” (groups of adjacent layers) as a unit, allow attention outputs to be shared across layers via residual channels and injected nearly identically into deeper layers. Deep attention no longer needs to rebuild information from scratch, breaking the bottleneck. Meanwhile, sharing is restricted within the block, keeping communication and memory overhead controllable and predictable.

Engineering Takeaways: In deeply stacked architectures, an “information highway” style cross-layer direct connection is often more cost-effective than simply deepening the network. During implementation, it is crucial to evaluate its compatibility with pipeline parallelism split points.

4. Training Paradigm: Multi-Stage Context Extension and Agent RL

Four-Stage Context: 8K → 64K → 256K → 1M: Staged progression rather than a single leap is chosen for three reasons: First, data mixture evolves with length—short stages focus on high-quality general corpus, while long stages gradually increase the proportion of long documents and code repositories. Second, staged progression provides a gradual path for the extrapolation adaptation of positional encoding. Third, every order of magnitude increase changes the numerical dynamics of attention and routing; staging makes it easier to isolate and locate stability issues.

Million-Token Agent RL: Long context is merely “working memory”; agent capability depends on post-training. Training trajectories span up to a million tokens, covering multi-turn tool calling, environmental feedback, and error recovery. Reward design must balance outcome and process signals to mitigate the credit assignment problem over long trajectories. Additionally, RL and long-context capabilities are deeply coupled—a sufficiently large window is a necessary condition for the agent to maintain state.

5. Implementation: Three Reusable Paths

1. Operator Layer: Implement the lower-bounded decay version of chunked linear attention kernels based on underlying matrix multiplication libraries. Before deployment, run numerical consistency tests (against high-precision reference implementations) followed by performance stress tests to verify the actual utilization of tensor acceleration hardware.
2. Training Framework Layer: Quantile balancing relies only on forward routing statistics and can be integrated as a hook/plugin in any framework supporting MoE. Block-level attention residual involves inter-layer connections, requiring evaluation of compatibility with parallel splitting strategies. It is recommended to continuously monitor expert load quantiles, attention branch gating means, and loss curves at each context stage.
3. Compute & Deployment Layer: Ultra-long context training and RL post-training demand high elasticity in compute power. Teams finding self-built clusters too expensive can build elastic training and evaluation pipelines using general cloud services (such as compute, container platforms, object storage, and logging/monitoring services provided by Baidu AI Cloud), scaling resources up or down by stage. On the inference side, focus on designing KV cache management—in hybrid attention architectures, the linear branch does not need to cache complete KVs, significantly reducing the memory cost of long-context services.

6. Conclusion: From “Bigger” to “More Stable, Longer, Smarter”

The correct way to view this generation of trillion-parameter MoE models is not to fixate on fractional benchmark improvements, but to understand how it pushes three “chokepoint” engineering problems into new solution spaces: numerical safety via parameterization design rather than patches; load balancing via routing control rather than gradient penalties; and deep networks via information sharing rather than blind deepening. These three principles are independent of any specific model and can be smoothly migrated to proprietary architectures and training pipelines. For most teams, this is the true value to take away from open-source technical reports.