How Trillion-Scale Sparse MoE Models Achieve Million-Token Long Context Capabilities

1. Background: Model Scaling Hits Three Walls

Over the past few years, the enhancement of large model capabilities has largely followed the path of “more parameters, more data, and more compute.” However, this trajectory is simultaneously approaching three limits.

The first is the Compute Wall. In dense architectures, inference overhead is proportional to the total number of parameters, tightly binding model capacity to inference costs. Achieving greater representation and knowledge capacity requires paying for the full parameter set during every forward pass.

The second is the Context Wall. Standard softmax attention incurs computational and memory overhead on the order of O(n²) relative to sequence length n. As context expands from 8K to 1M tokens, attention overhead scales quadratically. Yet real-world demands are pushing precisely in this direction: repository-level code comprehension, cross-analysis of hundred-page documents, and the ultra-long trajectories accumulated during multi-turn agentic tool calls all require the model to “read long, remember, and retrieve accurately.”

The third is the Task Wall. The value of models is shifting from “answering questions” to “completing tasks.” Agentic tasks routinely involve planning-execution-reflection loops spanning dozens of steps, demanding far superior long-range consistency and tool-calling reliability than static Q&A.

A recent technical report from an open-source multimodal large model provides a complete engineering blueprint for simultaneously overcoming these three walls. It utilizes a sparse MoE architecture with approximately 2.8 trillion total parameters and around 104 billion activated parameters per forward pass. It natively supports visual multimodality and a 1-million-token context. On the attention side, it employs a hybrid design of Gated Incremental Linear Attention and Gated Multi-head Latent Attention, utilizing attention residual connections for cross-layer information sharing. The MoE side features 896 routed experts equipped with specialized stabilization and load balancing mechanisms. Pretraining progresses through stages (8K→64K→256K→1M), followed by post-training with million-token-level Agentic RL. The following sections break this down into Architecture Overview → Overcoming the Three Major Challenges → Developer Takeaways.

2. Architecture Overview: The Combination of Sparsification and Hybrid Attention

2.1 MoE: Decoupling “Capacity” from “Compute”

The core idea of MoE is to replace the feedforward layers with multiple parallel experts, where a gating network selects only a few experts per token for computation. The 2.8 trillion total parameters correspond to 104 billion activated parameters, yielding a sparsity ratio of roughly 27:1. Knowledge capacity is carried by all experts collectively, while inference only pays for the ~1/27 that are activated. This is the prerequisite that makes trillion-scale models trainable and deployable in engineering practice.

2.2 Hybrid Attention: Division of Labor Between “Fast Memory” and “Precise Retrieval”

• Gated Incremental Linear Attention: An improved variant within the linear attention family. It compresses historical information into a fixed-size recurrent state, continuously integrating new information with exponential decay. This reduces sequence dimension complexity from O(n²) to nearly O(n), serving the role of “long-range memory.”
• Gated Multi-head Latent Attention: Compresses the Key-Value (KV) cache into a low-dimensional latent space, drastically reducing inference KV Cache memory. It excels at “needle in a haystack” style retrieval that requires precise traceback to the original text.
• Gated Fusion: Dynamically allocates weights between the two memory types, allowing the model itself to decide whether a given content segment should be compressed into memory or precisely retained.

This hybrid design directly determines the inference-side feasibility of a million-token context. Pure softmax attention makes the KV Cache and attention matrix unmanageable for 1M sequences, whereas the “linear state + latent compression” combination brings memory footprint back into an engineering-usable range.

2.3 Attention Residual: Building an “Information Highway” for Deep Networks

As layers stack deeper, the layer-by-layer transmission of information and gradients degrades—a representation bottleneck in deep Transformers. Attention residuals allow shallow attention outputs to directly connect to deeper layers. This alleviates representation degradation caused by depth while reducing the memory footprint of intermediate activations and cross-card communication volume, which is critical for ultra-long context training.

2.4 Capability Positioning

This class of models provides various access methods for general users and developers (online chat, inference API, self-deployment via open-source weights). Typical capabilities cover long-text input, file comprehension, content writing, and data analysis—high-level capabilities entirely underpinned by the three layers of technical breakthroughs discussed below.

3. Challenge 1: Computational Efficiency and Numerical Stability in Ultra-Long Contexts

3.1 The Real Bottleneck Lies Inside the Operators

Linear attention is typically trained using a “chunked parallel” approach: the sequence is divided into chunks, recurrent states propagate between chunks, and matrix-based parallel computation occurs within chunks. The problem is that the recurrence process requires taking the reciprocal of the cumulative decay factor Γ. Under BF16 half-precision, this reciprocal is highly prone to overflow or underflow. To bypass numerical issues, traditional implementations force the diagonal blocks (within chunks) to fall back to explicit position-pair computation paths, failing to leverage hardware dense matrix multiplication acceleration units and severely dragging down throughput.

3.2 Solution: Adding a “Lower Bound” to the Decay Factor

The model changes the log-decay from unbounded negative softplus to a bounded scaled Sigmoid, fixing the lower bound at g_min = -5:

# Traditional parameterization: log-decay is unbounded, reciprocal easily overflows
g_t = -softplus(u_t)          # g_t ∈ (-∞, 0), exp(g_t) can underflow to 0
# Bounded parameterization: scaled sigmoid
g_t = g_min * sigmoid(u_t)    # g_min = -5, g_t ∈ [-5, 0]
# The reciprocal scaling factor exp(-g_t) ∈ [1, e^5 ≈ 148], absolutely safe under BF16

3.3 Gains: A One-Line Change, Triple Returns

The modification is nearly a single line, but the returns are threefold: First, the reciprocal scaling factor within any 16-token sub-block is locked into a safe numerical range, eliminating the need for special computation branches in diagonal blocks. Second, all causal blocks (diagonal and non-diagonal) uniformly utilize dense GEMM kernels, fully saturating hardware matrix acceleration units and significantly boosting overall throughput. Third, the decay mechanism inherently carries relative positional information, supporting direct extrapolation to 1M context under No Position Encoding (NoPE) settings, bypassing the need for specialized positional encoding modifications for ultra-long sequences.

3.4 Curriculum-Based Multi-Stage Context Extension

Million-token context is not trained in a single step. The 8K→64K→256K→1M multi-stage progression is essentially curriculum learning:

Stage	Training Purpose
8K	Solidify foundational language and multimodal capabilities
64K	Introduce long-range dependencies, stabilize attention distribution
256K	Expand retrieval radius, cover document-level tasks
1M	Tackle repository-level comprehension and ultra-long agentic trajectories

Each stage provides stable initialization for the next, avoiding training divergence that would occur if ultra-long sequences were introduced directly.

4. Challenge 2: Training Stability and Load Balancing for 896 Experts

4.1 More Experts, More “Bias”

A classic MoE ailment is routing imbalance: a few experts get overwhelmed by tokens while most remain idle. This wastes parameter capacity at best, and triggers token dropping at worst, causing inconsistencies between training and inference behavior. Scaling the expert count to 896 severely amplifies this issue.

4.2 The Auxiliary Loss Dilemma

A common industry practice is to append a “load balancing auxiliary loss” to the training objective, forcibly penalizing load skew. However, its inherent flaws are clear: the optimization direction of the auxiliary loss does not always align with the main objective (next-token prediction). Tuning the weight too high harms model quality; too low fails to achieve balance, and it introduces additional hyperparameter burdens.

4.3 Quantile Balancing: Moving Balance from Loss Function to Routing Logic

The Quantile Balancing (QB) strategy employed by this model eliminates auxiliary loss. It observes the expert load distribution in real-time and dynamically adjusts routing scores based on quantiles, naturally down-weighting overloaded experts and giving idle experts an opportunity. The balancing logic acts at the routing decision layer rather than in the main loss, preserving the purity of the training objective while achieving dynamic load balancing across a massive scale of experts. Combined with a gated linear unit variant activation function and a latent expert structure optimized for stable training, it jointly ensures training stability under extreme sparsity.

# Traditional approach: auxiliary loss pulls against main loss
loss = lm_loss + alpha * aux_balance_loss
# Quantile balancing (illustrative): dynamic correction at the routing score stage
overload = quantile(expert_load, 0.75)            # Observe load quantile
route_bias += eta * (overload - load) / overload  # Down-weight overloaded, up-weight idle
scores = affinity + route_bias                    # Does not touch main loss

5. Challenge 3: Forging “Long Context” into “Long-Range Capability”

5.1 Pretraining Sets the Ceiling, Post-Training Sets the Floor

Long-context pretraining solves “reading it in,” but agentic tasks demand “getting it done”: multi-step planning, tool calling, and strategy correction based on environmental feedback. These capabilities are difficult to acquire through pure supervised fine-tuning—imitation learning only reproduces expert trajectories and cannot teach the model to explore through failure.

5.2 Million-Token-Level Agentic Reinforcement Learning

The model overlays million-token-level Agentic RL post-training on top of multi-stage pretraining. It allows the model to actually execute multi-step operations within long-horizon task trajectories, receiving reward signals based on final outcomes. This transforms the potential of “ultra-long context understanding” into the capability of “reliably completing long-range tasks.” Pretraining and post-training form a closed loop here: the former pushes the window to the million level, and the latter teaches the model to utilize this window for real tasks. This is the key to its top-tier performance among open-source large models in benchmark testing and its prominent agentic capabilities.

6. Developer Takeaways

1. Architecture Evaluation Requires Operator-Level Scrutiny. A single-line parameterization change unlocking hardware acceleration and million-token extrapolation demonstrates that evaluating large models cannot rely solely on benchmarks. Low-level details like numerical stability and operator implementation efficiency directly dictate training and inference costs.
2. Distinguish “Memory” from “Retrieval” in Long-Context Apps. Hybrid attention provides a crucial hint: different tokens require different intensities of memory strategies. The same applies to the application side. Retrieval-Augmented Generation (RAG), context compression, and native long windows are not mutually exclusive; combining them often yields superior results.
3. Choose Deployment Paths Based on Needs. The combination of open-source weights and technical reports means teams can reproduce and validate based on public benchmarks, or opt for a cloud service route. For example, quickly invoking long-context inference capabilities via cloud model services (like Baidu AI Cloud) for business validation, or self-deploying open-source weights within a container platform, hosting corpora and checkpoints in object storage, and utilizing logging and monitoring services to ensure observability for long-cycle tasks.
4. Focus Evaluation on Long-Horizon Tasks. High benchmark scores do not equal reliability in agentic scenarios. Teams should construct multi-step task sets tailored to their specific business, focusing heavily on instruction-following consistency and tool-calling success rates over long trajectories.

7. Conclusion

The value of this technical blueprint lies not in a single metric, but in demonstrating a clear problem-solving path: resolving numerical stability at the operator level using bounded parameterization, balancing efficiency and capability at the architecture level using hybrid attention and attention residuals, taming 896 experts at the system level using auxiliary-loss-free quantile balancing, and forging “long windows” into “long capabilities” at the training level using curriculum-based context extension and million-level Agentic RL. It is highly anticipated that the combination of “trillion-scale sparsity + hybrid long-range attention + agentic reinforcement learning” will become an increasingly standard configuration for next-generation foundation models.