Core Technologies Behind Ultra-Long-Context Sparse MoE Large Models

1. Background: Two Evolution Lines in Model Architecture Are Converging

Recent progress in foundation models follows two clear trajectories. The first is the continuous scaling of parameter counts—from hundreds of billions toward trillions—while keeping per-token activated parameters at the tens-of-billions level through fine-grained experts and extreme sparse activation, striking a balance between compute budgets and model capacity. The second is the expansion of context windows from tens of thousands of tokens to millions, where long context has evolved from a supporting feature for retrieval-augmented generation into a native foundation for agentic long-horizon reasoning, multi-turn tool use, and cross-document information synthesis.

When ultra-sparse MoE and million-token context become simultaneous design goals, three coupled technical tensions immediately surface:

1. Computational efficiency: Standard full attention incurs quadratic compute and memory costs with sequence length. Million-token contexts cannot rely on dense attention alone, necessitating sub-quadratic operators such as linear attention—but these often suffer from poor hardware utilization in practice.
2. Training stability: As expert counts grow from dozens to hundreds or thousands, routing collapse and load imbalance are amplified, becoming a critical determinant of training stability.
3. Deep representation barriers: Supporting more experts and longer memory requires deeper networks, which introduces representation bottlenecks, activation memory pressure, and rising cross-device communication overhead.

Post-training paradigms are also driving architectural evolution. Agentic reinforcement learning (RL) requires computing rewards and updating gradients over task trajectories spanning millions of tokens, demanding numerical stability and compute efficiency at extreme lengths. Recent public work generally organizes solutions around three pillars: hybrid stacks of linear attention with gated full attention, sparse MoE with massive expert counts, and multi-stage context expansion. Among these, three techniques stand out for their reuse potential:

Technique	Core Problem Addressed	Reuse Value
Bounded decay parameterization	Linear attention numerical overflow and low hardware utilization	High: minimal modification, directly portable
Quantile balancing (QB)	Load balance and training stability with nearly a thousand experts	High: zero gradient interference, no auxiliary loss
Block-level attention residuals	Deep network representation bottlenecks and memory/communication costs	Medium-high: naturally compatible with hybrid architectures

2. Technique 1: Bounded Decay Parameterization—Making Linear Attention Fully Utilize Matrix Accelerators

2.1 The Problem

Incremental linear attention compresses all history into a fixed-size matrix state and forgets old information via per-token decay factors. For parallel training on long sequences, implementations typically use chunked (tiled) computation—for example, tiles of 16 tokens. Off-diagonal blocks handle cross-chunk information flow (“historical state → current token”), while diagonal blocks handle intra-tile token interactions.

Diagonal block computation involves a reciprocal scaling factor of the form 1/(1−e^{Σg}), where Σg is the accumulated log-decay within a tile. The issue: if log-decay is produced by an unbounded function such as negative softplus, Σg can accumulate extreme values under limited precision (BF16), causing severe overflow or underflow in the reciprocal. To avoid numerical failures, traditional implementations must retain an explicit scalar position-pair computation path for diagonal blocks, bypassing matrix multiplication units. This directly causes significant operator throughput loss on long sequences—one of the classic reasons linear attention is “cheap in theory but slow in practice.”

2.2 The Solution: Locking Decay in a Safe Range

The core modification is deliberately restrained: replace the unbounded negative softplus parameterization of log-decay with a Scaled Sigmoid that has a lower bound:

g_t = g_min · σ(α · h_t), where g_min is a fixed negative value (e.g., −5)

This constrains each step’s decay to the interval (e^{g_min}, 1), keeping accumulated decay—and its reciprocal scaling factor—within any 16-token tile numerically safe. Overflow and underflow are eliminated at the source.

# Illustration: bounded log-decay parameterization
log_decay = g_min * sigmoid(scale * h)   # g_min = -5.0
# log_decay ranges in (-5, 0), corresponding to decay factors in (e^-5, 1)
# Accumulated decay within any 16-token tile is bounded,
# so its reciprocal scaling factor cannot overflow or underflow

2.3 Benefits and Reuse Value

• Higher throughput: The special computation branch for diagonal blocks is eliminated. All causal tiles—diagonal and off-diagonal—can uniformly call dense GEMM, fully mapped to hardware matrix multiplication accelerators and high-performance operator libraries, substantially improving long-sequence operator throughput.
• Numerical stability: No guard branches or reduced-precision fallbacks are needed under BF16 mixed-precision training.
• Long-context extrapolation friendly: The decay factor implicitly encodes positional information. This parameterization natively supports direct extrapolation to million-token contexts under a No Positional Encoding (NoPE) setup, removing the need for position interpolation or base-frequency tuning.
• Minimal modification cost: Only a single parameterization expression changes. It is portable to any incremental linear attention implementation—a classic “one-line change, global benefit” case.

3. Technique 2: Auxiliary-Loss-Free Quantile Load Balancing—A Stabilizer for Large-Scale Expert Networks

3.1 Why Auxiliary Losses Are No Longer Sufficient

The classic MoE challenge is router self-reinforcement: experts that gain an early advantage attract more tokens, eventually causing a few experts to overload while most sit idle, degrading the model’s effective capacity. The standard industry remedy is an auxiliary balancing loss, but it has two structural flaws. First, the auxiliary objective’s gradient direction often conflicts with the primary objective (language modeling or multimodal modeling); a slightly larger weight harms quality, while a slightly smaller weight fails to balance—making the hyperparameter extremely sensitive. Second, when expert counts scale to hundreds or thousands, a single global loss term cannot achieve fine-grained, differentiated adjustments across experts.

3.2 How Quantile Balancing Works

Quantile Balancing (QB) moves “balancing” entirely out of the loss function and instead applies it as a dynamic bias on routing scores, creating a statistical feedback loop:

1. Collect each expert’s actual token load within a statistical window.
2. Compute high and low quantile thresholds of the load distribution.
3. Experts with load above the high quantile have their routing bias decreased; those below the low quantile have it increased.
4. The bias participates only in routing score computation, not in gradient updates, keeping the main loss clean.

# Illustration: quantile balancing (QB) core loop
load       = token counts per expert in the statistical window
q_hi, q_lo = quantile(load, 0.9), quantile(load, 0.1)
bias[load > q_hi] -= eta        # overloaded experts: lower routing bias
bias[load < q_lo] += eta        # idle experts: raise routing bias
logits      = router_scores + bias
# Key: bias does not participate in gradient updates, preserving the main loss

3.3 Comparison with Auxiliary Loss Balancing

Dimension	Auxiliary Loss Balancing	Quantile Balancing (QB)
Gradient interference	Conflicting with the main objective	Zero interference
Hyperparameter count	Weight coefficient hard to tune	Quantile thresholds and step size intuitive and controllable
Response mode	Indirect, lagged via gradients	Direct, immediate via statistical feedback
Scalability to large expert counts	High tuning cost	Naturally suited to hundreds to thousands of experts

Under extreme sparsity configurations, QB maintains dynamic load balance across experts without sacrificing main-task quality—a direct answer to stabilization in massive-expert MoE training. Higher expert utilization also means greater effective model capacity under the same activated parameter budget.

4. Technique 3: Block-Level Attention Residuals—Shortcuts for Deep Networks

Increasing network depth brings two familiar problems. First is the deep representation bottleneck: attention outputs must be re-distilled layer by layer, and retrieval information captured in shallow layers gets diluted through the stack, yielding diminishing returns from added depth. Second is system overhead: the activation memory and cross-device communication required for each layer’s attention output grow linearly with depth, making memory and communication scarce resources when training on million-token contexts.

Block Attention Residuals establish cross-layer information paths between attention layers at the block granularity: attention outputs are shared and accumulated across layers by block, allowing deep layers to directly reuse shallow-layer attention results rather than re-deriving them from scratch. The benefits are compounded:

• Reduced depth bottleneck: Information gains a high-speed path that bypasses nonlinear stacking, benefiting both convergence and expressiveness of deep networks.
• Lower memory and communication costs: Cross-layer sharing reduces activations that must be repeatedly stored and transmitted, significantly improving system economics for ultra-long-context training.
• Strong architectural compatibility: Naturally compatible with hybrid stacks of linear attention layers and full attention layers, without altering the computational semantics of the operators themselves.

5. Training Paradigm: Multi-Stage Context Expansion and Agentic Reinforcement Learning

5.1 Staged Context Expansion

A million-token context is not trained in one step. The industry’s reliable path is multi-stage progression, typically following a ladder such as 8K → 64K → 256K → 1M:

• Each stage’s context multiplier is moderate, giving optimizers and routing systems time to adapt and avoiding loss spikes from one-shot jumps.
• Data mixing ratios evolve in parallel, with long-document, multi-document reasoning, and repository-level code data accounting for a progressively larger share.
• Combined with NoPE or decay-implied positional designs, extrapolation risk is further reduced.

5.2 Million-Token-Level Agentic RL

In the post-training phase, long-horizon agentic tasks—multi-turn tool calls, cross-document information synthesis, long-chain planning—serve as the training environment, with rewards computed and policies updated over trajectories of millions of tokens. This is feasible only because the three technical innovations above form a closed loop: without linear attention and block-level residuals providing efficiency and memory economy, the compute cost of million-token trajectories would be prohibitive; without QB’s load balancing, the high-intensity updates of large-scale expert networks during RL would fail to stabilize. Architecture efficiency, training stability, and post-training paradigms converge here in a rare state of coherence.

5.3 Native Multimodality

On the main backbone, visual tokens and text tokens are uniformly encoded and fed into the same attention and MoE parameters, rather than appended through a bolt-on module, enabling more coherent cross-modal long-horizon reasoning. Full attention layers can adopt gated Multi-head Latent Attention (Gated MLA), which compresses KV cache while preserving precise retrieval capability, reducing serving costs for long-context deployment.

6. Engineering Takeaways for Developers

1. Prioritize numerical boundaries over operator sophistication: For any operator involving reciprocals, divisions, or cumulative exponentials, first constrain the numerical range through parameterization, then optimize hardware utilization.
2. Move balancing mechanisms out of the loss function when possible: Dynamic bias plus quantile feedback is a low-cost, low-interference, and scalable solution for large expert counts.
3. Expand context in staged increments: A path starting from 8K and scaling up gradually is far more robust than one-shot jumps.
4. Evaluate NoPE-style designs seriously: Letting mechanisms like decay implicitly carry positional information can significantly simplify extrapolation engineering.
5. Plan context management early for agent post-training: Reward design, trajectory sampling, context truncation, and compression strategies determine the actual gains from the RL phase.
6. Choose wisely between building and borrowing: Training trillion-parameter models exceeds the budget of most teams. A more practical path is to access long-context, multimodal, and agentic capabilities via mature large-model cloud services—for example, through API-based offerings from providers like Baidu AI Cloud—and concentrate in-house efforts on business orchestration, data feedback loops, and evaluation systems.

7. Conclusion

Looking across this generation of frontier work, three clear threads emerge. At the operator level, bounded decay parameterization resolves numerical stability and unlocks hardware throughput. At the system level, auxiliary-loss-free quantile balancing and block-level attention residuals jointly secure training stability and memory/communication economy. At the paradigm level, multi-stage context expansion and million-token-level agentic reinforcement learning transform long context from a benchmark metric into a trainable agentic capability. It is reasonable to expect that the combination of ultra-sparse MoE, hybrid attention, and long-context agentic post-training will become the default recipe for next-generation foundation models. For developers at large, understanding the engineering logic behind these three layers offers more lasting value than memorizing any single model’s parameter table.