Key Techniques Behind Trillion-Parameter MoE Models with Million-Token Context

The evolution of large language models in recent years has followed two clear trajectories: parameter efficiency and context length. When ultra-sparse MoE architectures with trillion-scale total parameters and tens of billions of activated parameters are combined with million-token context windows and agentic reinforcement learning (RL) post-training, the capability boundary of the new generation of foundation models is pushed outward once again. Working through the stack in order — attention, MoE, training, and engineering — this article dissects the three problems such architectures must solve.

1. Background: From “Stacking Parameters” to “Sparsification Plus Long Context”

Continuously scaling dense architectures drives nearly linear growth in training and inference costs. MoE architectures take a different route: the total parameter count is enormous, but only a small fraction is activated on each forward pass. This sparsity delivers higher effective capacity under the same compute budget, which is why MoE has become the mainstream choice for ultra-large models — the ratio of total to activated parameters can reach twenty to thirty times. At the same time, agent applications require a model to digest inputs such as code repositories, multi-turn tool-call traces, and very long documents — often hundreds of thousands to millions of tokens — within a single session. Long context has shifted from a “nice to have” to a “must have”.

Pushing context from 8K to 1M, however, is far more than widening the extrapolation range of positional encoding. It simultaneously stresses three fronts: the computational complexity of attention, memory footprint, and training stability. That is exactly why this new generation of architectures demands systematic innovation.

2. Attention Side: A Hybrid Design of Linear and Latent Attention

2.1 Why Linear Attention

The compute and memory cost of standard softmax attention grows quadratically with sequence length, which is unaffordable at the million-token scale. Linear attention rewrites the attention computation into a low-rank, recurrent form, reducing complexity to linear. It maintains a fixed-size state matrix through a gated delta rule, making it a natural fit for streaming and ultra-long-sequence scenarios.

2.2 The Key Pain Point: Numerical Stability Blocks Hardware Acceleration

In practice, linear attention relies on chunked parallelism: the sequence is split into tiles of roughly 16–64 tokens, computation within a tile runs as parallel matrix multiplications, and historical state is passed across tiles with a cumulative decay factor. The trouble lies in the reciprocal of the decay factor. Traditional implementations parameterize the log-decay as an unbounded negative value (for example, the negated output of a Softplus). Under BF16’s limited precision, this reciprocal easily overflows or underflows within a tile, forcing the diagonal blocks to fall back to explicit, position-by-position computation instead of a pure matrix-multiplication path. As a result, utilization of the hardware matrix-multiply acceleration units drops sharply.

2.3 Lower-Bounded Decay Parameterization: A Small Change That Solves a Big Problem

A recent industry approach constrains the log-decay to an interval with a lower bound. The typical technique replaces the unbounded activation with a scaled Sigmoid, where the scaling lower bound is commonly set to -5:

g = g_min · Sigmoid(LinearProjection(h))

# Traditional parameterization: unbounded log-decay; its reciprocal can easily overflow or underflow
g = -F.softplus(self.proj(h))            # range (-inf, 0), uncontrolled
# Lower-bounded parameterization: decay is locked to a safe interval
g_min = -5.0
g = g_min * torch.sigmoid(self.proj(h))  # range [-5, 0], reciprocal is bounded

With this formulation, the reciprocal scaling factors within any 16-token tile stay locked in a safe numerical range, so the diagonal blocks can run entirely on matrix multiplications and the underlying GPU kernels become markedly more efficient. The engineering value is clear: the model structure and parameter count stay almost unchanged — you only swap the activation form and rewrite the kernel — yet you gain both numerical stability and hardware utilization. It is a textbook “low-effort, high-reuse” technique.

2.4 Gated Latent Attention Hybrids and Cross-Layer Residuals

Another notable trend is hybridizing linear attention with multi-head latent attention-style designs. The latter projects keys and values into a low-dimensional latent space to compress the KV cache and uses gating to dynamically regulate information flow, preserving high accuracy on medium-length passages. Linear attention, with its fixed-size state, handles ultra-long-range information compression and memory. The two are combined at the inter-layer or intra-layer level, balancing “close reading of nearby content” with “skimming of distant content”.

Deep networks also face a depth-wise representation bottleneck: information gradually dilutes as it passes through layers. Block-level attention residuals allow the attention output from several layers back to connect directly into deeper layers, enabling cross-layer information sharing and mitigating over-smoothing. As a side benefit, some cross-device activation transfers are eliminated, so memory and communication overhead drop in tandem.

3. MoE Side: Load Balancing in the Era of Massive Experts

3.1 Why Load Balancing Is the Lifeline

MoE performance depends heavily on routing balance. If a few “star experts” are selected too often, they become compute and communication hotspots, while a large number of experts receive insufficient training and their capacity goes to waste. The traditional remedy is an auxiliary balancing loss, but it inherently conflicts with the primary training objective, and its weighting hyperparameter is extremely sensitive: set it too low and it does nothing; set it too high and it damages language capability.

3.2 Auxiliary-Loss-Free Dynamic Balancing

A newer approach removes the auxiliary loss entirely and intervenes directly in the routing logic. When the expert count reaches the hundreds, the goal should not be “perfect balance” but “eliminating extreme skew”. Quantile-based dynamic adjustment is a representative technique: it anchors on a quantile of the in-batch load distribution and only suppresses experts that deviate significantly above the quantile while compensating underused ones. It converges faster and introduces zero interference with the main loss.

# Update the routing bias from selection frequency after each training step (illustrative)
freq = expert_counts / expert_counts.sum()      # actual selection frequency
anchor = torch.quantile(freq, 0.5)              # quantile anchor
bias += lr_b * (anchor - freq)                  # bias down for overheated experts, up for underused ones

3.3 Training Stabilization Techniques

As the expert pool grows, early training becomes prone to routing oscillation and expert collapse. Common stabilization measures include injecting noise into routing scores combined with temperature annealing, orthogonal initialization of expert parameters, and mapping expert representations into a shared latent space to reduce parameter redundancy. Together with an improved gated activation function (a SwiGLU-style variant), these measures keep gradient flow stable under extreme sparsity and reduce loss spikes in the mid and late stages of training.

4. Training Side: Multi-Stage Context Extension and Agentic Post-Training

4.1 Curriculum-Based Context Extension

Pretraining directly at million-token lengths is both inefficient and unstable, so the industry generally adopts a multi-stage curriculum: 8K → 64K → 256K → 1M. Each stage first consolidates base capabilities on shorter sequences, then gradually extends the length while adjusting the data mix in step — adding naturally long samples such as long documents, code repositories, and multi-turn conversations — and, when necessary, adjusting the base frequency of the positional encoding for extrapolation. At every stage transition, long-range retrieval and reasoning metrics should be monitored continuously to avoid a scenario where “length goes up but capability goes down”.

4.2 Agentic RL Post-Training

Pretraining grants the ability to “read long text”; agents need to “act on long text” — tool calls spanning dozens of turns, plus planning and error correction for long-horizon tasks. Post-training therefore needs agent-oriented reinforcement learning, where the environment consists of multi-step interactive tasks and rewards come from final task completion together with verifiable intermediate signals. This raises two challenges: first, long trajectories make credit assignment difficult, requiring a combination of process rewards and outcome rewards; second, producing interaction data is expensive, calling for efficient parallel environments and experience replay mechanisms.

4.3 Recommended Evaluation Dimensions

Evaluating long-context models should not stop at “needle in a haystack” retrieval. Coverage should include four dimensions: multi-hop cross-segment reasoning, consistency of long-document summarization, code-level long-dependency completion, and multi-turn agent task success rates — corresponding to “can find it, can connect it, can understand it, can get it done”.

5. Engineering Recommendations

1. Validate the numerical path of attention kernels first. Before building custom high-performance kernels, run small-scale experiments to confirm the numerical range of the decay factors after chunked parameterization, so overflow is not discovered only days into training.
2. Start load balancing from the auxiliary-loss-free route. When the expert count exceeds 64, give priority to statistics-based dynamic bias methods, which significantly cut hyperparameter search costs.
3. Extend context in curriculum stages. Fix an evaluation set for each stage so that a capability drop does not force expensive retraining.
4. Optimize the two inference bottlenecks separately. In sparse MoE, the memory bottleneck lies in expert weights; in long context, it lies in the KV cache. Plan expert parallelism, cache management, and quantization strategies independently.
5. Leverage mature platforms to lower trial-and-error costs. If a team lacks the conditions for full-stack in-house development, it can validate prototypes with the LLM services, GPU compute, and fine-tuning toolchains offered by established cloud providers such as Baidu AI Cloud, and concentrate its effort on data and application scenarios.

6. Conclusion

Combining ultra-sparse MoE with million-token context is, at its core, a reallocation of the compute budget: away from the uniform spending of dense models and toward structured spending based on “on-demand activation plus hierarchical memory”. Lower-bounded decay parameterization, auxiliary-loss-free load balancing, and cross-layer attention residuals tackle the three toughest problems — putting linear attention into production, training massive expert pools, and breaking the depth-wise representation bottleneck. All three involve minimal modification and are highly reusable, making them well worth evaluating in your own technology stack. As agentic post-training matures, “long context plus strong agents” can be expected to become a standard capability of next-generation foundation models.