Deep Dive: The Next Generation of Ultra-Long Context Sparse MoE LLMs

1. Why This Technical Report is Worth Reading

A recent technical report on a next-generation multimodal large language model has sparked widespread discussion in the community. Stripping away the specific model naming and promotional rhetoric, the true value of this report lies in how it addresses three of the most notoriously difficult engineering challenges in current LLM development within a single unified system:

• Computational efficiency of ultra-long context: The model natively supports a 1 million token context window. According to the report, it can extrapolate directly without positional encoding, placing stringent demands on the efficiency and numerical stability of attention operators.
• Training stability of ultra-large sparse MoE: With total parameters around 2.8 trillion and only about 104 billion activated per token, the sparsity is roughly 3.7%. The number of routed experts reaches the 896 mark. More experts mean higher risks of load imbalance and routing collapse.
• Long-horizon agentic reasoning: The model does not treat long context as an isolated selling point but rather as the foundation for Agentic RL (Agentic Reinforcement Learning) post-training. It performs reinforcement learning on million-token-level task trajectories, truly transforming “reading long” into “getting things done.”

In terms of capabilities, such models typically cover general scenarios like long-document understanding, content generation, and data analysis, and are made accessible to general users and developers via the web and open APIs. However, the true determinant of their ceiling is the solution to the three technical problems outlined below.

2. Architectural Positioning: Reading Design Philosophy from Configurations

Let’s first review the core configurations disclosed in the report: 2.8 trillion total parameters with approximately 104 billion activated per token; native visual multimodal input; a 1M token context window; attention composed of a hybrid of Gated Multi-head Latent Attention and incremental linear attention, incorporating cross-layer attention residuals; an MoE side with 896 routed experts, equipped with a latent space expert structure and auxiliary-loss-free balancing; and training utilizing an 8K→64K→256K→1M multi-stage curriculum, with Agentic RL post-training executed on million-token-level trajectories.

From these configurations, three clear design orientations emerge:

1. Trading extreme sparsity for capacity: Each token invokes only about 3.7% of the network capacity. Sparsity is used to buy knowledge and multimodal capacity, while the activation volume controls inference costs.
2. Hybrid attention for efficiency: A few critical layers retain precise global attention, while the majority use low-cost linear attention, allowing “fast” and “accurate” to each serve their purpose.
3. Structural stability design: Whether it’s the cross-layer residual sharing on the attention side or the latent expert space and auxiliary-loss-free balancing on the MoE side, all point to the same goal—preventing training explosions at extreme scales.

3. Challenge 1: Attention Efficiency under 1M Token Context

3.1 Why Softmax Attention Cannot Survive up to 1M

Standard attention has two unavoidable costs. On the computation side, the attention matrix scales at O(n²); 1M tokens imply a matrix on the scale of 1M×1M, and even with chunking, the overhead is unbearable. On the storage side, the KV Cache expands linearly with sequence length. Long documents or code repository-level inputs will cause the VRAM budget of inference services to spiral out of control rapidly. The industry has thus developed two paths: latent compression attention that compresses KV, and linear attention (incremental attention) that rewrites the attention form. The former preserves accuracy but has limited compression ratios, while the latter ensures efficiency but is weak in exact recall. The report’s approach is to mix the two across layers.

3.2 The Old Problem of Linear Attention: Numerical Stability

For linear attention to achieve engineering-usable throughput, chunked parallelism is essential: the sequence is sliced into fixed-length tiles (e.g., 16 tokens per chunk), with intra-chunk parallelism and inter-chunk recursion. The problem lies in the computation of the intra-chunk diagonal blocks. Incremental linear attention carries a decay factor that accumulates with position, and intra-chunk normalization requires its reciprocal. Traditional implementations use unbounded negative softplus parameterization for log-decay, meaning the decay value can arbitrarily approach zero, causing its reciprocal to overflow or underflow in low-precision formats like BF16. To bypass numerical issues, previous implementations had to open a “special channel” for diagonal blocks, degrading into an explicit per-position computation path and abandoning matrixization. The result was that even though most computations were already dense matrix multiplications, the remaining diagonal blocks consumed a disproportionate amount of wall-clock time, keeping the utilization of the GPU’s dense GEMM units persistently low.

3.3 Bounded Decay Parameterization: Minimal Change, Significant Gains

The solution provided in the report is conceptually clean: add a lower bound to the decay term. Changing log-decay from unbounded negative softplus to scaled Sigmoid:

g_t = g_min · σ(e_A · h_t), where g_min is fixed at −5

Since the Sigmoid output naturally falls within the (0,1) interval, g_t is constrained to [−5, 0], corresponding to a per-step decay coefficient exp(g_t) in the range of approximately [0.0067, 1]. The reciprocal scaling factor within any 16-token tile is thereby locked in a numerically safe range.

# Traditional parameterization: unbounded, reciprocal prone to overflow (illustrative)
g = -F.softplus(w_a * h)              # g ∈ (-∞, 0)
# Bounded decay parameterization: bounded, numerically safe (illustrative)
G_MIN = -5.0
g = G_MIN * torch.sigmoid(w_a * h)    # g ∈ [G_MIN, 0]

The benefits can be summarized in three points:

• Complete removal of the diagonal block special branch: Whether for diagonal or non-diagonal blocks, all causal tiles can directly invoke dense matrix multiplication (GEMM) kernels, significantly improving hardware utilization and throughput.
• Minimal implementation change: It only touches the parameterization function of the decay term without altering the overall mathematical form of attention, resulting in a very low porting cost for existing operator libraries.
• Simplified length extension pipeline: Bounded decay does not depend on fine-grained positional encoding schemes. Combined with the absence of positional encoding, it can directly train and extrapolate on 1M context.

This is the most worthwhile point in the entire report for operator engineering teams to reuse—a classic “one line of math for a tier of performance” change.

3.4 Division of Labor in the Hybrid Architecture

It must be emphasized that linear attention is not omnipotent: it excels at compressing history into a fixed-size state with a cost roughly linear to the sequence length, but it is naturally weak at “needle in a haystack” precise retrieval. Therefore, such solutions typically adopt a ratio where a few layers retain Gated Multi-head Latent Attention while the majority use incremental linear attention. For teams planning to develop their own long-context models, this inter-layer ratio is itself one of the hyperparameters most worth conducting ablation studies on.

4. Challenge 2: Load Balancing for 896 Experts

4.1 The Bottleneck of the Auxiliary Loss Route

The fundamental contradiction of MoE is that routing is a discrete decision, and gradients cannot directly pass through it. The industry has long relied on “auxiliary losses”—adding a penalty term to the main loss to force load uniformity. However, auxiliary losses have three long-standing issues: gradient conflicts with the main objective, where better balancing might sacrifice more modeling capability; extreme sensitivity to balancing coefficients, leading to high tuning costs; and when the number of experts scales to nearly a thousand, the indirect penalty signal becomes increasingly difficult to finely control each expert’s load.

4.2 Auxiliary-Loss-Free Quantile Balancing

The report adopts an auxiliary-loss-free dynamic balancing approach: instead of adding any term to the loss function, it directly manipulates the routing scores. It maintains a routing bias for each expert, dynamically adjusted during training based on the quantile of the expert’s actual load: overloaded experts have their selection probability reduced, while underloaded experts have theirs increased.

# Auxiliary-loss-free load balancing (mechanism illustration, not a specific implementation)
bias = torch.zeros(num_experts)          # One routing bias per expert
for batch in loader:
    scores = gate(x) + bias              # Bias only participates in "who is selected", not in weighting
    topk = scores.topk(k)                # Top-K routing
    load = count_tokens_per_expert()     # Count actual load within the window
    q = torch.quantile(load, q_target)   # Quantile threshold for overload/underload
    bias[load > q] -= delta              # Decrease weight for overloaded experts
    bias[load < q] += delta              # Increase weight for underloaded experts

Compared to auxiliary losses, the main objective here remains completely pure, and balancing and capability no longer constrain each other. There are no extra balancing coefficients to repeatedly sweep. The bias adjustment is a smooth feedback control, remaining robust even at a scale of nearly a thousand experts. It does introduce new hyperparameters (adjustment step size, statistics window) and is not a free lunch, but engineering-wise, it is usually much cheaper than sweeping auxiliary loss coefficients.

4.3 Supporting Stability Designs

Routing balancing is only one necessary condition for stability. The report highlights two other notable supporting designs on the MoE side: first, expert computation is arranged in a latent space (compress before routing), reducing routing noise and cutting down VRAM and communication volume on the expert side; second, GLU-like gated activation functions are used to transform the feed-forward layers, improving the expressive efficiency per unit of activated parameter through the gating structure. Together with load balancing, these form the complete puzzle of “training stability under extreme sparsity.”

5. Challenge 3: Deep Representation Bottleneck and Attention Residuals

5.1 Depth is Both Capability and Bottleneck

Making models deeper is a universal means to improve representation capabilities, but deep networks suffer from information decay: features extracted by layers near the input may be diluted after passing through dozens of layers, and gradients face the same long trek. Standard residual connections mitigate this problem, but in networks with over a hundred layers, the serial dependency of “each layer starting from the full output of the previous layer” also drives up memory and communication overhead.

5.2 Block-Level Attention Residual

The report’s approach is to introduce cross-layer residual sharing between attention modules, equivalent to building “elevated highways” for deep networks. Its value is threefold: it alleviates the representation bottleneck brought by depth, allowing long-range information to bypass intermediate layers and reach later stages directly; some activations can be reused, reducing memory footprint; in distributed training, cross-layer sharing also means less communication, tangibly reducing bandwidth pressure for pipeline parallelism. This technique is strongly coupled with 1M context training—ultra-long sequence training is already memory-intensive, and every bit of activation memory saved can be converted into longer sequences or larger batches.

6. Training Paradigm: Curriculum Expansion and Agentic RL

6.1 8K → 64K → 256K → 1M Curriculum Pre-training

Long-context capability is not trained in a single step. The standard industry practice is curriculum progression: first solidify language and multimodal foundational capabilities on short sequences, then gradually expand the window in stages. The path disclosed in the report consists of four stages: 8K, 64K, 256K, and 1M. The significance of staging is that the data distribution and optimizer state remain relatively stable at each phase, avoiding loss spikes caused by directly jumping to ultra-long sequences. Simultaneously, it requires common techniques like data replay and mixed streaming of long and short samples to prevent the general capabilities accumulated in short stages from being washed away in long stages. On the multimodal side, document-level data with interleaved text and images (long reports, image-text tutorials, code mixed with screenshots) is the natural intersection of long-context and multimodal capabilities.

6.2 Million-Token-Level Agentic RL Post-Training

If long-context pre-training solves “reading it in,” Agentic RL solves “getting it done.” Agent tasks are characterized by trajectories that often span dozens of tool calls, tens of thousands to a million tokens of observations and intermediate results, with sparse and delayed rewards. Performing reinforcement learning on trajectories of this length is an extreme test for sampling throughput, memory management, and credit assignment. Building RL post-training directly on a 1M context foundation is logically consistent: a major root cause of agent failure is that “key information is drowned or forgotten in the long context.” Only when the foundation genuinely possesses stable million-token perception can the reward signals of long-horizon tasks be meaningful. This also explains why the report deliberately binds long context and Agentic RL as a single main thread, rather than two independent selling points.

7. Engineering Reuse Checklist and Rational Judgment

7.1 Three Most Worthwhile Technical Points to Reuse

For teams engaged in LLM training and inference infrastructure, the three highest reuse values in this report are:

1. Bounded decay parameterization: Minimal change that directly improves the implementation efficiency of linear attention operators on GPU dense matrix multiplications. Almost any team developing their own long-context model can learn from this.
2. Auxiliary-loss-free quantile load balancing: Its value becomes more apparent as the expert scale grows. It can be combined with mainstream MoE training frameworks for validation.
3. Block-level attention residual: Alleviates deep bottlenecks and reduces memory communication overhead with minimal structural changes, making it especially suitable for ultra-long sequence training scenarios.

7.2 How to Rationally View Report Conclusions

It is necessary to stay clear-headed: a technical report presents the “author’s perspective of the optimal chain of evidence.” The report claims the model is in the first tier on multiple public benchmarks, but leading benchmarks does not equate to leading in scenarios—long-document QA, code repository understanding, and multi-turn agent tasks each assess very different capability dimensions. Furthermore, open weights do not mean full reproducibility; details regarding data mixing and hyperparameter sweeps are often ambiguous. A rational approach is to treat it as a high-quality design reference, conducting small-scale replication and ablation based on your own workload characteristics, rather than blindly copying the conclusions.

7.3 A Pragmatic Path for Application Teams

For the vast majority of application teams, reproducing pre-training from scratch is unrealistic. A more pragmatic path is to validate business value on top of existing capabilities: prioritize using cloud-hosted model services to directly evaluate target scenarios like long-document understanding, multimodal information extraction, and multi-turn tool calling, then decide whether to invest in building from scratch. Platforms like Baidu AI Cloud provide large model services, alongside general components like object storage, container compute, logging, and monitoring alerts, allowing you to set up a complete pipeline of “data hosting — batch evaluation — performance monitoring” at a low cost. When evaluating, it is recommended to focus on testing three things: key information recall rate under long context, structured extraction accuracy for multimodal interleaved documents, and end-to-end completion rate for long-horizon tasks, rather than just looking at general leaderboard scores.

8. Conclusion

The new generation of work represented by this report signifies that LLM competition has shifted from “stacking parameters” to a systems engineering phase that “simultaneously manages length, sparsity, and agentic capabilities.” Bounded decay parameterization, auxiliary-loss-free load balancing, and block-level attention residuals are individually not complex. The difficulty lies in assembling them with curriculum training and Agentic RL into a cohesive whole that remains stable at a scale of 2.8 trillion parameters, 896 experts, and a 1M context. For practitioners, understanding the constraint derivations behind these designs is more important than chasing any specific model—after all, constraints dictate architecture, and architecture dictates the ceiling.