Ultra-Sparse MoE Models with Million-Token Contexts: A Technical Breakdown
This article analyzes the engineering mechanisms behind a new generation of ultra-sparse mixture-of-experts (MoE) large language models designed for million-token contexts. It focuses on three practical problems: achieving both long-range efficiency and precise retrieval with hybrid attention, training massive sparse expert systems stably, and extending context and agentic task capabilities through staged training and post-training. The discussion includes numerically stable decay parameterization, auxiliary-loss-free load balancing, deployment trade-offs, and an evaluation checklist for moving from benchmarks to production.
Introduction: What Should Developers Look at in New-Generation Large Models?
Every so often, a new large model starts a wave of discussion. These models usually emphasize three things at the same time: parameter scale, context length, and agentic capabilities. A common description is that the model “supports long-text input and can perform document understanding, content generation, and data analysis,” followed by a long list of architecture terms and benchmark scores.
The hype eventually fades. What developers really need to answer is: which mechanisms are worth borrowing, which capabilities can actually be applied to business workloads, and what costs will be incurred?
This article does not benchmark any specific product. Instead, it abstracts the public technical information of this class of models into three reusable engineering questions:
How can attention balance long-range efficiency and precise retrieval?
This maps to “hybrid linear attention + numerically stable decay parameterization.”How can ultra-large-scale sparse experts be trained stably?
This maps to “auxiliary-loss-free load balancing + activation function design + block-level attention residuals.”How can a model push context and tasks toward the long range?
This maps to “multi-stage context extension + agentic reinforcement learning post-training.”
For each question, this article explains the underlying principle, key implementation points, and evaluation methods. Any specific configurations mentioned below come from public technical reports and are discussed only for technical analysis.
1. Technical Overview: One Main Line and Three Engineering Problems
1.1 Overall Configuration: Sparsification, Native Multimodality, and Million-Token Context
A typical configuration for this new generation of models looks like this:
| Dimension | Typical Configuration | Engineering Implication |
|---|---|---|
| Parameter scale | Total parameters around 2.8 trillion; about 104 billion activated per token, with roughly 96% sparsity | Compute per token is about 3.7% of total parameters; memory usage and expert communication still scale with total parameters |
| Modality | Native multimodal: vision and text are jointly pretrained | Reduces the cross-modal alignment loss introduced by separate visual adapters |
| Context | Native support for 1 million tokens, using a no-positional-encoding design | Mitigates position-encoding extrapolation issues, but imposes new requirements on chunked attention and state management |
| Attention | Gated linear attention with incremental decay parameterization, interleaved by layer with multi-head latent attention; block-level attention residuals enable cross-layer information sharing | Trades off between long-sequence efficiency and precise retrieval |
| MoE | Routed experts plus shared experts; uses activation functions from the SwiGLU family | Combines dense-channel semantics with dynamic sparse computation |
| Training pipeline | Multi-stage pretraining from 8K → 64K → 256K → 1M, followed by million-token-level agentic reinforcement learning post-training | Long-range capabilities are cultivated stage by stage |
1.2 Three Engineering Problems
Computational efficiency for ultra-long context
Attention cost grows quadratically with sequence length. For example, compared with 64K tokens, the attention computation for 128K tokens is about 4 times as large, while 256K tokens is about 16 times as large. At the same time, KV-cache memory grows linearly with length. If every token goes through full attention, both training and inference become difficult to scale.Training stability with a massive number of experts
The number of routed experts can reach hundreds or even thousands. Expert granularity is often finer: each expert has fewer parameters, and there are more experts. This helps balance compositional expressiveness and training efficiency. However, load skew can cause some experts to be undertrained or even “die.” Routing collapse, numerical instability, and communication hotspots follow.Long-range reasoning for agents
Multi-step tasks often involve dozens or even hundreds of tool calls. Critical evidence may appear hundreds of thousands of tokens earlier in the context. This requires combining long-context capability with task planning. The post-training stage must also prepare for long trajectories and sparse rewards.
1.3 How to Evaluate an Architecture
When facing a list of architecture terms, four questions can quickly reveal the technical substance:
Is there ablation evidence?
When a mechanism is removed, how much do the metrics drop?Are the costs transparent?
How do memory, communication, latency, and engineering complexity change?Is it reusable?
Can it be transferred to the pretraining or post-training pipeline of other models?Where are the capability boundaries?
Does the designer explain suitable scenarios and known limitations?
2. Attention Design: Hybrid Architecture and Numerically Stable Decay Parameterization
2.1 Linear Attention: The “Forgetting” Problem Behind Linear Cost
Standard attention needs to store the full K/V cache. Linear attention rewrites attention as a recurrent update of a matrix state, reducing the complexity over sequence length to linear:
S_t = A_t ⊙ S_{t-1} + K_t V_tᵀ, O_t = S_tᵀ Q_t
Where:
S_tis the state matrix under a single-head view, with shaped × d, acting as a compressible “memory.”K_t,V_t, andQ_tare alld-dimensional vectors.A_tis a channel-wise decay coefficient, also called a forgetting gate.⊙denotes element-wise multiplication.
When A_t = 1, this corresponds to linear attention without gating. When A_t = 0, it corresponds to fast-weight updates without a forgetting mechanism. Common gated linear attention introduces an input-dependent A_t, allowing the model to decide what to remember and what to forget.
There are two main problems:
Limited state capacity
A fixed-size state cannot fully preserve every detail across millions of tokens.Numerical stability for chunked parallelism
To run efficiently on GPU matrix-multiply units, a sequence is usually split into chunks and computed in parallel. The problem is that decay relationships between positions inside a chunk need to be computed “separately first, then divided.” If the decay exceeds the range of low-precision floating point, this division can overflow or underflow.
2.2 Numerical Risk in Chunked Computation
Assume the chunk length is C. For position i inside a chunk, the cumulative log decay is L_i ≤ 0, and u_i = exp(L_i) ∈ (0, 1].
The attention score and output inside a chunk can be written as:
A_ij = (Q_i · K_j) · u_i / u_j (j ≤ i), O_i = Σ_j A_ij V_j
If we absorb u_i into Q and 1 / u_j into K, the entire chunk can be computed with one matrix multiplication:
Ã_ij = (u_i · Q_i) · (K_j / u_j)ᵀ
If there is no lower bound on decay, the log decay for a single token can become extremely steep. For BF16, with relative precision around 0.4% and maximum value around 3.4×10³⁸:
| Maximum Absolute Cumulative Log Decay Within a Chunk | Maximum Value of 1/u | BF16 Behavior |
|---|---|---|
| ≈ 3 | ≈ 20 | Safe |
| ≈ 16 | ≈ 10⁷ | Safe |
| ≈ 80 | ≈ 5.5×10³⁴ | Near the limit, with little headroom |
| ≈ 90 | ≈ 1.2×10³⁹ | Overflows to infinity; the chunk loses precision |
Both directions carry risk:
- The
K / uside can overflow. - The
u_i · Q_iside can underflow to zero under steep decay, causing distant tokens to be “hard forgotten.”
More importantly, once overflow happens, diagonal chunks can no longer use matrix multiplication and must fall back to per-position broadcast multiply-add operations. These operations are limited by memory bandwidth and can cancel out the compute-density advantage of linear attention.
2.3 Lower-Bounded Decay Parameterization: A Quantifiable Solution
Some public technical reports describe a solution that parameterizes log decay with a lower bound:
g(x) = g_min · Sigmoid(a · xᵀz + b), where g_min = -5 is fixed
This replaces the original unbounded negative softplus form with a sigmoid-based form. The mathematical comparison is:
| Parameterization | Output Range | Lower Bound | Upper Bound on 1/u Within a Chunk (C=16) | BF16 Headroom |
|---|---|---|---|---|
| Negative softplus | (−∞, 0) | None | No upper bound | May overflow |
| Scaled sigmoid | [−5, 0) | −5 | e⁸⁰ ≈ 5.5×10³⁴ | About 1.6 orders of magnitude |
The scaled sigmoid form brings three benefits:
Predictable numerical behavior
For a chunk lengthC,1/u ≤ exp(|g_min| · C). Under BF16,|g_min| · Cshould not exceed about 88, or about 11 under FP16. With|g_min| = 5andC = 16, the headroom exceeds one order of magnitude.Unified operators
Both diagonal and non-diagonal chunks can use dense matrix multiplication. This removes special branches and avoids extra element-wise corrections such as logarithms and exponentials.Clear physical meaning
Single-step decay is constrained to preserve at least about 0.7% of historical information. It will neither remember indefinitely nor instantly clear the state.
Unlike rotary position encoding, this decay does not depend on wavelengths tied to specific sequence lengths. In theory, as long as the model has seen similar lengths during training, there is no explicit extrapolation mismatch. Combined with a no-positional-encoding design, it can also save substantial positional computation for ultra-long contexts.
2.4 Reference Implementation: Core Logic of Chunked Linear Attention
The following illustrative code preserves the core logic of chunked linear attention. It omits masking details, normalization, grouped-query attention, and other engineering parts, and focuses on numerical handling.
import torchG_MIN = -5.0 # Lower bound for log decayBLOCK = 16 # Block size; under BF16, |G_MIN| * BLOCK <= 88def chunk_linear_attention(q, k, v, h, A, g):"""Chunked parallel form of gated linear attention (single head, scalar decay)q, k, v : [T, d], computed in fp32h : [d, d], initial matrix state (can come from the previous segment)A : [T, 1], write gate for fast-weight updateg : [T, 1], log decay, guaranteed g_min <= g < 0Returns o and the updated state h"""T, d = q.shapeq, k, v = q.float(), k.float(), v.float()# 1. Cumulative log decay within each block, with numerical safety checks# BF16: |g_min| * BLOCK <= 88; FP16: <= 11assert T % BLOCK == 0 and abs(G_MIN) * BLOCK <= 88g_block = g.view(-1, BLOCK, 1)L = g_block.cumsum(dim=1) # cumulative within block, always <= 0u = torch.exp(L) # (0, 1]# If mixed with an unbounded parameterization, extra checks are needed# to ensure u does not underflow to 0assert torch.isfinite(u).all() and (u > 0).all()u_last = u[:, -1:] # cumulative decay at the end of each block# 2. Row scaling and column scaling: avoid any special branch for diagonal blocksq_s = q.view(-1, BLOCK, d) * u # multiply by u_ik_s = k.view(-1, BLOCK, d) / u # multiply by 1 / u_jv_s = v.view(-1, BLOCK, d)a_s = A.view(-1, BLOCK, 1)# 3. Intra-block and cross-block computationo = torch.empty_like(q_s)for b in range(q_s.shape[0]):qb, ks_b, vs_b = q_s[b], k_s[b], v_s[b]# Cross-block contribution: multiply by the state first,# then by the block-end cumulative decayo[b] = (qb @ h) * u_last[b]# Intra-block contribution: one dense matrix multiplication with masking;# no special branchcausal = torch.ones(BLOCK, BLOCK, device=q.device).tril().bool()attn = (qb @ ks_b.transpose(0, 1)).masked_fill(~causal, 0.0)o[b] += attn @ vs_b# Write gate + block-end cumulative decay: update the state in# normalized form to offset magnitude driftnorm = (a_s[b] * u_last[b]).sum().clamp_min(1e-6)h = (u_last[b] * h + (a_s[b] * k_s[b]).transpose(0, 1) @ vs_b) / normreturn o.reshape(T, d), h
A few practical points matter during implementation:
Precision division of labor
Low-precision tensor cores handle matrix multiplication, while fp32 handles accumulation and state maintenance. Decay terms, cumulative terms, and1/ushould always be computed in fp32, then converted back to low precision when needed.Dynamic range checks
As shown by the assertions in the code, before going live, teams should perform quantile statistics on the input distribution to confirm that cumulative decay stays within a safe range.Gating dimension
If the decay is channel-wise, meaning ad-dimensional vector rather than a scalar, the row scaling and column scaling above no longer hold. The channel-wise scaling must be fused into the operator; otherwise, the implementation still falls back to the per-position path.Block size is tunable
block_sizeandg_minjointly determine the numerical safety region. They should be tuned as hyperparameters together with the precision strategy.
2.5 Multi-Head Latent Attention and Output Gating
Linear attention is good at long-range compression, but it is not necessarily strong at precise retrieval. A common approach is to insert a global attention layer every few layers, forming a hybrid architecture.
The global layer can use a design based on “low-rank latent cache + output gating”:
C_t = W_c x_t # compressed latent cache (optional RMSNorm)O_t = (W_g x_t) ⊙ (W_o [C_t; H_t]) # output gating with element-wise sigmoid
Engineering teams need to balance two aspects:
Inference side
Compressed caches can significantly reduce KV-cache memory usage and memory access during decoding. However, the up-projection introduces additional operators. When integrating with inference-system capabilities such as paged cache and prefix reuse, operator fusion is required.Training side
The newly added projection matrices should use scaled initialization, where the initialization standard deviation is related to the input dimension. The output layer should contribute close to zero on the residual path at the beginning of training, ensuring stability in the early stage of the hybrid architecture.
2.6 Block-Level Attention Residuals: Cross-Layer Reuse and Reduction
In deep networks, later layers often can only indirectly use fine-grained features from earlier layers. This phenomenon is often called the “deep representation bottleneck.”
Block-level attention residuals allow the model to aggregate representations from multiple layers within and across blocks. The general idea is:
# Illustrative logic: layers in a block share an aggregated representationagg = x0 # representation at block entryfor layer in block:agg = agg + layer(norm(agg)) # layer-wise accumulation# Later layers in the same block can directly read agg and the original x0# Between blocks, only the aggregated result and a few cross-block residuals# are passed, reducing cache and communication volume
Main benefits:
- Deeper layers more easily obtain fine-grained information from shallow layers.
- Intra-block aggregation can be fused as layers are traversed, while inter-block communication passes only a small number of residuals, reducing cache and communication overhead.
Main costs:
- Aggregation introduces additional memory access and needs operator-level fusion.
- Variance in the residual stream accumulates with layer count, so normalization such as RMSNorm is often required.
- More frequent data dependencies make pipeline-parallel scheduling harder.
Therefore, this type of design is more suitable for teams with large-scale training infrastructure.
3. Mixture of Experts: Stable Training and Efficient Inference at Trillion-Parameter Scale
3.1 The Dual Identity Created by Sparsification
Using total parameters of 2.8 trillion and 104 billion activated parameters per token as an example:
For training compute
Each token computes only about 3.7% of the parameters, making the matrix multiplication scale close to that of a roughly 10-billion-parameter dense model.For memory and communication
Loading and communication still involve all parameters. Thousands of fine-grained experts also need expert parallelism to distribute experts across devices. When a subset of experts is activated, a global-to-local collective communication operation is performed.
The higher the sparsity, the cheaper each token becomes, but the higher the requirements on communication and load balancing.
3.2 Activation Functions: Finding Smoothness in Gated Linear Units
SwiGLU is one of the most common gated activations. SiTU-GLU smooths the activation on the gating branch. Its approximate form is:
SiTU-GLU(x) = SiTU(W1 x) ⊙ (W3 x)# SiTU is a smooth activation that is differentiable everywhere# Its output is not additionally scaled# W2 projects the result back to the model dimension# SiLU is x * sigmoid(x)
The specific choice should be based on ablation experiments. When evaluating gated activations, teams can focus on:
- Whether the gating branch is always “on,” which can cause the gate to degenerate.
- Whether activation tails contain extreme values, which can lead to overflow during low-precision training.
- Whether large activation values are friendly to quantization and low-precision inference.
3.3 Quantile Load Balancing: Removing the Extra Gradient from Auxiliary Losses
The traditional approach is to add a load-balancing loss outside the main loss. However, auxiliary losses can interfere with the gradient of the main task, and their weights are difficult to tune.
An auxiliary-loss-free approach instead maintains a bias b_i for each expert. The bias affects routing decisions but does not enter gradient computation:
# Each expert maintains a bias b_iaffinity = router(x) # routing affinityscore = affinity + bias # used only for top-k selectiontopk = topk(score, k)weight = affinity.gather(topk) # weights entering the gradient use the original affinity# Quantile-style update (once per training step or every few steps)counts = all_reduce(count_tokens(topk)) # global expert load countsratio = counts / counts.mean()err = torch.sign(torch.quantile(ratio, 0.75) - ratio) # goal: balance within quartilesbias += lr * err.sign()
The difference between this quantile-style approach and traditional balancing is:
- Traditional balancing usually tries to push every expert’s load toward the average.
- The quantile-style approach only pulls experts whose load is overheated in the tail back toward the middle of the distribution, for example within quartiles, while allowing some degree of imbalance.
This has two advantages:
- It does not conflict with the inherently imbalanced expert preferences that can emerge in language modeling.
- It does not sacrifice routing quality merely to achieve balance.
Supporting practices include:
- Collecting global statistics across devices with all-reduce, and smoothing the bias to avoid oscillation.
- Avoid setting the capacity factor too tightly, which can increase token-drop rates.
- Monitoring balance using expert-distribution entropy, dropped-token ratio, and the ratio of maximum load to average load.
3.4 The Hidden Constraint of Attention Scaling on MoE
Long context and mixture-of-experts may look independent, but they are actually coupled:
- Attention scores are scaled by the square root of the dimension. Activation functions and routing gates make expert weight distributions different from those of dense models.
- Dynamic range is amplified at the scale of hundreds of billions of parameters.
As a result, this type of model needs to configure several things together:
- Numerical-stability techniques for the router, such as regularization similar to z-loss.
- Attention scaling or Softmax upper bounds, also known as soft-cap.
- High-precision, such as fp32, accumulation in the backbone network.
- Low-precision matrix multiplication in selected local regions.
If any of these are missing, the system may appear to train fine at small scale but diverge at large scale.
4. Training Pipeline: Growing Context Length and Task Complexity in Stages
4.1 Multi-Stage Context Extension
Training directly from scratch at million-token context is extremely expensive. A common approach is staged progression:
| Stage | Context Length | Goal |
|---|---|---|
| Stage 1 | 8K | Learn basic language capabilities |
| Stage 2 | 64K | Introduce medium-length dependencies |
| Stage 3 | 256K | Learn cross-paragraph and cross-file information integration |
| Stage 4 | 1M | Develop long-range retrieval and long-document task capabilities |
Each stage requires adjustments in three areas:
Data mixture
Mix real corpora and synthetic data according to the length distribution of each stage. Long samples should not merely be produced by simple concatenation.Learning rate and parallelism strategy
For long-sequence training, the chunking granularity of sequence parallelism and the frequency of activation recomputation must be retuned.Position scheme
If rotary position encoding is used, interpolation or base-frequency adjustment is needed. This class of models uses no positional encoding, so this step can be skipped, but attention must be converted into a chunked form.
4.2 Agentic Reinforcement Learning Post-Training
The goal of million-token-level agentic reinforcement learning is to teach the model to keep making correct tool calls over long trajectories.
In practice, this usually includes three steps:
Verifiable rewards
Task outcomes can be determined programmatically, such as whether code passes tests or whether data results are correct. This reduces bias from reward models.Long-trajectory sampling
Sample multi-step tool-call trajectories and set an upper limit on rollout length.Policy optimization
Use group-normalized advantages combined with sequence-level importance ratios to reduce gradient noise over long trajectories.
Several failure modes should be considered in advance:
Context exhaustion
The task is not finished before the token budget is exhausted. Compression, summarization, or proactive forgetting strategies are needed.Sparse rewards
If the first few steps are all wrong, the entire trajectory may produce no useful gradient. Process rewards or curriculum learning may be required.Training instability
Numerical errors under low precision on long sequences can amplify into policy drift. The training framework needs to support a high-precision backbone with selected low-precision regions.
5. Capability Boundaries and Evaluation Recommendations
5.1 Capability Boundaries of This Architecture
| Dimension | Strengths | Risks / Costs |
|---|---|---|
| Long context | Linear inference complexity; training can progress in stages | Limited state capacity; “middle-section” information may decay too much over ultra-long ranges |
| Precise retrieval | Global attention layers supplement exact matching | Few global layers; extremely fine-grained retrieval tasks may still be inaccurate |
| MoE sparsification | Per-token compute is close to a small dense model | Memory and expert communication scale with all parameters; fine-grained routing increases scheduling complexity |
| Multimodality | Native joint pretraining enables better cross-modal alignment | Visual tokens consume context budget; resolution and token count must be balanced |
| Agentic capability | Long-horizon tool calling and task planning | Tool-call error rates accumulate over steps; retry and recovery mechanisms are needed |
5.2 Evaluation Checklist: From “Passing Benchmarks” to “Going Live”
Long-context quality
Use length-tier tests such as 8K, 32K, 128K, and 1M. Pay attention to middle-section information hit rate, not only whether the model can find a needle in a haystack.Multi-step task success rate
Track tool-call parameter correctness, task completion rate, average number of steps, and failure retry rate.Inference economics
Measure time-to-first-token, decoding throughput, cost per million tokens under long context, and the benefit from cache hits.Training stability
Monitor gradient norm distributions, expert-load entropy, and whether activation quantiles drift during training.Business regression
Before launch, run regression tests with real business samples to confirm that replacing the old model does not cause metric degradation.
5.3 Deployment Considerations on a Cloud Platform
When deploying this type of capability on a cloud platform, a general technical stack can be used:
Data layer
Use object storage to manage long-text corpora and model artifacts. Lifecycle policies can help control cost.Serving layer
Use a container platform to host inference services, with elastic scaling based on queue depth.Observability
Use logging services and monitoring/alerting to track token usage, latency, and failure rates, and set budget thresholds.
More specifically:
- On the training side, connect metrics such as expert load and gradient norms to monitoring so that divergence can be detected early.
- On the application side, prioritize a hybrid strategy: use long-context capability for long-document understanding, and use smaller models or caching for high-frequency, low-latency tasks.
Specific product forms and capabilities should be verified against the latest official documentation.
Conclusion: From a Parameter Narrative to an Engineering Narrative
The core value of a new generation of ultra-sparse MoE long-context models is not simply pushing parameters to the trillion scale. The real value lies in turning three engineering problems into measurable design choices:
- A lower bound on decay turns numerical stability into a computable constraint.
- Auxiliary-loss-free quantile balancing turns load balancing from “an interfering term in the main loss” into “part of the routing policy.”
- Multi-stage context extension breaks long-range capability from a one-time challenge into controllable training stages.
For developers and enterprises, the more important question is not a model’s benchmark score, but whether these mechanisms can be transferred to their own workloads. In scenarios such as long-document analysis, multi-turn tool calling, and codebase understanding, the key question is whether lower per-token cost can be exchanged for more stable long-range performance.
When evaluating a model, it is useful to return to three questions:
- How does it perform on your data?
- What is the cost per million tokens?
- Can it recover and retry after failure?
Answering these three questions before deciding whether to adjust the architecture is more reliable than chasing any single leaderboard.
