Ultra-Sparse MoE Models with Million-Token Contexts: A Technical Breakdown

    This article analyzes the engineering mechanisms behind a new generation of ultra-sparse mixture-of-experts (MoE) large language models designed for million-token contexts. It focuses on three practical problems: achieving both long-range efficiency and precise retrieval with hybrid attention, training massive sparse expert systems stably, and extending context and agentic task capabilities through staged training and post-training. The discussion includes numerically stable decay parameterization, auxiliary-loss-free load balancing, deployment trade-offs, and an evaluation checklist for moving from benchmarks to production.

    Introduction: What Should Developers Look at in New-Generation Large Models?

    Every so often, a new large model starts a wave of discussion. These models usually emphasize three things at the same time: parameter scale, context length, and agentic capabilities. A common description is that the model “supports long-text input and can perform document understanding, content generation, and data analysis,” followed by a long list of architecture terms and benchmark scores.

    The hype eventually fades. What developers really need to answer is: which mechanisms are worth borrowing, which capabilities can actually be applied to business workloads, and what costs will be incurred?

    This article does not benchmark any specific product. Instead, it abstracts the public technical information of this class of models into three reusable engineering questions:

    • How can attention balance long-range efficiency and precise retrieval?
      This maps to “hybrid linear attention + numerically stable decay parameterization.”

    • How can ultra-large-scale sparse experts be trained stably?
      This maps to “auxiliary-loss-free load balancing + activation function design + block-level attention residuals.”

    • How can a model push context and tasks toward the long range?
      This maps to “multi-stage context extension + agentic reinforcement learning post-training.”

    For each question, this article explains the underlying principle, key implementation points, and evaluation methods. Any specific configurations mentioned below come from public technical reports and are discussed only for technical analysis.

    1. Technical Overview: One Main Line and Three Engineering Problems

    1.1 Overall Configuration: Sparsification, Native Multimodality, and Million-Token Context

    A typical configuration for this new generation of models looks like this:

    Dimension Typical Configuration Engineering Implication
    Parameter scale Total parameters around 2.8 trillion; about 104 billion activated per token, with roughly 96% sparsity Compute per token is about 3.7% of total parameters; memory usage and expert communication still scale with total parameters
    Modality Native multimodal: vision and text are jointly pretrained Reduces the cross-modal alignment loss introduced by separate visual adapters
    Context Native support for 1 million tokens, using a no-positional-encoding design Mitigates position-encoding extrapolation issues, but imposes new requirements on chunked attention and state management
    Attention Gated linear attention with incremental decay parameterization, interleaved by layer with multi-head latent attention; block-level attention residuals enable cross-layer information sharing Trades off between long-sequence efficiency and precise retrieval
    MoE Routed experts plus shared experts; uses activation functions from the SwiGLU family Combines dense-channel semantics with dynamic sparse computation
    Training pipeline Multi-stage pretraining from 8K → 64K → 256K → 1M, followed by million-token-level agentic reinforcement learning post-training Long-range capabilities are cultivated stage by stage

    1.2 Three Engineering Problems

    • Computational efficiency for ultra-long context
      Attention cost grows quadratically with sequence length. For example, compared with 64K tokens, the attention computation for 128K tokens is about 4 times as large, while 256K tokens is about 16 times as large. At the same time, KV-cache memory grows linearly with length. If every token goes through full attention, both training and inference become difficult to scale.

    • Training stability with a massive number of experts
      The number of routed experts can reach hundreds or even thousands. Expert granularity is often finer: each expert has fewer parameters, and there are more experts. This helps balance compositional expressiveness and training efficiency. However, load skew can cause some experts to be undertrained or even “die.” Routing collapse, numerical instability, and communication hotspots follow.

    • Long-range reasoning for agents
      Multi-step tasks often involve dozens or even hundreds of tool calls. Critical evidence may appear hundreds of thousands of tokens earlier in the context. This requires combining long-context capability with task planning. The post-training stage must also prepare for long trajectories and sparse rewards.

    1.3 How to Evaluate an Architecture

    When facing a list of architecture terms, four questions can quickly reveal the technical substance:

    1. Is there ablation evidence?
      When a mechanism is removed, how much do the metrics drop?

    2. Are the costs transparent?
      How do memory, communication, latency, and engineering complexity change?

    3. Is it reusable?
      Can it be transferred to the pretraining or post-training pipeline of other models?

    4. Where are the capability boundaries?
      Does the designer explain suitable scenarios and known limitations?

    2. Attention Design: Hybrid Architecture and Numerically Stable Decay Parameterization

    2.1 Linear Attention: The “Forgetting” Problem Behind Linear Cost

    Standard attention needs to store the full K/V cache. Linear attention rewrites attention as a recurrent update of a matrix state, reducing the complexity over sequence length to linear:

    1. S_t = A_t ⊙ S_{t-1} + K_t V_tᵀ, O_t = S_tᵀ Q_t

    Where:

    • S_t is the state matrix under a single-head view, with shape d × d, acting as a compressible “memory.”
    • K_t, V_t, and Q_t are all d-dimensional vectors.
    • A_t is a channel-wise decay coefficient, also called a forgetting gate.
    • ⊙ denotes element-wise multiplication.

    When A_t = 1, this corresponds to linear attention without gating. When A_t = 0, it corresponds to fast-weight updates without a forgetting mechanism. Common gated linear attention introduces an input-dependent A_t, allowing the model to decide what to remember and what to forget.

    There are two main problems:

    • Limited state capacity
      A fixed-size state cannot fully preserve every detail across millions of tokens.

    • Numerical stability for chunked parallelism
      To run efficiently on GPU matrix-multiply units, a sequence is usually split into chunks and computed in parallel. The problem is that decay relationships between positions inside a chunk need to be computed “separately first, then divided.” If the decay exceeds the range of low-precision floating point, this division can overflow or underflow.

    2.2 Numerical Risk in Chunked Computation

    Assume the chunk length is C. For position i inside a chunk, the cumulative log decay is L_i ≤ 0, and u_i = exp(L_i) ∈ (0, 1].

    The attention score and output inside a chunk can be written as:

    1. A_ij = (Q_i · K_j) · u_i / u_j (j ≤ i), O_i = Σ_j A_ij V_j

    If we absorb u_i into Q and 1 / u_j into K, the entire chunk can be computed with one matrix multiplication:

    1. Ã_ij = (u_i · Q_i) · (K_j / u_j)ᵀ

    If there is no lower bound on decay, the log decay for a single token can become extremely steep. For BF16, with relative precision around 0.4% and maximum value around 3.4×10³⁸:

    Maximum Absolute Cumulative Log Decay Within a Chunk Maximum Value of 1/u BF16 Behavior
    ≈ 3 ≈ 20 Safe
    ≈ 16 ≈ 10⁷ Safe
    ≈ 80 ≈ 5.5×10³⁴ Near the limit, with little headroom
    ≈ 90 ≈ 1.2×10³⁹ Overflows to infinity; the chunk loses precision

    Both directions carry risk:

    • The K / u side can overflow.
    • The u_i · Q_i side can underflow to zero under steep decay, causing distant tokens to be “hard forgotten.”

    More importantly, once overflow happens, diagonal chunks can no longer use matrix multiplication and must fall back to per-position broadcast multiply-add operations. These operations are limited by memory bandwidth and can cancel out the compute-density advantage of linear attention.

    2.3 Lower-Bounded Decay Parameterization: A Quantifiable Solution

    Some public technical reports describe a solution that parameterizes log decay with a lower bound:

    1. g(x) = g_min · Sigmoid(a · xᵀz + b), where g_min = -5 is fixed

    This replaces the original unbounded negative softplus form with a sigmoid-based form. The mathematical comparison is:

    Parameterization Output Range Lower Bound Upper Bound on 1/u Within a Chunk (C=16) BF16 Headroom
    Negative softplus (−∞, 0) None No upper bound May overflow
    Scaled sigmoid [−5, 0) −5 e⁸⁰ ≈ 5.5×10³⁴ About 1.6 orders of magnitude

    The scaled sigmoid form brings three benefits:

    1. Predictable numerical behavior
      For a chunk length C, 1/u ≤ exp(|g_min| · C). Under BF16, |g_min| · C should not exceed about 88, or about 11 under FP16. With |g_min| = 5 and C = 16, the headroom exceeds one order of magnitude.

    2. Unified operators
      Both diagonal and non-diagonal chunks can use dense matrix multiplication. This removes special branches and avoids extra element-wise corrections such as logarithms and exponentials.

    3. Clear physical meaning
      Single-step decay is constrained to preserve at least about 0.7% of historical information. It will neither remember indefinitely nor instantly clear the state.

    Unlike rotary position encoding, this decay does not depend on wavelengths tied to specific sequence lengths. In theory, as long as the model has seen similar lengths during training, there is no explicit extrapolation mismatch. Combined with a no-positional-encoding design, it can also save substantial positional computation for ultra-long contexts.

    2.4 Reference Implementation: Core Logic of Chunked Linear Attention

    The following illustrative code preserves the core logic of chunked linear attention. It omits masking details, normalization, grouped-query attention, and other engineering parts, and focuses on numerical handling.

    1. import torch
    2. G_MIN = -5.0 # Lower bound for log decay
    3. BLOCK = 16 # Block size; under BF16, |G_MIN| * BLOCK <= 88
    4. def chunk_linear_attention(q, k, v, h, A, g):
    5. """
    6. Chunked parallel form of gated linear attention (single head, scalar decay)
    7. q, k, v : [T, d], computed in fp32
    8. h : [d, d], initial matrix state (can come from the previous segment)
    9. A : [T, 1], write gate for fast-weight update
    10. g : [T, 1], log decay, guaranteed g_min <= g < 0
    11. Returns o and the updated state h
    12. """
    13. T, d = q.shape
    14. q, k, v = q.float(), k.float(), v.float()
    15. # 1. Cumulative log decay within each block, with numerical safety checks
    16. # BF16: |g_min| * BLOCK <= 88; FP16: <= 11
    17. assert T % BLOCK == 0 and abs(G_MIN) * BLOCK <= 88
    18. g_block = g.view(-1, BLOCK, 1)
    19. L = g_block.cumsum(dim=1) # cumulative within block, always <= 0
    20. u = torch.exp(L) # (0, 1]
    21. # If mixed with an unbounded parameterization, extra checks are needed
    22. # to ensure u does not underflow to 0
    23. assert torch.isfinite(u).all() and (u > 0).all()
    24. u_last = u[:, -1:] # cumulative decay at the end of each block
    25. # 2. Row scaling and column scaling: avoid any special branch for diagonal blocks
    26. q_s = q.view(-1, BLOCK, d) * u # multiply by u_i
    27. k_s = k.view(-1, BLOCK, d) / u # multiply by 1 / u_j
    28. v_s = v.view(-1, BLOCK, d)
    29. a_s = A.view(-1, BLOCK, 1)
    30. # 3. Intra-block and cross-block computation
    31. o = torch.empty_like(q_s)
    32. for b in range(q_s.shape[0]):
    33. qb, ks_b, vs_b = q_s[b], k_s[b], v_s[b]
    34. # Cross-block contribution: multiply by the state first,
    35. # then by the block-end cumulative decay
    36. o[b] = (qb @ h) * u_last[b]
    37. # Intra-block contribution: one dense matrix multiplication with masking;
    38. # no special branch
    39. causal = torch.ones(BLOCK, BLOCK, device=q.device).tril().bool()
    40. attn = (qb @ ks_b.transpose(0, 1)).masked_fill(~causal, 0.0)
    41. o[b] += attn @ vs_b
    42. # Write gate + block-end cumulative decay: update the state in
    43. # normalized form to offset magnitude drift
    44. norm = (a_s[b] * u_last[b]).sum().clamp_min(1e-6)
    45. h = (u_last[b] * h + (a_s[b] * k_s[b]).transpose(0, 1) @ vs_b) / norm
    46. return o.reshape(T, d), h

    A few practical points matter during implementation:

    • Precision division of labor
      Low-precision tensor cores handle matrix multiplication, while fp32 handles accumulation and state maintenance. Decay terms, cumulative terms, and 1/u should always be computed in fp32, then converted back to low precision when needed.

    • Dynamic range checks
      As shown by the assertions in the code, before going live, teams should perform quantile statistics on the input distribution to confirm that cumulative decay stays within a safe range.

    • Gating dimension
      If the decay is channel-wise, meaning a d-dimensional vector rather than a scalar, the row scaling and column scaling above no longer hold. The channel-wise scaling must be fused into the operator; otherwise, the implementation still falls back to the per-position path.

    • Block size is tunable
      block_size and g_min jointly determine the numerical safety region. They should be tuned as hyperparameters together with the precision strategy.

    2.5 Multi-Head Latent Attention and Output Gating

    Linear attention is good at long-range compression, but it is not necessarily strong at precise retrieval. A common approach is to insert a global attention layer every few layers, forming a hybrid architecture.

    The global layer can use a design based on “low-rank latent cache + output gating”:

    1. C_t = W_c x_t # compressed latent cache (optional RMSNorm)
    2. O_t = (W_g x_t) ⊙ (W_o [C_t; H_t]) # output gating with element-wise sigmoid

    Engineering teams need to balance two aspects:

    1. Inference side
      Compressed caches can significantly reduce KV-cache memory usage and memory access during decoding. However, the up-projection introduces additional operators. When integrating with inference-system capabilities such as paged cache and prefix reuse, operator fusion is required.

    2. Training side
      The newly added projection matrices should use scaled initialization, where the initialization standard deviation is related to the input dimension. The output layer should contribute close to zero on the residual path at the beginning of training, ensuring stability in the early stage of the hybrid architecture.

    2.6 Block-Level Attention Residuals: Cross-Layer Reuse and Reduction

    In deep networks, later layers often can only indirectly use fine-grained features from earlier layers. This phenomenon is often called the “deep representation bottleneck.”

    Block-level attention residuals allow the model to aggregate representations from multiple layers within and across blocks. The general idea is:

    1. # Illustrative logic: layers in a block share an aggregated representation
    2. agg = x0 # representation at block entry
    3. for layer in block:
    4. agg = agg + layer(norm(agg)) # layer-wise accumulation
    5. # Later layers in the same block can directly read agg and the original x0
    6. # Between blocks, only the aggregated result and a few cross-block residuals
    7. # are passed, reducing cache and communication volume

    Main benefits:

    • Deeper layers more easily obtain fine-grained information from shallow layers.
    • Intra-block aggregation can be fused as layers are traversed, while inter-block communication passes only a small number of residuals, reducing cache and communication overhead.

    Main costs:

    • Aggregation introduces additional memory access and needs operator-level fusion.
    • Variance in the residual stream accumulates with layer count, so normalization such as RMSNorm is often required.
    • More frequent data dependencies make pipeline-parallel scheduling harder.

    Therefore, this type of design is more suitable for teams with large-scale training infrastructure.

    3. Mixture of Experts: Stable Training and Efficient Inference at Trillion-Parameter Scale

    3.1 The Dual Identity Created by Sparsification

    Using total parameters of 2.8 trillion and 104 billion activated parameters per token as an example:

    • For training compute
      Each token computes only about 3.7% of the parameters, making the matrix multiplication scale close to that of a roughly 10-billion-parameter dense model.

    • For memory and communication
      Loading and communication still involve all parameters. Thousands of fine-grained experts also need expert parallelism to distribute experts across devices. When a subset of experts is activated, a global-to-local collective communication operation is performed.

    The higher the sparsity, the cheaper each token becomes, but the higher the requirements on communication and load balancing.

    3.2 Activation Functions: Finding Smoothness in Gated Linear Units

    SwiGLU is one of the most common gated activations. SiTU-GLU smooths the activation on the gating branch. Its approximate form is:

    1. SiTU-GLU(x) = SiTU(W1 x) ⊙ (W3 x)
    2. # SiTU is a smooth activation that is differentiable everywhere
    3. # Its output is not additionally scaled
    4. # W2 projects the result back to the model dimension
    5. # SiLU is x * sigmoid(x)

    The specific choice should be based on ablation experiments. When evaluating gated activations, teams can focus on:

    • Whether the gating branch is always “on,” which can cause the gate to degenerate.
    • Whether activation tails contain extreme values, which can lead to overflow during low-precision training.
    • Whether large activation values are friendly to quantization and low-precision inference.

    3.3 Quantile Load Balancing: Removing the Extra Gradient from Auxiliary Losses

    The traditional approach is to add a load-balancing loss outside the main loss. However, auxiliary losses can interfere with the gradient of the main task, and their weights are difficult to tune.

    An auxiliary-loss-free approach instead maintains a bias b_i for each expert. The bias affects routing decisions but does not enter gradient computation:

    1. # Each expert maintains a bias b_i
    2. affinity = router(x) # routing affinity
    3. score = affinity + bias # used only for top-k selection
    4. topk = topk(score, k)
    5. weight = affinity.gather(topk) # weights entering the gradient use the original affinity
    6. # Quantile-style update (once per training step or every few steps)
    7. counts = all_reduce(count_tokens(topk)) # global expert load counts
    8. ratio = counts / counts.mean()
    9. err = torch.sign(torch.quantile(ratio, 0.75) - ratio) # goal: balance within quartiles
    10. bias += lr * err.sign()

    The difference between this quantile-style approach and traditional balancing is:

    • Traditional balancing usually tries to push every expert’s load toward the average.
    • The quantile-style approach only pulls experts whose load is overheated in the tail back toward the middle of the distribution, for example within quartiles, while allowing some degree of imbalance.

    This has two advantages:

    • It does not conflict with the inherently imbalanced expert preferences that can emerge in language modeling.
    • It does not sacrifice routing quality merely to achieve balance.

    Supporting practices include:

    • Collecting global statistics across devices with all-reduce, and smoothing the bias to avoid oscillation.
    • Avoid setting the capacity factor too tightly, which can increase token-drop rates.
    • Monitoring balance using expert-distribution entropy, dropped-token ratio, and the ratio of maximum load to average load.

    3.4 The Hidden Constraint of Attention Scaling on MoE

    Long context and mixture-of-experts may look independent, but they are actually coupled:

    • Attention scores are scaled by the square root of the dimension. Activation functions and routing gates make expert weight distributions different from those of dense models.
    • Dynamic range is amplified at the scale of hundreds of billions of parameters.

    As a result, this type of model needs to configure several things together:

    • Numerical-stability techniques for the router, such as regularization similar to z-loss.
    • Attention scaling or Softmax upper bounds, also known as soft-cap.
    • High-precision, such as fp32, accumulation in the backbone network.
    • Low-precision matrix multiplication in selected local regions.

    If any of these are missing, the system may appear to train fine at small scale but diverge at large scale.

    4. Training Pipeline: Growing Context Length and Task Complexity in Stages

    4.1 Multi-Stage Context Extension

    Training directly from scratch at million-token context is extremely expensive. A common approach is staged progression:

    Stage Context Length Goal
    Stage 1 8K Learn basic language capabilities
    Stage 2 64K Introduce medium-length dependencies
    Stage 3 256K Learn cross-paragraph and cross-file information integration
    Stage 4 1M Develop long-range retrieval and long-document task capabilities

    Each stage requires adjustments in three areas:

    • Data mixture
      Mix real corpora and synthetic data according to the length distribution of each stage. Long samples should not merely be produced by simple concatenation.

    • Learning rate and parallelism strategy
      For long-sequence training, the chunking granularity of sequence parallelism and the frequency of activation recomputation must be retuned.

    • Position scheme
      If rotary position encoding is used, interpolation or base-frequency adjustment is needed. This class of models uses no positional encoding, so this step can be skipped, but attention must be converted into a chunked form.

    4.2 Agentic Reinforcement Learning Post-Training

    The goal of million-token-level agentic reinforcement learning is to teach the model to keep making correct tool calls over long trajectories.

    In practice, this usually includes three steps:

    1. Verifiable rewards
      Task outcomes can be determined programmatically, such as whether code passes tests or whether data results are correct. This reduces bias from reward models.

    2. Long-trajectory sampling
      Sample multi-step tool-call trajectories and set an upper limit on rollout length.

    3. Policy optimization
      Use group-normalized advantages combined with sequence-level importance ratios to reduce gradient noise over long trajectories.

    Several failure modes should be considered in advance:

    • Context exhaustion
      The task is not finished before the token budget is exhausted. Compression, summarization, or proactive forgetting strategies are needed.

    • Sparse rewards
      If the first few steps are all wrong, the entire trajectory may produce no useful gradient. Process rewards or curriculum learning may be required.

    • Training instability
      Numerical errors under low precision on long sequences can amplify into policy drift. The training framework needs to support a high-precision backbone with selected low-precision regions.

    5. Capability Boundaries and Evaluation Recommendations

    5.1 Capability Boundaries of This Architecture

    Dimension Strengths Risks / Costs
    Long context Linear inference complexity; training can progress in stages Limited state capacity; “middle-section” information may decay too much over ultra-long ranges
    Precise retrieval Global attention layers supplement exact matching Few global layers; extremely fine-grained retrieval tasks may still be inaccurate
    MoE sparsification Per-token compute is close to a small dense model Memory and expert communication scale with all parameters; fine-grained routing increases scheduling complexity
    Multimodality Native joint pretraining enables better cross-modal alignment Visual tokens consume context budget; resolution and token count must be balanced
    Agentic capability Long-horizon tool calling and task planning Tool-call error rates accumulate over steps; retry and recovery mechanisms are needed

    5.2 Evaluation Checklist: From “Passing Benchmarks” to “Going Live”

    • Long-context quality
      Use length-tier tests such as 8K, 32K, 128K, and 1M. Pay attention to middle-section information hit rate, not only whether the model can find a needle in a haystack.

    • Multi-step task success rate
      Track tool-call parameter correctness, task completion rate, average number of steps, and failure retry rate.

    • Inference economics
      Measure time-to-first-token, decoding throughput, cost per million tokens under long context, and the benefit from cache hits.

    • Training stability
      Monitor gradient norm distributions, expert-load entropy, and whether activation quantiles drift during training.

    • Business regression
      Before launch, run regression tests with real business samples to confirm that replacing the old model does not cause metric degradation.

    5.3 Deployment Considerations on a Cloud Platform

    When deploying this type of capability on a cloud platform, a general technical stack can be used:

    • Data layer
      Use object storage to manage long-text corpora and model artifacts. Lifecycle policies can help control cost.

    • Serving layer
      Use a container platform to host inference services, with elastic scaling based on queue depth.

    • Observability
      Use logging services and monitoring/alerting to track token usage, latency, and failure rates, and set budget thresholds.

    More specifically:

    • On the training side, connect metrics such as expert load and gradient norms to monitoring so that divergence can be detected early.
    • On the application side, prioritize a hybrid strategy: use long-context capability for long-document understanding, and use smaller models or caching for high-frequency, low-latency tasks.

    Specific product forms and capabilities should be verified against the latest official documentation.

    Conclusion: From a Parameter Narrative to an Engineering Narrative

    The core value of a new generation of ultra-sparse MoE long-context models is not simply pushing parameters to the trillion scale. The real value lies in turning three engineering problems into measurable design choices:

    • A lower bound on decay turns numerical stability into a computable constraint.
    • Auxiliary-loss-free quantile balancing turns load balancing from “an interfering term in the main loss” into “part of the routing policy.”
    • Multi-stage context extension breaks long-range capability from a one-time challenge into controllable training stages.

    For developers and enterprises, the more important question is not a model’s benchmark score, but whether these mechanisms can be transferred to their own workloads. In scenarios such as long-document analysis, multi-turn tool calling, and codebase understanding, the key question is whether lower per-token cost can be exchanged for more stable long-range performance.

    When evaluating a model, it is useful to return to three questions:

    1. How does it perform on your data?
    2. What is the cost per million tokens?
    3. Can it recover and retry after failure?

    Answering these three questions before deciding whether to adjust the architecture is more reliable than chasing any single leaderboard.