Million-Token Long-Context MoE Multimodal LLMs: How They Are Built

    Long-context, trillion-parameter, and native multimodal capabilities are converging in modern large language models. Using a July 2026 trillion-parameter MoE multimodal model as a case study—roughly 2.8 trillion total parameters with about 104 billion activated—this article breaks down the engineering behind long-context sparse models: the capacity/compute decoupling of MoE, the compute and KV-cache costs of long context, gated multi-head latent attention, lower-bounded decay linear attention, block-level attention residuals, Stable LatentMoE routing with SiTU-GLU, quantile balancing without an auxiliary loss, multi-stage context extension (8K to 1M) and agentic reinforcement post-training. It closes with deployment cost, evaluation, and selection guidance, aimed at engineers who want to know which techniques can be ported to their own models and inference systems.

    Introduction: Three Technology Curves Converging on Long-Context Sparse Multimodal Models

    Around 2026, the race in large-model capability shows three clear curves:

    • Context length expanding from a few thousand tokens to the million-token scale;
    • Parameter scale reaching the trillions, with only a small fraction activated through Mixture-of-Experts (MoE);
    • Multimodality shifting from a “bolt-on visual encoder” to native visual understanding.

    A trillion-parameter MoE multimodal model released in July 2026 embodies all three curves at once. It has roughly 2.8 trillion total parameters with about 104 billion activated; it natively supports visual input and a 1M-token context; it introduces a series of stability mechanisms in attention and expert routing; and it is trained in multiple stages of 8K → 64K → 256K → 1M, completing agentic reinforcement learning post-training at the million-token level. It packages long-text input, file understanding, content generation, and data analysis as first-class capabilities, primarily for everyday users, and offers several access methods.

    For engineering teams, the more valuable question is not “how strong is it,” but:

    • What real pain points do these designs solve?
    • Which of them can be ported at low cost to your own model and inference system?
    • What underestimated costs does a million-token context bring?

    Note: This article uses this class of model as a sample and offers only a neutral, technical analysis. The architectural elements are compiled from public information, and the sample parameters are illustrative rather than the internal implementation of any specific product. Benchmark behavior described here is qualitative; refer to the original technical report for exact figures.

    Core Conclusions of This Article

    1. Capacity and compute are decoupled: Total parameters determine memory and bandwidth cost, while activated parameters determine per-token compute. The two can differ by an order of magnitude, so you cannot size a deployment by activated parameters alone.
    2. Long-context cost is an extra item on top of the feed-forward network: Attention computation grows quadratically, and the key-value cache swells linearly with length. Combining low-rank caching, hybrid attention, and linearly-bounded decay is the main path to holding this cost down.
    3. The larger the expert count, the more routing balance becomes the bottom line for whether training converges at all: Quantile balancing replaces the “auxiliary loss” with a “dynamic bias,” balancing effectiveness against low interference with the main task.
    4. Cross-layer information sharing is a lever for depth scaling: Block-level attention residuals turn expensive attention computation from “once per layer” into “once per block,” with direct benefits for both memory and communication.
    5. Long context takes two steps: First complete pretraining with progressive context extension, then use agentic reinforcement learning to fill in “long-horizon planning—tool calling—self-correction.”

    1. Product Positioning: A Default Entry Point for Long Documents, File Understanding, Writing, and Data Analysis

    From the original description, the product positioning of this class of model can be summarized as “the file is the entry point, and long text is the workflow”:

    Capability Typical scenario Implicit requirement on the model
    Long-text input Hundreds of pages of contracts, annual reports, meeting notes, entire books High-fidelity retrieval and cross-paragraph reasoning, not simple truncation
    File understanding Upload a document, then question, compare, and extract directly Visual parsing of layout, tables, and charts
    Content generation Producing reports, proposals, and long-form text from multiple source materials Cross-document integration with structural and factual consistency
    Data analysis Upload a spreadsheet or a file with data charts for interpretation Numerical understanding plus a loop of code execution and tool calling

    Targeting everyday users means the product cannot chase only the “nominal context length.” Whether a single question-and-answer is good depends on whether it finds the right information, sees all of it, and states it accurately within a real long document. Multiple access methods (conversational apps, open APIs, third-party tool integrations, and so on) extend the capability to batch document processing and enterprise workflows, while raising the bar for concurrency, streaming output, and context management.

    There is an engineering fact that is easy to overlook here: a 1M context is not a model parameter but a function of “prefill latency × memory × cost.” Under the same model configuration, input length determines the compute and memory demand of a single request. So the sections below start from scale and sparsity and then break down the compute and storage ledger behind long context one item at a time.


    2. Scale and Sparsity: What 2.8 Trillion Total Parameters and 104 Billion Activated Parameters Mean

    2.1 Decoupling Capacity from Compute

    The core idea of MoE is to split the feed-forward layers into a large number of “experts,” where a router selects a few experts to participate in computation for each token. As a result, the same model has two scale metrics:

    • Total parameters ≈ 2.8 trillion, which determine the model’s knowledge capacity as well as the memory footprint and bandwidth demand at deployment;
    • Activated parameters ≈ 104 billion, about 3.7% of the total, which determine the forward computation per token.
    Dimension Dense model Sparse MoE (this sample)
    Parameter count All parameters participate in every token’s computation 2.8T total parameters, only about 104B activated
    Forward compute (rough estimate) Proportional to total parameters Proportional to activated parameters, about 2.1×10¹¹ FLOPs/token
    Memory footprint Proportional to total parameters Also proportional to total parameters, not reduced by sparsity
    Training stability Single gradient path Routing decisions can shift abruptly, requiring dedicated mechanisms

    Using the rule of thumb that forward compute is about 2× the activated parameter count, each generated token needs roughly 2×10¹¹ floating-point operations—comparable to the compute of a ~120-billion-parameter dense model, while the weights are at the trillion scale. This yields two deployment principles:

    • Estimate throughput and per-card compute demand from activated parameters;
    • Estimate memory and interconnect bandwidth demand from total parameters (then factor in quantization and how experts are sharded).

    2.2 A Rough Memory Estimate

    Storing weights in BF16, ignoring optimizer states and runtime buffers:

    • 2.8 trillion × 2 bytes ≈ 5.6 TB of weights;
    • With 4-bit weight quantization, about 1.4 TB;
    • Either way, this far exceeds the memory capacity of a single server, so multi-node, multi-card sharding and expert parallelism are required.

    2.3 Expert Count and Routing Design

    The routing expert count in this sample reaches 896. Such fine-grained splitting gives finer knowledge division, but also concentrates the engineering problems into three areas:

    1. Expert distribution: Experts are sharded across devices. When a particular expert becomes a “hot spot,” the device holding it becomes the short board, and global throughput drops accordingly.
    2. Communication amplification: Tokens are sent to the device holding their expert and results are gathered back, so every MoE layer carries two rounds of collective communication, which is especially pronounced in cross-node scenarios.
    3. Routing fluctuation: Early in training or when the data distribution shifts, routing tends to drift toward a “winner-takes-all” pattern.

    There is no silver bullet for these problems; they require coordinated optimization in three places: the router balancing mechanism (see Chapter 6), expert-parallel topology and communication optimization, and the capacity factor and drop policy. The takeaway of this chapter is that sparsity lowers the compute cost but shifts the complexity onto routing, balancing, and communication.


    3. The Triple Bottleneck of Long Context: Compute, Storage, and Effective Utilization

    3.1 Compute: Quadratic Growth

    Standard causal self-attention must compute the relationship between every pair of tokens in the sequence, with a complexity of O(n²·d), where n is the context length. When length grows from 32K to 1M (32×), attention computation grows by roughly 1024×.

    Although the causal mask makes half the matrix irrelevant, and FlashAttention-style tiled algorithms have moved the intermediate matrix out of memory to bring attention close to being compute-bound, the total FLOPs of the attention portion still grow with the square of the length. Suppose a long-document task’s input length grows tenfold: attention cost rises about a hundredfold, while the feed-forward network rises only tenfold.

    3.2 Storage: The Key-Value Cache Grows Linearly with Length

    During decoding, each generated token must attend over all historical keys and values. The key-value cache (KV Cache) can be estimated with the formula below:

    KV cache ≈ num_layers L × sequence length n × per-token per-layer cache dim d_kv × bytes per element b

    Taking a model with hidden dimension 8192 and 64 layers as the illustrative target, compare the cache size for a 1M context under different attention designs:

    Attention design Per-token per-layer cache dim 1M context cache (illustrative)
    Standard multi-head attention (64 heads × 128 dim, separate K and V) 16384 ≈ 2 TB
    Grouped-query attention with 8 KV heads (128 dim) 2048 ≈ 256 GB
    Latent attention (512-dim latent vector + 64-dim decoupled position term) 576 ≈ 74 GB

    Converted to binary units, roughly 72 GB and 37 GB.

    Two notable conclusions:

    • Latent attention compresses the 1M cache into a range that “a single machine with multiple cards can hold,” which is one of the preconditions for making million-token context practical;
    • The bottleneck in the decoding phase is often not compute but bandwidth: every token must read the entire cache once. The smaller the cache, the faster decoding is and the more concurrent requests can be served.

    3.3 Effective Utilization: Position Distribution and the “Truly Usable Length”

    The real-world effectiveness of a long context depends on where information sits in the sequence. The key clauses a user provides are often in the middle of hundreds of thousands of characters, yet attention naturally tends to capture content near the start and end of the sequence, so information in the middle of a long context is easily “drowned out.”

    Therefore, nominal length is not equal to effective length. When evaluating a model, whether key information can be reliably retrieved at different depths, and whether questions that require piecing an answer together across multiple locations can be answered correctly, matter more than “what is the maximum supported length.”

    3.4 Design Implications

    To address the bottlenecks above, this class of model uses a combination approach:

    • Gated multi-head latent attention (low-rank caching) to shrink storage;
    • Linear attention to carry long-range information transfer and reduce compute growth;
    • Attention residuals to strengthen deep representations;
    • Multi-stage context extension to ensure long-distance dependencies are genuinely learned;
    • Cache reuse on the inference side to further amortize cost.

    The sections below break down the attention design layer by layer.


    4. The Attention Side: Gated Latent Attention, Lower-Bounded Decay, and Block-Level Attention Residuals

    4.1 Gated Multi-Head Latent Attention: Compress the Cache, Then Add a Gate to the Information Flow

    Gated MLA (Gated Multi-head Latent Attention) can be understood through two elements:

    First, low-rank joint compression. Multi-head attention has high parallelism, but each head stores a full key-value, so cache cost grows linearly with the number of heads. Latent attention first projects keys and values into a low-dimensional latent space much smaller than the total head dimension and caches them there, then up-projects to reconstruct each head’s keys and values when needed. In implementation, a portion of the dimensions typically carries decoupled positional information (illustratively 512 latent dims + 64 position dims), while the remaining dims share the same cache.

    Second, gating. The gating mechanism introduces a learnable “valve” into the attention output or information flow, letting the model decide by content which information enters the residual stream and which is suppressed. After softmax, attention weights are always positive and sum to 1, so they cannot express “this step, overall, does not need to look elsewhere.” A multiplicative gate provides exactly this ability, which helps:

    • Suppress irrelevant content;
    • Reduce interference from entropy collapse in deep attention.

    Illustrative implementation (PyTorch style, with variable names and dimensions used only to explain the logic):

    1. import math
    2. import torch
    3. import torch.nn as nn
    4. import torch.nn.functional as F
    5. class GatedLatentAttention(nn.Module):
    6. """Gated multi-head latent attention (illustrative, not a specific model)
    7. Key points:
    8. 1) Keys/values are first compressed into a latent space; only
    9. compressed_kv and k_rope are cached during decoding;
    10. 2) A decoupled positional key (with rotary embeddings) is kept
    11. separate and not compressed;
    12. 3) A per-channel gate on the output controls how much information
    13. is written into the residual stream.
    14. """
    15. def __init__(self, dim: int, n_heads: int, d_head: int,
    16. d_latent: int = 512, d_rope: int = 64):
    17. super().__init__()
    18. self.h, self.dh, self.d_rope = n_heads, d_head, d_rope
    19. # Query: low-rank compression + decoupled positional part
    20. self.q_a_proj = nn.Linear(dim, d_latent, bias=False)
    21. self.q_a_layernorm = nn.RMSNorm(d_latent)
    22. self.q_b_proj = nn.Linear(d_latent, n_heads * (d_head + d_rope), bias=False)
    23. # Keys/values: joint compression; only these two tensors are cached at decode
    24. self.kv_a_proj = nn.Linear(dim, d_latent + d_rope, bias=False)
    25. self.kv_a_layernorm = nn.RMSNorm(d_latent)
    26. self.kv_b_proj = nn.Linear(d_latent, n_heads * d_head * 2, bias=False)
    27. self.o_proj = nn.Linear(n_heads * d_head, dim, bias=False)
    28. self.gate_proj = nn.Linear(dim, n_heads * d_head, bias=False) # output gate
    29. def forward(self, x: torch.Tensor, freqs_cis: torch.Tensor):
    30. B, T, D = x.shape
    31. # ---- Query ----
    32. q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(x)))
    33. q = q.view(B, T, self.h, self.dh + self.d_rope)
    34. q_nope, q_pe = q.split([self.dh, self.d_rope], dim=-1)
    35. q_pe = apply_rope(q_pe, freqs_cis) # rotary only on positional part
    36. q = torch.cat([q_nope, q_pe], dim=-1) # (B, T, H, Dh + Dr)
    37. # ---- Keys/values ----
    38. kv_a = self.kv_a_proj(x)
    39. c_kv, k_pe = kv_a.split([512, self.d_rope], dim=-1) # illustrative dims
    40. k_pe = apply_rope(k_pe.unsqueeze(2), freqs_cis) # shared across heads
    41. # Decode phase: cache only c_kv and k_pe, not each head's full K/V
    42. kv = self.kv_b_proj(self.kv_a_layernorm(c_kv))
    43. k_nope, v = kv.view(B, T, self.h, 2 * self.dh).split(self.dh, dim=-1)
    44. # Concatenate [content key | positional key] inside each head
    45. k = torch.cat([k_nope, k_pe.expand(-1, T, self.h, -1)], dim=-1)
    46. # ---- Attention (replaceable by a FlashAttention-style impl in training) ----
    47. scale = 1.0 / math.sqrt(self.dh + self.d_rope)
    48. scores = torch.einsum("bthd,bshd->bhts", q, k) * scale
    49. mask = torch.ones(T, T, dtype=torch.bool, device=x.device).tril()
    50. scores = scores.masked_fill(~mask, float("-inf"))
    51. o = torch.einsum("bhts,bshd->bthd", scores.softmax(dim=-1), v)
    52. # ---- Output gate + output projection ----
    53. o = o.reshape(B, T, -1)
    54. o = o * torch.sigmoid(self.gate_proj(x))
    55. return self.o_proj(o)
    56. def apply_rope(x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
    57. """Apply complex-form rotary position embeddings to the input (illustrative)"""
    58. xc = torch.view_as_complex(x.float().reshape(*x.shape[:-1], -1, 2))
    59. return torch.view_as_real(xc * freqs_cis).flatten(-2).type_as(x)

    Three points are worth noting when reusing this design:

    • The cached object is the latent vector, not the reconstructed keys and values, so quantization, cache migration, and reuse must all be done at latent-vector granularity;
    • The positional term is not compressed, and rotary embeddings act on only a few dimensions, so long-context extrapolation strategies mainly affect this part;
    • The gate adds a small number of parameters and per-channel multiplications, in exchange for finer control over the information flow.

    4.2 Linear Attention: Why a “Decay Lower Bound” Is Needed

    When full attention carries long-range information transfer, the quadratic compute and linearly growing cache pressure become heavy. Linear attention rewrites attention as a recurrent state S that can be updated over time, which is analogous to a “memory that can be updated as time goes on”:

    State update: Sₜ = Dₜ ⊙ Sₜ₋₁ + kₜᵀvₜ

    Output: oₜ = qₜSₜ

    Here Dₜ is a diagonal decay matrix obtained by exponentiating the log-decay gₜ (in gated delta-net-style schemes, Dₜ is a scalar), and setting it per channel improves expressiveness. This formulation has two direct benefits:

    • The size of the recurrent state is independent of sequence length (it depends on dimensionality), so the “memory” of a million-token context does not swell with length;
    • It contains no absolute positional encoding, so length extrapolation needs no change to the position scheme.

    But when implementing on a GPU, people do not recurse token by token; instead they compute by chunks to reuse matrix-multiply units. Let the cumulative decay Γₜ = Σᵢ≤ₜ gᵢ. The i-th row within a chunk can be written as:

    oᵢ = q̃ᵢ · Σⱼ≤ᵢ k̃ⱼᵀvⱼ + q̃ᵢ · S_before-chunk

    where q̃ᵢ = qᵢ ⊙ exp(Γᵢ) and k̃ⱼ = kⱼ ⊙ exp(−Γⱼ).

    The derivation uses only one identity: under per-channel decay, exp(Γᵢ,d − Γⱼ,d) can always be factored into a product of a term that depends only on i and a term that depends only on j. So a pairwise relationship that seems to require per-channel enumeration collapses into an ordinary matrix multiply, and the whole diagonal block can be handed to the Tensor Cores.

    The cost is that exp(−Γⱼ) must “hold up.” The traditional approach parameterizes the log-decay with Negative-Softplus (the negative of softplus), whose value can approach negative infinity without bound. Once an extremely large negative decay appears within a block, exp(−Γ) underflows to 0 in BF16, and then a subsequent division produces Inf or NaN, forcing the diagonal block to fall back to a per-position path and leaving the Tensor Cores idle.

    A lower-bounded decay parameterization changes the log-decay to a bounded-below Scaled-Sigmoid:

    gₕ(z) = g_min · σ(aₕz), with g_min = −5

    No matter what value the input z takes, the decay for every token and every channel is clamped to (g_min, 0). This allows a quantitative argument:

    • With a chunk length of 16, the maximum magnitude of cumulative decay is 16 × 5 = 80;
    • The smallest retention factor is e⁻⁸⁰ ≈ 1.8×10⁻³⁵, above the smallest BF16 normal number (about 1.18×10⁻³⁸);
    • The largest reciprocal factor is e⁸⁰ ≈ 5.5×10³⁴, well below the FP32 maximum (about 3.4×10³⁸).

    Both q̃ and k̃ can therefore be safely constructed with FP32 accumulation, and no position will “overflow.” The lower bound constrains only the maximum decay magnitude—how fast each channel can forget in the steepest case—without affecting the long-range memory ability of slowly decaying channels. In effect, it draws a safe line between expressiveness and numerical stability.

    Compare the numerical behavior of different schemes:

    Log-decay parameterization Decay magnitude exp(−Γ) (chunk len 16) Diagonal-block implementation With positional encoding
    Scaled-Sigmoid (g_min = −5) Lower-bounded ≤ e⁸⁰ ≈ 5.5×10³⁴, representable in FP32 Ordinary matmul + causal mask Can use NoPE, extrapolates directly
    Scaled-Sigmoid (g_min = −20, illustrative) Bounded but too steep e³²⁰ ≈ 10¹³⁹, exceeds FP32 range Still needs special handling or a smaller chunk Same as above
    Negative-Softplus (unbounded) Unbounded May underflow to 0, reciprocal diverges Diagonal block needs per-position path —

    The last two rows are illustrative derivations used to make two points: the lower-bound value must match the chunk length, and once the exponent can exceed the target precision range, the stability benefit disappears.

    Illustrative pseudocode: lower-bounded per-channel gating + chunked linear attention

    1. import torch
    2. G_MIN = -5.0
    3. CHUNK = 16 # chunk length must keep CHUNK * |G_MIN| within a safe range
    4. def kda_gate(z: torch.Tensor):
    5. """Lower-bounded per-channel log-decay: g = g_min * sigmoid(z) in (g_min, 0)"""
    6. return G_MIN * torch.sigmoid(z)
    7. def gated_linear_attn(q, k, v, z, state=None):
    8. """q, k: (B, T, H, Dk) v: (B, T, H, Dv) z: (B, T, H, Dk)
    9. Returns the output and updated state; illustrative, describing the compute path only.
    10. """
    11. B, T, H, Dk = q.shape
    12. Dv = v.shape[-1]
    13. out = torch.empty(B, T, H, Dv, device=q.device, dtype=torch.float32)
    14. state = torch.zeros(B, H, Dk, Dv, device=q.device, dtype=torch.float32) \
    15. if state is None else state
    16. for s in range(0, T, CHUNK):
    17. qc = q[:, s:s + CHUNK].transpose(1, 2).float() # (B, H, L, Dk)
    18. kc = k[:, s:s + CHUNK].transpose(1, 2).float()
    19. vc = v[:, s:s + CHUNK].transpose(1, 2).float()
    20. gc = kda_gate(z[:, s:s + CHUNK]).transpose(1, 2).float() # (B, H, L, Dk)
    21. cum = gc.cumsum(dim=-2) # cumulative decay Γ (FP32)
    22. q_tilde = qc * cum.exp() # q̃ = q * exp(Γ)
    23. k_tilde = kc * (-cum).exp() # k̃ = k * exp(-Γ)
    24. # ① Intra-block (diagonal) part: ordinary matmul + causal mask
    25. # exp(Γ_i - Γ_j) within the same block is obtained exactly via q̃, k̃ dot product
    26. L = qc.shape[-2]
    27. causal = torch.ones(L, L, device=q.device, dtype=torch.bool).tril()
    28. scores = torch.matmul(q_tilde, k_tilde.transpose(-1, -2)) # (B, H, L, L)
    29. scores = scores.masked_fill(~causal, 0.0)
    30. out[:, s:s + CHUNK] += torch.matmul(scores, vc).transpose(1, 2)
    31. # ② Inter-block part: all diagonal and off-diagonal blocks share the same GEMM path
    32. out[:, s:s + CHUNK] += torch.einsum(
    33. "bhld,bhdv->bhlv", q_tilde, state
    34. ).transpose(1, 2)
    35. # ③ State recursion: exp(-Γ) is protected by the lower bound, no overflow
    36. new_state = torch.einsum("bhld,bhlv->bhdv", k_tilde, vc)
    37. state = new_state + cum[:, :, -1:].transpose(-1, -2).squeeze(-2).exp() \
    38. .unsqueeze(-1) * state
    39. return out.to(q.dtype), state

    Inference implementations typically do not recompute the entire cumulative sequence at every decoding step. A common approach maintains an incremental “cross-block state + local state within the current block” and merges them once a full block is processed. The lower bound guarantees that the scaling factor of the local state always stays in a safe range, so the merge operation need not worry about numerical degradation.

    4.3 Block-Level Attention Residuals: Letting Deep Layers “See” Shallow Ones While Reducing Cost

    Deep Transformers often encounter representation homogenization: as the number of layers grows, the output directions of the layers become increasingly similar, the effective representation dimension shrinks, and deeper layers rely more and more on information already in the residual stream. The basic idea of attention residuals (AttnRes) is to propagate the result of attention computation across multiple layers in a “residual” form, so deep layers directly use the relational information that shallow layers have already constructed.

    In a per-layer implementation, every layer must perform an attention projection and access the corresponding keys and values; the deeper the model, the greater the memory and communication cost. Block-level attention residuals instead share the same attention output within several “blocks”: each layer keeps only the feed-forward sublayer plus lightweight projection and gating operations, and the attention part is computed only at the first layer of a block (or on demand).

    Illustrative pseudocode: block-level attention residuals

    1. # Illustrative pseudocode: divide L layers into N blocks, sharing one attention result per block
    2. L, N = 64, 4 # assumed layer count L and block count N, both illustrative
    3. assert L % N == 0
    4. m = L // N # number of layers per block
    5. # Each block uses independent parameters; blocks do not affect one another
    6. attn_res = [None] * N # cache each block's current attention residual output
    7. for blk in range(N):
    8. for i in range(m):
    9. if i == 0: # first layer in the block: compute and cache the residual
    10. attn_res[blk] = compute_attention_residual(x)
    11. # All layers in the block share the same attention residual; FFN is still per-layer
    12. x = x + attn_res[blk] + compute_ffn(x)

    Benefits and costs:

    Dimension Per-layer attention residual Block-level attention residual
    Attention residual computations Once per layer Once per block (down to 1/4 in the illustration)
    Attention-related activation cache Grows linearly with layer count Stored per block, significantly reduced
    Cross-layer information flow Between adjacent layers Shared within a block, passed step by step across blocks
    Cost — Sharing one attention signal within a block coarsens the granularity of expression

    In parallel training and long-context inference, attention often involves tensor parallelism, sequence parallelism, and even cross-device communication. Turning “once per layer” into “once per block” saves not only computation but also communication and memory.

    4.4 Summary of Design Tradeoffs

    Design Main benefit Cost introduced When to apply
    Gated multi-head latent attention Cache significantly shrunk, decode bandwidth reduced Up-projection adds extra compute The longer the context, the greater the benefit
    Lower-bounded-decay linear attention Long-range transfer linearized, simple operator implementation Limited range of expressible decay When a million-token context or NoPE extrapolation is needed
    Block-level attention residuals Reduced attention compute, cache, and communication Block-sharing coarsens granularity Deep networks, parallel-training scenarios

    5. The Sparse Side: Stable LatentMoE and “Sparse Does Not Mean Unstable”

    The routing side of this sample contains 896 experts, forming an overall “Stable LatentMoE” architecture that can be broken down into two keywords:

    • Latent: Expert parameters are not stacked in the full hidden space but organized in a compressed or low-rank latent space. This reduces the parameter count of each expert and of the overall routing matrix, and makes the “common basis” shared by different experts easier to learn, alleviating expert redundancy.
    • Stable: The training process is treated as a first-class goal, with constraints set separately in activation functions, routing balance, and precision policy (see 5.2, 5.3, and 5.4).

    It should be noted that this name is not elaborated much in public materials; the interpretation above is a reasonable inference from a general perspective, not a description of its exact implementation.

    On the activation side, this sample uses SiTU-GLU: within the GLU gating structure, a shape-adjustable SiTU-style activation replaces the more common SiLU and others. At extreme expert scale, the token distribution each expert receives is highly uneven, and few-shot experts have large gradient noise and are prone to scale drift. A more controllable activation shape in the gating branch helps:

    • Suppress gradient spikes;
    • Keep the output scale consistent across experts.

    Since the activation function is an element-wise operation with little impact on throughput, this “small change for stability” tradeoff is common in MoE.


    6. Routing Balance: Why Quantile Balancing Needs No Auxiliary Loss

    6.1 Why Routing Balance Is a MoE “Lifeline”

    In MoE training, routing naturally tends toward positive feedback: experts selected more early on get fuller gradient updates and become stronger, which leads to more tokens selecting them, until a few experts are overloaded and many experts are “starved.” A common community practice is the load-balancing auxiliary loss: in addition to the language-modeling loss, add a penalty term for uneven distribution to steer routing toward uniformity.

    But it has two side effects:

    1. Gradient interference: The auxiliary loss is added directly into the total loss, and its coefficient requires careful tuning. A small coefficient gives insufficient balance; a large one harms the main task.
    2. Balance is “static”: An auxiliary loss with a fixed coefficient is only approximately effective under specific expert counts and training stages. When routing evolves rapidly, it cannot keep up with the dynamic changes.

    6.2 The Mechanism of Quantile Balancing

    The idea of Quantile Balancing (QB) is to move “balance” out of the loss function and turn it into a set of dynamic biases on the router. Its core steps:

    1. Statistics: Within a sliding window, record the actual token count each expert handles (or the sum of routing probabilities);
    2. Take a quantile: Rather than directly comparing “absolute counts,” take a quantile (such as the median or the upper quartile) of the per-expert load distribution as the reference;
    3. Update the bias: Maintain a bias per expert that is used only for routing scores and does not participate in gradient updates, adjusting it by a fixed step:
      • Experts whose load is above the reference quantile get their bias lowered;
      • Experts whose load is below the reference quantile get their bias raised;
    4. Form a loop: The new bias enters the next round of routing scoring, and eventually each expert’s load converges toward the reference quantile.

    Using a “quantile” rather than the “mean” as the reference is friendlier to long-tail experts. A few extreme-overload experts will not push the reference line too high, thereby avoiding oscillation of the balancing process between the mean and the long tail.

    Illustrative pseudocode: quantile balancing (single-step update)

    1. import torch
    2. class QuantileBalancer:
    3. """Quantile balancing (illustrative, not a specific model)
    4. Idea: use "historical load quantile - current load" to decide the
    5. direction and magnitude of each expert's bias adjustment. The bias
    6. acts only on routing scores and produces no gradient, equivalent to
    7. removing the load-balancing auxiliary loss.
    8. """
    9. def __init__(self, num_experts: int, momentum: float = 0.99,
    10. update_step: float = 1e-3, quantile: float = 0.5):
    11. self.num_experts = num_experts
    12. self.momentum = momentum
    13. self.update_step = update_step
    14. self.quantile = quantile
    15. self.hist = torch.zeros(num_experts) # historical load per expert (moving avg)
    16. self.bias = torch.zeros(num_experts) # routing bias (no gradient update)
    17. @torch.no_grad()
    18. def update(self, token_load: torch.Tensor, update_bias: bool):
    19. """token_load: current load statistics with shape (num_experts,)
    20. Returns the updated bias; the caller adds it to the router scores, then takes top-k.
    21. """
    22. self.hist.mul_(self.momentum).add_(token_load, alpha=1 - self.momentum)
    23. if not update_bias:
    24. return self.bias
    25. # Use a quantile of historical load as reference: positive -> relatively overloaded,
    26. # so the bias should be lowered
    27. target = torch.quantile(self.hist, self.quantile)
    28. delta = self.hist - target
    29. delta = delta / (delta.abs().max() + 1e-6) # normalize to avoid runaway steps
    30. self.bias.add_(delta, alpha=-self.update_step)
    31. self.bias.clamp_(-0.1, 0.1) # bias range limit (illustrative)
    32. return self.bias

    6.3 Comparison and Benefits

    Dimension Load-balancing auxiliary loss Quantile Balancing (QB)
    Where balance acts Total loss (gradient level) Routing bias (forward scoring)
    Impact on main-task gradients Added directly, may interfere No extra gradient term
    Parameter sensitivity Balance coefficient needs careful tuning Mainly update step, window length, quantile choice
    Adaptation to expert distribution Static constraint Dynamic, follows the load distribution
    Implementation cost Low Low: add a small amount of routing statistics and bias updates

    6.4 Engineering Tips

    • The bias is just a “temporary chip for routing”; it does not participate in gradient updates and must be explicitly detached;
    • The update step is related to the expert count. The more experts there are, the smaller the expected load per expert, so the update step needs to be adjusted accordingly;
    • If the bias is turned off during evaluation, the actual routing distribution may differ from training, which affects reproduction and benchmark comparison.

    7. From 8K to 1M: Multi-Stage Context Extension and Agentic Post-Training

    7.1 Multi-Stage Context Extension: Why “Short First, Then Long”

    This sample’s pretraining proceeds stage by stage as 8K → 64K → 256K → 1M, a curriculum-style approach common in the industry:

    Stage Length Main goal Notes
    One 8K Build a language and knowledge foundation; highest training efficiency Warm up the optimizer, stabilize routing distribution
    Two 64K Introduce medium-distance dependencies, adapt to long-document structure Gradually mix in long-text corpora
    Three 256K Learn to integrate information across chapters and documents Monitor loss spikes under long sequences
    Four 1M Cover the extreme length, stabilize long-range retrieval and reasoning Parallel strategy, memory, and communication pressure are highest

    Three key reasons:

    1. Efficiency: Short-sequence stages have far higher throughput than long-sequence stages, so learning “general capability” first and “long-range capability” later lowers overall cost;
    2. Stability: Training directly on ultra-long sequences makes attention logits and activations prone to drift, and optimizer states need drastic re-shaping;
    3. Recoverability: The stage-by-stage checkpoints naturally form “rollback points”; if a stage degrades, you can roll back and adjust the data mix.

    The parallel strategy also needs to switch across stages:

    • Short-sequence stages rely mainly on data parallelism combined with tensor parallelism and expert parallelism;
    • Long-sequence stages introduce sequence parallelism or context parallelism, sharding the same sequence across multiple cards and using ring communication to accumulate attention results step by step.

    7.2 Million-Token-Level Agentic Post-Training

    Post-training introduces agentic reinforcement learning (Agentic RL) at the million-token level. Training samples are no longer single-turn questions and answers but long trajectories produced by the model interacting with tools over multiple turns:

    • A trajectory contains multiple document retrievals, tool calls, and intermediate reasoning, with a single trajectory reaching hundreds of thousands of tokens;
    • The reward signal comes from whether the task is truly completed, for example whether the key fields in a file are extracted correctly and whether the final report cites the correct basis.

    This directly addresses a real pain point: long-context models often “can see but cannot use”—the key evidence is already in the context, yet the model loses the thread during long-horizon planning. Reinforcement learning optimizes precisely the ability to “continuously hold the goal in a long trajectory and correct errors in time.”

    Implementation must address four categories of engineering problems:

    Problem Symptom General approach (illustrative)
    Large variance in sample length High padding ratio within a batch, wasted compute Length-bucketed sampling
    Sparse rewards Delayed success/failure signal, large gradient noise Group sampling + relative advantage estimation, supplemented by process rewards
    Inference/training mismatch Sampling and training engines differ in operator implementation, distorting importance ratios Fix the sampling engine version, or use trajectory-level importance ratios
    MoE routing jitter The same input selects different experts in rollout vs. training Control non-determinism, fix routing statistics

    8. Three Techniques Most Worth Reusing

    8.1 Technique 1: Lower-Bounded Log-Decay Parameterization

    Dimension Description
    Reuse value High: engineering changes concentrate in the gate activation and chunked operators, with significant payoff
    Problem solved The linear-attention diagonal block cannot be parallelized on Tensor Cores, and values may underflow
    Mechanism g = g_min · σ(az), g_min = −5, decay magnitude constrained to a finite interval
    Relative benefit Diagonal and off-diagonal blocks both go through dense matmul; naturally fits long-context extrapolation under NoPE
    Implementation tip Chunk length and g_min must match (chunk length × \ g_min\ cannot exceed the exponent range of the target precision); per-channel decay requires custom operator support

    8.2 Technique 2: Quantile Balancing (QB)

    Dimension Description
    Reuse value High: a small change to the routing layer removes the auxiliary loss and reduces tuning burden
    Problem solved The auxiliary loss interferes with the main task, and a fixed-coefficient balance cannot keep up with routing evolution
    Mechanism Maintain a sliding-window load, take a quantile as reference, update a routing-only bias by step size
    Relative benefit No auxiliary loss, dynamic balance, minimal extra overhead
    Implementation tip The bias does not participate in gradients; adjust the update step when expert scale changes; watch the bias on/off switch during evaluation

    8.3 Technique 3: Block-Level Attention Residuals (Block AttnRes)

    Dimension Description
    Reuse value Medium-high: the network-structure change involves retraining, but the memory and communication gains are direct
    Problem solved Representation homogenization in deep Transformers; the cache and communication cost of per-layer attention residuals
    Mechanism Share one attention residual output within a block, passed step by step across blocks
    Relative benefit Attention-related activations and communication drop to 1/m of the original (m = block size, illustrative)
    Implementation tip Block size must trade off “expression granularity” against “cost”; ablate jointly with depth and parallel topology

    9. Deployment Perspective: Inference, Cost, Evaluation, and Selection

    9.1 The Inference-Side Cost Ledger

    Taking this sample as an example, assume hidden dimension 8192 with latent attention (576 dims per token per layer), giving about 72 KB of cache per token:

    Concurrency scenario Cache size (illustrative estimate)
    One 128K-context session ≈ 9 GB
    One 1M-context session ≈ 74 GB
    Eight concurrent 128K sessions ≈ 74 GB
    Sixteen concurrent 32K sessions ≈ 37 GB

    This is only the cache itself, not counting weights, runtime buffers, or fragmentation. Trillion-scale weights mean that a single long-context request often engages an entire group of servers. To bring costs down, start from five angles:

    1. Prefix cache reuse: When documents such as contracts, manuals, and code repositories are questioned repeatedly, a cache hit can skip the repeated prefill computation;
    2. Prefill/decode disaggregation: Let prefill clusters focus on long inputs and decode clusters focus on generation, scaling each independently along its own bottleneck;
    3. Tiered caching: Offload latent-vector caches by heat to host memory or a distributed cache, keeping single-card memory for active sessions;
    4. Speculative decoding: Use a lightweight draft model or a self-speculative strategy to relieve the decode bandwidth bottleneck under long context;
    5. Quantization and sparsification: Weight quantization reduces memory, cache quantization increases concurrency; choose tiers according to the business’s quality tolerance.

    9.2 Evaluation: From “How Much Fits” to “How Well It Is Used”

    Build a three-layer evaluation:

    • Retrieval layer: Insert key information at different depths of a long document (beginning, middle, end) and examine whether the model can pinpoint it accurately;
    • Reasoning layer: Questions that require evidence across multiple locations, examining information-integration ability;
    • Task layer: Document summarization, contract-clause comparison, table analysis, and multi-turn writing, examining end-to-end effectiveness.

    Also watch a few “counterintuitive” metrics:

    • Effective context length: the input length at which accuracy begins to drop noticeably;
    • Time to first token (TTFT): the first experience a user perceives under long inputs;
    • Cache share per request: the larger the cache, the fewer concurrent sessions can be served.

    9.3 Selection Guidance

    Scenario Key demand Focus
    Long-document Q&A and summarization Stable retrieval and integration Effective context length, cache cost, hit rate
    Multi-file comparison and writing Cross-document consistency Instruction following, structural stability
    Table and chart understanding in files Multimodal parsing Visual parsing quality, coordinate and numeric accuracy
    Large-scale batch document processing Throughput and cost Concurrency, elastic scaling
    Enterprise intranet deployment Data stays in domain Private deployment form, auditing and permissions

    On deployment form, enterprises can choose flexibly according to data-compliance and cost requirements. For example, platforms such as Baidu AI Cloud typically offer general capabilities like cloud APIs, private deployment, and heterogeneous compute scheduling, allowing the cache tiering, elastic scaling, and monitoring/alerting needed for long-context inference to be managed within unified infrastructure.


    10. Conclusion: Beyond Capability, It Is Systems Engineering of “Efficiency and Stability”

    The significance of this class of model goes beyond “bigger parameters, longer context”:

    • On the attention side, latent attention, lower-bounded-decay linear attention, and block-level attention residuals answer the three questions of storage, compute, and depth;
    • On the sparse side, Stable LatentMoE, SiTU-GLU, and quantile balancing answer how to keep training stable at extreme expert scale;
    • On the training flow, multi-stage context extension and million-token-level agentic post-training move “long context” from “can read” to “can use.”

    For developers, the most valuable takeaway is not a single trick but a methodology:

    1. Do the math first: estimate memory from total parameters, compute from activated parameters, and cache from context length;
    2. Then find the bottleneck: compute, storage, and effective utilization often cannot be optimal at the same time and require combined design;
    3. Finally look at stability: sparsification amplifies routing and numerical problems, and the “unremarkable” mechanisms—balance, gating, bounded parameterization—are the preconditions for a model to train all the way to extreme lengths.

    Context length will keep growing, and expert scale will keep expanding. The truly scarce capability is finding a sustainable balance point among compute, memory, and stability.