Million-Token Context and Ultra-Sparse MoE Architecture Explained: A Technical Breakdown

    This article breaks down the architecture of a recent open-source multimodal large model that combines native vision-language understanding, a 1-million-token context window, and an ultra-sparse Mixture-of-Experts (MoE) design with 896 routed experts. It examines the two scaling pressures driving the design—parameter count and context length—and then walks through the reusable techniques used to address them: lower-bounded decay parameterization for linear attention, gated MLA combined with block-level attention residuals, latent-space low-rank routing, gated SiLU activations, aux-loss-free quantile load balancing, and a multi-stage context curriculum paired with long-horizon agentic RL. Each technique is assessed by its cost-to-benefit ratio and is accompanied by an implementation checklist and a list of common pitfalls.

    1. Background: Two Difficulty Curves Rising at Once

    Over the past two years, the capability expansion of large models has mainly proceeded along two axes: parameter scale and context length. When the two are stacked together, the problem can no longer be solved by “throwing more GPUs at it.” Instead, three coupled hard constraints emerge:

    1. Sequence length drives quadratic growth in attention compute; at 1M tokens, naive full attention is infeasible.
    2. Once total parameters enter the trillion range, the forward/backward memory and communication volume of a dense structure exceed the normal capacity of a single machine or even a single cluster, forcing a move toward extreme sparsity (2.8 trillion total parameters, with roughly 104 billion activated per token).
    3. Sparsification itself introduces training instability: routing collapse, expert load imbalance, deep representation degradation, and difficult long-range credit assignment.

    A technical report from a recent open-source multimodal large model discloses a scheme that is precisely a systematic answer to these three constraints: native visual multimodality, a 1-million-token context, an ultra-sparse MoE with 896 routed experts, linear attention fused with Gated Multi-head Latent Attention (Gated MLA) on the attention side, and a latent-space low-rank routing structure on the MoE side. The following sections break this down from the highest to the lowest reusability value.

    2. Overall Architecture: A Three-Layer Design

    1. Input tokens (text / image patches)
    2. ↓
    3. [ Embedding ]
    4. ↓
    5. ┌──── Repeat N Blocks ────────────────────────────┐
    6. │ Block AttnRes: cross-block weighted aggregation (not layer-wise identity) │
    7. │ ├─ Linear attention layers (with lower-bounded decay, majority) │
    8. │ └─ Gated MLA layers (a few full-attention layers) │
    9. │ → MoE FFN: 896 routed experts + shared experts │
    10. │ · Latent-space low-rank routing │
    11. │ · Gated SiLU-family activation │
    12. │ · Quantile-based dynamic load balancing (aux-loss-free) │
    13. └─────────────────────────────────────────────────┘
    14. ↓
    15. [ Output Head ]

    The design logic is clear: attention handles “breadth” (long-range information aggregation), MoE handles “capacity” (parameter count), and the residual structure handles “depth” (information transfer between layers).

    3. Attention Side: Lower-Bounded Decay Parameterization Is the Highest-ROI Change

    3.1 The Pain Point: Diagonal Blocks Get Kicked Out of the Tensor Core

    Chunk-parallel implementations of linear attention and incremental-network-style methods split computation into two categories:

    • Off-diagonal blocks: involve only forward cumulative decay, equivalent to a standard matrix multiply that can run directly on Tensor Cores.
    • Diagonal blocks: involve pairwise token interactions within a block and require a triangular matrix inverse (UT transform / WY representation), which contains reciprocal terms of the cumulative decay, 1/Γ.

    If the log-decay has no lower bound, the cumulative decay Γ can approach 0 arbitrarily closely, and 1/Γ will inevitably overflow in BF16. A common engineering workaround routes diagonal blocks through a separate explicit “position-pair” path, using element-wise multiply-accumulate instead of GEMM. The consequence: this operator is both slow and hard to fuse into a kernel, and it introduces an extra branch, making a high-performance implementation very difficult on CUDA / CUTLASS.

    3.2 The Mechanism: Scaled Sigmoid Replacing Negative-Softplus

    The key change is a single line: replace the unbounded log-decay parameterization with a scaled sigmoid that has a fixed lower bound.

    1. # Original approach (no lower bound): log_decay ∈ (-inf, 0]
    2. # g = -softplus(a @ h) # Can be arbitrarily small, 1/Γ overflows
    3. # Lower-bounded decay parameterization (g_min is a hyperparameter, typically -5)
    4. g_min = -5.0
    5. g = g_min * torch.sigmoid(a @ h) # log_decay ∈ (g_min, 0)
    6. decay = g.exp() # Single-step retention ∈ [e^g_min, 1)

    The effect is that single-step retention is locked into [e^-5, 1) ≈ [0.0067, 1). The reason -5 and the 16-token tile are a matched pair can be derived directly:

    • Within any 16-token block, the cumulative-decay lower bound is e^(-5×16) = e^-80 ≈ 1.8e-35;
    • Its reciprocal is about e^80 ≈ 5.5e34, still within the representable range of BF16 (upper limit ~3.39e38);
    • If the block size is changed to 32 tokens, the reciprocal becomes e^160, which overflows directly.

    This yields a general parameter-selection rule: tile_size × |g_min| ≤ ln(BF16_MAX) ≈ 88, i.e., |g_min| ≲ 88 / tile_size. This is a more valuable conclusion than “just copy -5”—changing the tile size or the precision (FP16’s upper limit is 65504, ln≈11, so g_min must be tightened to about -0.7 per token) both require recomputation.

    3.3 The Benefits

    • Eliminates the special-case branch for diagonal blocks; diagonal and off-diagonal causal tiles both run through dense Tensor Core GEMM, reducing both kernel count and maintenance cost.
    • Keeps the numerical range controllable, so BF16 training does not need a separate FP32 upcast for diagonal blocks.
    • A hard lower bound on decay means the model never “forgets entirely,” giving more controllable long-range information retention.
    • Combined with NoPE (no positional encoding), the linear attention layers do not depend on rotary positional encoding, so 1M context can be extrapolated more naturally without recalibrating the RoPE base.

    4. Block-Level Attention Residual (Block AttnRes)

    A typical degeneration in deep Transformers is the “deep representation bottleneck”: layer-wise identity residuals yield diminishing marginal returns after dozens of layers, and backpropagation must store the activations of every layer, so memory and pipeline-parallel communication grow linearly with depth.

    The block-level residual approach partitions L layers into several blocks. Each block’s input no longer comes only from the previous layer, but from a learnable weighted aggregation of the outputs of several preceding blocks:

    1. # 4 layers per block, aggregate the previous 3 block outputs
    2. x = w0 * emb + w1 * blk1_out + w2 * blk2_out + w3 * blk3_out
    3. x = Block4Layers(x) # Internally still standard residual
    4. blk4_out = x

    Three benefits: cross-layer information gets a direct connection, mitigating deep representation collapse; backpropagation only needs to retain block-boundary tensors, significantly reducing activation memory; and the pipeline-parallel stage boundaries align with block boundaries, so point-to-point communication is transferred at block granularity rather than layer granularity.

    5. MoE Side: A Stability Trio Under Extreme Sparsity

    5.1 Latent-Space Low-Rank Routing

    At a scale of 896 experts, performing top-k scoring directly in the token latent space causes two problems: the gating matrix has a large parameter count and compute cost, and the scores are sensitive to the scale of the hidden states. Projecting the scoring into a low-dimensional latent space and then back up (low-rank routing) significantly reduces gating overhead and smooths the score distribution, making the top-k boundary more stable.

    5.2 Gated SiLU-Family Activation

    Beyond the standard SiLU(xW) ⊙ (xV) gating structure, this class of schemes introduces learnable scaling and shift parameters. The goal is to stabilize the mean and variance of activations in an extremely deep and extremely sparse network, preventing a few experts from “eating” too much gradient due to activation drift. In engineering terms it can be seen as scale normalization applied to GLU-family activations, at minimal cost.

    5.3 Quantile-Based Dynamic Load Balancing (Aux-Loss-Free)

    The traditional approach adds a load-balancing auxiliary loss to the training objective. The problem: the auxiliary loss’s gradient pollutes the main-task gradient, artificially distorts the expert-selection signal, and requires careful tuning of its coefficient.

    Quantile balancing instead becomes purely online control: it only adjusts a discrete scoring bias bias[i] for each expert, which does not enter the backward graph.

    1. # Updated after each step ends, does not receive gradients
    2. load = all_reduce(expert_token_counts) # Global load
    3. target = total_tokens * expert_share[i] # Target quantile share (uniform by default)
    4. err = (load[i] - target) / target
    5. # Both PID and sign-based updates work; the learning rate must be far smaller than the main optimizer
    6. bias[i] -= lr_bal * err
    7. # Use score[i] + bias[i] for top-k during the forward pass, but use score[i] to compute gating weights

    Key points:

    • The bias only affects routing decisions, not the weighting coefficients, so it does not change the semantic scale of expert outputs;
    • The target share can be any distribution, naturally supporting non-uniform ratios such as “shared experts + routed experts”;
    • After removing the auxiliary loss, language-modeling loss becomes the only gradient source, so sparse training converges more cleanly;
    • Monitoring metrics should look at the entropy and CV (coefficient of variation) of the load distribution, not just peak load, to avoid “the mean meets target but the long tail starves.”

    6. Training Curriculum: Progressive Context Laddering + Long-Horizon Agentic RL

    Context length is not maxed out in one shot. The path disclosed in the report is a multi-stage pretraining of 8K → 64K → 256K → 1M, with each stage paired with adjustments:

    • Data mixture: short stages focus on high-quality general corpora; long stages introduce naturally long-sequence samples such as long documents, code repositories, and books.
    • Packing and masking: when packing multiple samples into one sequence, a document-level attention mask is mandatory to prevent cross-document attention contamination.
    • Positional scheme: linear attention layers use NoPE, while the few full-attention layers retain positional encoding. The two have different length-extrapolation properties and must be validated stage by stage.
    • Memory strategy: when sequence length doubles, activation memory is the main bottleneck, addressed with block-level residuals, recomputation, and sequence parallelism.
    • Checkpoints: at the end of each stage, run long-context diagnostics (long-document retrieval, multi-hop QA, cross-paragraph coreference), and feed failing samples back into the next stage.

    On the post-training side, million-token-level Agentic RL is introduced, whose core difficulty is long-horizon credit assignment: trajectories can span hundreds of thousands of tokens, with sparse final rewards. Feasible practices include relying primarily on verifiable rewards (task completion, unit-test pass rate) and avoiding model rewards; using trajectory length as a curriculum, gradually extending from short tasks; and for very long trajectories, adopting segmented value estimation or outcome-reward backpropagation to reduce variance.

    7. Reuse Priority and Implementation Checklist

    Ordered by “change size / benefit ratio”:

    1. Lower-bounded decay parameterization: the change is only a few lines and directly improves both the GPU implementation efficiency and numerical safety of the linear attention operator. Land this first.
    2. Quantile load balancing: removing the auxiliary loss in favor of online bias control has a positive effect on both convergence stability and tuning cost.
    3. Block-level attention residual: requires modifying the model skeleton and the residual/recomputation logic of the training framework. The benefits are in depth and memory, making it a mid-term investment.
    4. Multi-stage context curriculum + Agentic RL: the largest investment, but it determines the ceiling for long-context and tool-use capabilities.

    Implementation typically requires supporting engineering capabilities such as: custom operator development and performance profiling; expert parallelism and overlap of all-to-all communication in the distributed training framework; a training-data pipeline (can be decoupled using object storage and message queues); long-context KV caching and chunked prefill on the inference side; and observability for training/serving based on log services and monitoring alerts.

    8. Common Pitfalls

    • Maxing out context blindly: if neither the data nor the evaluation needs long sequences, the long-context stage only burns compute without lifting scores.
    • Judging load balancing by training loss alone: a stable loss does not mean experts are not starving; routing entropy must be monitored independently.
    • Copying the g_min hyperparameter directly: the lower bound must be derived jointly with tile size and compute precision, and becomes invalid when the configuration changes.
    • Guessing the mix ratio of attention-layer types: too few full-attention layers hurt precise retrieval, while too many bring the long-context cost back up. Derive it from evaluation results.
    • Evaluating Agentic capability with single-step metrics: use end-to-end task success rate, average interaction turns, and tool-call correctness as the primary metrics.