Architecture Deep Dive: Sparse Multimodal LLMs for Million-Token Contexts

1. Background: When Context Windows Move Toward the Million-Token Scale

The capability frontier of large language models is shifting from single-turn Q&A to long-horizon agentic tasks. When a model needs to ingest code repositories, multimodal documents, or long conversation histories spanning hundreds of thousands or even millions of tokens in a single pass, conventional dense attention mechanisms hit two hard constraints: computational complexity grows quadratically with sequence length, and key-value cache memory scales linearly. These two factors directly determine the engineering ceiling for long-context capability.

The industry has largely converged on three main approaches. The first is replacing some full-attention layers with linear attention or state-space structures, reducing sequence-dimension computation to approximately linear. The second is adopting sparse Mixture-of-Experts (MoE) architectures, where a large total parameter count is driven by a much smaller set of activated parameters. The third is progressively extending context length through staged pretraining, then introducing agentic reinforcement learning during post-training so that models learn to plan and retrieve over long-horizon tasks.

A recently disclosed technical approach from a multimodal sparse model offers a fairly complete engineering answer: total parameters at the trillion scale, activated parameters at the hundred-billion scale, native visual multimodal input, million-token context support, and an accompanying public technical report. This article does not evaluate a specific product. Instead, it dissects three techniques with the highest engineering portability from an architecture design perspective: lower-bound decay linear attention parameterization, quantile-based load balancing, and block-wise attention residuals.

2. The Compute Math for Ultra-Long Contexts: Why Attention Must Change

Let us start with the numbers. Standard self-attention over a sequence of length N requires roughly O(N²·d) computation, with key-value cache memory of O(N·d·L), where d is the hidden dimension and L is the number of layers. When N grows from 8K to 1M tokens, computation increases by roughly 16,000x and cache by roughly 128x. Even aggressive block-wise computation, recomputation, and parallelization strategies cannot push this back into an acceptable range.

As a result, long-context models commonly adopt hybrid attention layers: some layers keep full attention for precise retrieval, while others switch to linear attention or incremental attention, bringing the backbone computation back to linear complexity. These linear attention variants typically rely on a decay factor Γ that accumulates along the sequence to control the weight of historical information:

S_t = Γ_t ⊙ S_{t-1} + K_t^T V_t

The problem lies precisely in this Γ.

3. Lower-Bound Decay Parameterization: Keeping Linear Attention on Tensor Cores

3.1 Numerical Overflow and the Diagonal Block Problem

In chunked computation, the reciprocal of the accumulated decay factor must be computed within each chunk to recover relative weights between positions inside the chunk. When the decay factor itself becomes extremely small after repeated multiplication, its reciprocal overflows directly under BF16 precision. Under other configurations, it underflows to zero. As a result, the diagonal region within a chunk cannot be expressed through regular matrix multiplication and must fall back to explicit per-position pairwise computation.

This fallback path is expensive: it bypasses tensor-core matrix multiplication and becomes a memory-bound element-wise operation, often reducing throughput by several times. For million-token contexts, this is unacceptable.

3.2 A Bounded Reparameterization

The fix is to reparameterize the unbounded decay into a bounded scaling form. A common approach is to replace the unconstrained negative Softplus form in log space with a scaled Sigmoid form that has a fixed lower bound, such as a constant like -5. This keeps the reciprocal scaling factor within any fixed-length chunk inside a safe numerical range.

The engineering value of this change:

• Minimal modification surface: only the decay parameter formula changes. The overall layer structure stays intact, making it an operator-level replacement.
• Restores the matrix-multiplication path: the in-chunk diagonal region can be computed through standard matrix multiplication, directly benefiting from tensor-core throughput.
• Numerically stable: no overflow under BF16 and no need for a separate high-precision branch for diagonal blocks.

From a migration perspective, any implementation using linear attention or incremental attention with in-chunk reciprocal scaling can evaluate this change. It belongs to the category of optimizations that are small in code footprint and immediate in throughput impact.

4. Quantile-Based Balancing: Load Balancing for Very Large Sparse MoE

4.1 Routing Collapse and the Cost of Auxiliary Losses

The core benefit of MoE comes from conditional computation: each token activates only a small number of experts, keeping per-forward-pass compute manageable despite a very large total parameter count. But when the number of experts reaches hundreds or nearly a thousand and the activation ratio is extremely low, training becomes prone to routing collapse — a small number of experts handle most tokens while the rest receive little useful gradient, wasting capacity.

The conventional fix is an auxiliary loss that penalizes imbalanced routing. It has two side effects. First, the auxiliary loss competes with the main task loss, making the weighting sensitive to tuning. Second, it imposes only a soft constraint, which may fail under extreme sparsity.

4.2 Dynamic Balancing Through Quantiles

An alternative is auxiliary-loss-free dynamic load balancing. The idea: compute quantile statistics over the routing score distribution each expert receives, then use quantile thresholds to dynamically adjust routing biases so that actual load per expert converges toward the target mean. This acts directly on routing decisions rather than through an indirect loss signal.

Key advantages:

1. No gradient interference: balancing logic does not enter the main loss, keeping the training objective cleaner.
2. Fast response: quantile statistics can be updated per step, adapting to distributions that shift during training.
3. Robust under extreme sparsity: expert utilization can be maintained even at very low activation ratios.

For teams training their own MoE models, the main value of this mechanism is reduced hyperparameter sensitivity — there is less need for extensive ablation studies on auxiliary loss weights.

5. Block-Wise Attention Residuals: Opening Up Deep Representation Pathways

5.1 The Representation Bottleneck from Depth

When Transformers are stacked very deep, a phenomenon resembling “representation collapse” can appear: features learned in shallow layers are progressively overwritten during forward propagation, and deeper layers have difficulty accessing early information directly. Residual connections mitigate vanishing gradients but do not explicitly solve the cross-layer information-sharing problem.

5.2 Engineering Cross-Layer Attention

Block-wise attention residuals establish cross-layer attention pathways among groups of layers, aggregating and distributing intermediate representations at block granularity. Compared with dense layer-by-layer cross-layer attention, the block-wise design significantly reduces memory and communication overhead:

• Only block-level rather than layer-level attention matrices are retained, keeping memory usage manageable.
• Communication patterns are regular, making implementation easier under tensor parallelism and pipeline parallelism.
• Deeper layers can directly reference representations from earlier layers within the same block, mitigating information decay.

This technique has broad relevance for scaling model depth, especially when pipeline-parallel training is required across multiple nodes and GPUs.

6. How the Training Pipeline Fits Together

When these techniques are placed into a full training pipeline, the logic is coherent:

Stage	Goal	Supporting Technique
Pretraining context extension	Progressively scale from 8K to 64K, 256K, 1M	Linear attention + long-context data mixing
Sparse capacity scaling	Trillion parameters, hundred-billion activated	Large-scale MoE + dynamic load balancing
Deep representation optimization	Preserve information across deep layers	Cross-layer attention residuals
Post-training	Long-horizon agentic reasoning	Million-token-scale reinforcement learning

Progressive context extension during pretraining allows the model to transition smoothly in distribution, avoiding instability from directly training on million-length sequences. Million-token-scale agentic reinforcement learning post-training then moves the model from “able to read long text” to “able to plan, retrieve, and verify autonomously in long tasks.”

7. Portable Takeaways for Developers

Each of the three techniques addresses a distinct bottleneck and is reusable across projects:

1. Bounded decay parameterization: applies to any implementation using linear attention or incremental attention. Small change, direct throughput benefit, with the core goal of restoring in-chunk matrix-multiplication paths.
2. Quantile-based dynamic load balancing: applies to in-house ultra-large MoE training. Replaces auxiliary-loss approaches and reduces tuning cost.
3. Block-wise cross-layer attention residuals: applies to representation optimization in deep Transformers, improving cross-layer information flow under controlled memory usage.

Ultra-long context and extreme sparsity are fundamentally problems of redistributing computation under resource constraints. Understanding the numerical stability, communication overhead, and training dynamics behind these architectural choices has more long-term value than memorizing any specific benchmark number. For teams building long-context capability on their own stack, starting with an operator-level lower-bound fix is often the highest return on investment.