Ultra-Long-Context Multimodal LLM Architecture: A Deep Dive into Attention, MoE, and Training Strategy

5 hr ago0 views

This article breaks down the architecture of a trillion-parameter multimodal large language model built for ultra-long contexts. It explains how the model pairs a 2.8-trillion-parameter total footprint with only about 104 billion activated parameters per forward pass, how it combines incremental attention (KDA) with gated multi-head latent attention (Gated MLA) to handle context windows of up to 1 million tokens, and how an 896-expert MoE design stays stable during training through quantile-based load balancing and latent-space expert routing. It also covers the model's multi-stage pretraining schedule — from 8K to 1M tokens — its Agentic reinforcement learning post-training, and the engineering considerations developers should keep in mind when deploying sparse, long-context models.

1. Model Overview: Positioning a Large-Scale Multimodal, Long-Context Architecture

In recent years, the capability ceiling of large language models has kept rising, but simply scaling up parameter counts does not guarantee linear returns. What actually determines a model’s practical value is usually its combined performance across three dimensions: multimodal input, ultra-long-context handling, and stability under large-scale sparse training. The model discussed in this article addresses all three dimensions at the architecture level, forming a relatively complete technical solution.

In terms of scale, the model has 2.8 trillion total parameters but activates only about 104 billion per forward pass. This “large total, small activation” design is fundamentally a trade-off between model capacity and inference cost: total parameters define the upper bound on the knowledge and skills the model can store, while activated parameters directly determine the compute consumed by each inference. Controlling the activation size is especially critical for long-context workloads, because once context length reaches hundreds of thousands or even millions of tokens, the cost of attention computation rises sharply.

The model natively supports visual multimodal input — image information is not attached through external plug-ins or post-hoc adapters, but is processed jointly with text representations inside the model itself. The advantage of a native multimodal design is that visual and language features can interact at earlier layers, which helps the model learn cross-modal alignment. The model also supports context windows of up to 1 million tokens, an aggressive target that requires targeted adjustments across attention, positional encoding, memory management, and training data scheduling.

Notably, the model does not enable 1M-token contexts from the start of training. Instead, it adopts a multi-stage pretraining strategy, extending the context length from 8K to 64K, then 256K, and finally 1M. This progressive expansion allows the model to first build stable language modeling at shorter lengths and then gradually adapt to longer-range dependencies, reducing instability during training.

2. Attention Mechanisms: Fusing Incremental Attention with Gated Latent Attention

The first challenge for ultra-long contexts comes from the computational complexity of attention. Standard self-attention scales quadratically with sequence length; at a million tokens, computing the full attention matrix directly is infeasible in both compute and memory. The model adopts a fused design that combines two complementary attention mechanisms.

Incremental attention (KDA) relies on caching to eliminate repeated computation. During autoregressive generation, the key and value tensors of past tokens can be cached and reused, so each step only computes attention between the new token and the cached history. This reduces the per-token attention cost from scaling with the full sequence length to scaling with the incremental portion. Caching alone, however, cannot fundamentally solve the memory problem in long-context scenarios — the cached key-value tensors themselves grow linearly with the context.

Gated multi-head latent attention (Gated MLA) attacks the problem from another angle. MLA uses low-rank compression to project each token’s keys and values into a latent space whose dimension is far smaller than the hidden layer’s. The added gating lets the model dynamically control how much each attention head relies on the compressed features. Its core benefit is a significant reduction in the size of the key-value cache required per token, while preserving the expressive flexibility of multi-head attention.

The fusion logic works as follows: KDA handles incremental computation and information reuse efficiently, while Gated MLA compresses the cache footprint without sacrificing expressiveness. Combined, the model can generate long text without recomputing attention over the entire history at every step, and without exhausting memory through unbounded cache growth.

To further improve long-range information flow, the model also introduces an attention residual (AttnRes) mechanism: attention outputs from lower layers are carried through residual connections to higher layers, so deep layers can directly reuse the global dependency information already extracted by shallower ones. This mitigates the “information dilution” problem that deep attention can suffer on ultra-long sequences — the difficulty a single attention layer faces when trying to focus on key information within an extremely long sequence.

3. Sparse Expert Networks: Engineering Stability for Massive-Scale MoE

Once total parameters reach the trillion scale, a dense architecture would activate every parameter on every forward pass, driving inference costs to impractical levels. Sparse mixture-of-experts (MoE) offers another route: each layer contains multiple parallel feed-forward expert modules, but only a subset is routed and activated per computation. This model’s MoE stack includes 896 routed experts, of which only a small number are activated each time.

The value of sparse MoE is that it decouples knowledge capacity from compute cost. The model can scale total parameters by adding experts — storing more specialized knowledge — while actual inference compute depends only on the number of activated experts. This design, however, introduces significant training challenges.

The first is router collapse. Early in training, the router may tend to assign most tokens to a handful of experts, leaving the rest under-trained and producing a “winner takes all” outcome. The model addresses this with a quantile balance (QB) mechanism, which applies quantile statistics and dynamic adjustment to the expert load distribution so that all experts receive relatively balanced gradient signals during training. The core idea is not to force every expert to receive exactly the same number of tokens, but to avoid extremely uneven load distributions in a statistical sense.

The second is numerical stability. With very few activated parameters spread across a very large number of experts, the effective gradient reaching any single expert can be extremely sparse, making loss spikes likely during training. The model adopts a Stable LatentMoE architecture that introduces a latent-space projection at the expert input while organizing the routing logic hierarchically. This partially decouples routing decisions from the experts’ internal computation, smooths the gradient propagation path, and reduces numerical fluctuation during training.

At the activation function level, the model uses SiTU-GLU. GLU-family activations are well validated in Transformer feed-forward networks, and their gating improves selective information transfer. SiTU further reshapes the activation response distribution so that the network responds more smoothly to extreme input values in deep sparse layers. This has a direct effect on training stability: input distributions in sparse expert networks tend to have heavier tails and larger variance, and an activation function that produces oversized gradients in extreme regions can easily trigger gradient explosions.

4. Training Strategy: Multi-Stage Pretraining and Agentic Reinforcement Learning

Training proceeds in two main stages: multi-stage pretraining and Agentic reinforcement learning (RL) post-training. The two stages differ clearly in both objectives and data composition.

The goal of multi-stage pretraining is to progressively establish stable language modeling and cross-modal understanding across different context lengths. The 8K stage builds basic language capability and vision-text alignment; the 64K stage introduces longer document-level dependencies and cross-paragraph reasoning; the 256K stage focuses on long documents and repository-scale code contexts; and the final 1M stage is dedicated to extreme-length sequences, including global information retrieval and long-range causal reasoning.

The engineering rationale for this staged expansion is straightforward: training directly on 1M-token sequences from scratch would expose the model to long-sequence noise while it is still learning basic language patterns, hurting training efficiency. Progressive expansion lets the model converge on core capabilities with short sequences first, then incrementally adapt to longer-range dependency patterns, with each stage building on the previous one.

Agentic RL post-training has a more focused objective: improving the model’s ability to plan, decide, and execute as an agent in long-horizon tasks. Unlike conventional supervised fine-tuning, Agentic RL emphasizes learning from feedback through trial and error in multi-step task environments. Within a million-token context framework, this training is particularly meaningful — the agent must track its goals, evaluate intermediate results, and adjust its strategy across very long interaction histories.

During Agentic RL, the trajectories the model processes may span hundreds of thousands or even millions of tokens. This demands two capabilities: long-range memory and information recall — the ability to locate key content within a massive interaction history when needed — and policy stability, meaning the agent does not frequently drift away from its global objective because of local reward fluctuations. The latter is especially important in practice, since agent tasks are often characterized by sparse rewards: meaningful feedback arrives only when the task is completed or a milestone is reached.

5. Engineering Observations and Takeaways

Viewed from a developer’s perspective, several aspects of this architecture deserve attention.

First, the ratio of activated to total parameters directly shapes the cost structure of serving deployments. With roughly 104 billion activated parameters, per-inference compute demand remains high, placing significant requirements on inference acceleration frameworks and hardware utilization. In production, developers need to pay close attention to KV cache management, expert routing schedules, and batching efficiency.

Second, ultra-long context does not mean unlimited context. Even though the model accepts up to a million input tokens, making effective use of the full window raises the bar for prompt engineering and task design. Information density naturally varies between the beginning and end of a context, and the model’s attention to different positions is uneven. Application layers still need to organize and stratify context carefully.

Third, inference scheduling for sparse MoE is a systems engineering problem. The parameters of 896 experts require efficient expert selection and loading at inference time. For server-side deployments, how expert parameters are laid out in memory and cached directly determines throughput and service quality.