Architecture and Training Practices for Ultra-Large Sparse Multimodal Models

1. Background: Why Trillion-Parameter Models Need a New Architectural Paradigm

When model parameter counts grow from hundreds of billions to trillions, input lengths expand from tens of thousands to millions of tokens, and models must process text and visual signals simultaneously, the engineering bottlenecks of conventional dense models and standard sparse models surface all at once. The core tensions appear at three levels.

First, the tension between attention computation and context length. The compute and memory overhead of standard self-attention on long sequences grows roughly quadratically with sequence length. Even with caching and chunking strategies, attention over million-token sequences can remain the dominant bottleneck in both training and inference.

Second, the tension between sparse expert scale and training stability. When the number of experts reaches hundreds or even nearly a thousand, routing becomes prone to load imbalance, expert degradation, and gradient instability. Some experts may remain permanently inactive while others stay overloaded, ultimately preventing the model capacity from being effectively utilized.

Third, the tension between long-horizon reasoning and agentic behavior learning. Models built for agentic scenarios must not only understand long documents but also maintain long-term consistency and verifiability across multi-step tool calls, environment feedback, and trajectory planning. This requires the post-training stage to construct interaction trajectories at the million-token level and to let the model learn stably from sparse rewards.

Consequently, a model with trillions of parameters, native multimodality, and ultra-long context cannot be achieved by stacking parameters alone. It requires systematic reconstruction across attention, sparse routing, activation functions, and training schedules.

2. Overall Architecture: A Multimodal Backbone Under Sparse Activation

The model discussed here follows a typical Mixture-of-Experts sparse architecture. Its total parameter count reaches approximately 2.8 trillion, while each forward pass activates only about 104 billion parameters. This design expands model capacity while controlling per-inference compute.

The backbone can be abstracted into three layers:

1. Input layer and multimodal encoding
   Text and visual signals are each encoded and then mapped into a unified representation space. The vision side does not rely on a separate external detector; instead, it adopts a native multimodal input approach so that image patch features can enter the backbone directly alongside text tokens.

2. Stacked Transformer layers
   Each layer contains both an attention module and a sparse feed-forward module. The attention part handles intra-sequence information interaction, while the sparse feed-forward part expands model capacity. Residual connections and normalization between the two keep signals stable.

3. Long-context position representation
   To support a 1M-token context, position encoding must be able to extrapolate. Combined with multi-stage length training, the model gradually adapts to long-range dependencies across different length ranges, avoiding the catastrophic forgetting that results from jumping directly to ultra-long sequences.

The key to this overall design is not “large” but “sparse and stable.” Total parameter scale primarily expands knowledge storage and pattern capacity, while activated parameter scale determines per-inference cost. Once decoupled, the model can gain stronger representational capability without significantly increasing computational expense.

3. Attention Mechanism: Fusing Incremental Attention with Gated Latent Attention

The attention module is one of the most representative designs in this model. It is not a single implementation but a fusion of two attention mechanisms, with cross-layer residuals completing the information enhancement.

3.1 Incremental Attention

Incremental attention primarily addresses new-information retrieval in long sequences. In long documents or long trajectories, the model needs to quickly locate relevant key-values in historical segments based on the current query. Conventional full attention computes similarity against all historical tokens, whereas incremental attention reduces repeated scanning cost by maintaining an updatable index or summary structure.

Its core idea is:

• Divide the historical sequence into chunks.
• Maintain a learnable summary vector or index vector for each chunk.
• Have the current query first interact with summary vectors to filter high-relevance chunks, then proceed to fine-grained attention.

This approach transforms “global scanning” into “filter first, then read closely,” significantly reducing computational complexity in ultra-long contexts.

3.2 Gated Multi-Head Latent Attention

Gated multi-head latent attention focuses on compressing the key-value cache. Conventional multi-head attention stores independent keys and values for each head; when layers are deep, heads are numerous, and context is long, cache size balloons rapidly.

Gated multi-head latent attention projects keys and values into a low-dimensional latent space and, on the query side, uses a gating mechanism to dynamically select which latent dimensions need to be recovered. It can be understood as a lossy but learnable key-value compression scheme. The role of gating is to let the model adaptively determine compression strength based on input content: preserving more detail in high-information-density regions and compressing redundant regions more aggressively.

3.3 Attention Residuals

A common problem in deep models is that deep-layer attention information decays or drifts layer by layer. The attention residual mechanism allows the attention output of a given layer to pass across multiple layers to higher layers, preserving key relationships captured in earlier layers.

For example, the attention result of layer i not only enters layer i+1 but can also be added to the attention output of layer i+k via a residual path. This way, even if intermediate layers undergo representation changes, early attention can still influence the final output. This design is especially important for coreference resolution and long-distance dependencies in ultra-long contexts.

4. Sparse Expert System: Stable Routing and Load Balancing

At trillion-parameter scale, a dense feed-forward sublayer would make single-layer parameter counts unacceptable. A sparse expert system therefore becomes necessary. This model configures 896 routed experts on the MoE side and uses a stable latent routing mechanism.

4.1 Latent Routing and Expert Selection

Unlike conventional Top-K gating, which computes expert probabilities directly from the current token representation, latent routing first learns a low-dimensional routing latent variable and then maps it to expert selection. This partially decouples routing decisions from the main representation space, reducing the impact of noise on routing stability.

Typically, each token activates only a few experts, for example 1 to 2. Routing scores are normalized and then weighted-summed with expert outputs. To ensure training stability, gating logits undergo differentiable noise injection or entropy regularization, preventing the model from prematurely degenerating into a state where a few experts always win.

4.2 Quantile Balancing Mechanism

Expert load imbalance is a major obstacle in large-scale MoE training. A common approach is to add an auxiliary load-balancing loss, but ordinary balancing losses can only constrain first-order statistics and cannot effectively handle extreme distributions.

The quantile balancing mechanism regulates load from a distributional perspective. It treats different quantiles of the expert load distribution as constraint targets, for example simultaneously limiting the maximum share of high-load experts and the minimum share of low-load experts. This allows experts in the middle of the distribution to receive sufficient gradient signal, reducing the “winner-takes-all” phenomenon.

Its relationship to the load-balancing loss can be understood as:

• Conventional loss: brings the average load of all experts close together.
• Quantile balancing: makes the shape of the load distribution more reasonable, rather than focusing only on the mean.

4.3 Activation Functions and Training Stability

The sparse expert system introduces new activation function designs, such as SiTU-GLU. This structure can be understood as a gated linear unit augmented with a smoother nonlinear transformation. Compared with conventional GELU or Swish, SiTU-GLU can provide more stable gradient responses in deep sparse models, reducing activation saturation in deep networks.

For a model containing nearly a thousand experts, gradients propagate back through two paths: routing and expert parameters. Anomalies in either path can disrupt the entire MoE layer. The choice of activation function therefore affects not only representational capability but also training stability directly.

5. Multi-Stage Long-Context Pretraining

The model is not trained on 1M-token lengths from the start. Using ultra-long sequences directly leads to wasted computation and unstable convergence. A staged extension strategy is the reasonable approach.

The typical stages can be summarized as:

Stage	Target Length	Core Task
Stage 1	8K	Learn basic language and visual representations
Stage 2	64K	Learn long-document structure and long-distance dependencies
Stage 3	256K	Strengthen cross-document information integration
Stage 4	1M	Model agentic trajectories and ultra-long tasks

At each stage, the data mix is adjusted dynamically. Short-sequence stages emphasize high-quality text and image-text pairs; long-sequence stages gradually introduce books, code repositories, multi-document collections, long video frame sequences, and agent trajectory data.

This yields three benefits:

1. Computational efficiency
   Short-sequence stages can rapidly establish foundational capabilities with larger batches.

2. Smooth extrapolation of position encoding
   Gradually extending length helps position representations remain stable across all length ranges.

3. Progressive capability building
   The model first grasps local semantics, then learns global structure, and finally adapts to ultra-long temporal sequences.

6. Post-Training Methods for Agents

After pretraining, the core goal of the post-training stage is to align model capabilities with agentic tasks. Unlike ordinary dialogue models, agent models need the following capabilities:

• Multi-step planning
• Tool invocation and result interpretation
• Understanding of environment feedback
• Goal maintenance across long trajectories
• Error recovery and replanning

Post-training therefore adopts Agentic Reinforcement Learning at the million-token level. Training samples are no longer single-turn question-answer pairs but complete multi-turn interaction trajectories. A trajectory may contain:

user goal → model plan → tool call → environment return →
model judgment → next-step plan → ... → final result → reward signal

Such trajectories can be far longer than ordinary dialogues, reaching hundreds of thousands or even a million tokens. Reinforcement learning signals typically come from final task completion and may also include process rewards for intermediate steps. To propagate gradients stably across ultra-long trajectories, the training framework must support segmented replay, state caching, and long-horizon credit assignment.

The key difference between Agentic RL and ordinary RLHF is that it optimizes not only “whether the answer is correct” but also “whether the process of achieving the goal is effective.” For example, a model may fail its first tool call, adjust its strategy based on the error message, and ultimately complete the task; such a trajectory should receive a positive reward. This training significantly improves model robustness in complex environments.

7. Key Engineering Considerations

From an engineering perspective, training a model of this scale requires solving a series of infrastructure problems.

7.1 Parallelism Strategy

A single model cannot fit into the memory of one GPU or one machine, so data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism must be combined. Expert parallelism is especially critical for MoE models: different experts reside on different devices, and routing requires cross-device communication. To reduce communication overhead, routing typically uses a capacity factor to limit the maximum number of tokens per expert.

7.2 Checkpointing and Fault Tolerance

Checkpoints for trillion-parameter models are enormous, and both saving and restoring are time-consuming. Engineering practice typically employs asynchronous checkpointing, sharded saving, and fast recovery mechanisms. When a node fails, recovery should proceed from the most recent consistent checkpoint rather than reloading everything from scratch.

7.3 Long-Sequence Memory Management

Intermediate activations and attention caches for million-token sequences require careful management. Common techniques include activation recomputation, chunked attention, key-value cache compression, and selective storage. The inference stage also needs cache eviction policies to prevent memory from growing continuously in long conversation scenarios.

8. How to Evaluate Such Models from a Technical Perspective

For developers or enterprise technical teams, evaluating an ultra-large sparse multimodal model should not rely on leaderboard scores alone. The following dimensions matter:

1. Real utilization of long context
   Whether the model can still accurately locate early information beyond 100K tokens is a more important metric than the supported length alone.

2. Stability of sparse routing
   Monitoring expert load distribution, expert utilization, and routing entropy helps determine whether the MoE is healthy.

3. Multimodal consistency
   The ability to align image and text information in long documents, especially in mixed image-text layouts.

4. Process quality on agentic tasks
   Not just the final answer, but also planning soundness, tool-call efficiency, and error recovery capability.

5. Inference cost and throughput
   Under the same hardware conditions, activated parameter scale, cache compression ratio, and long-sequence throughput directly determine practical usability.

9. Conclusion and Outlook

The architectural design of trillion-parameter sparse multimodal models has moved beyond a pure scale race. Its core advances are reflected in: attention mechanisms that handle million-token contexts more efficiently; sparse routing across nearly a thousand experts that remains stable in large-scale training; multi-stage pretraining that gradually shapes ultra-long-context capability; and agent-oriented reinforcement learning post-training that moves the model from “answering” to “acting.”

For technical teams, this means that when building or selecting large-scale models, a more systematic evaluation framework is needed: not only parameter scale and benchmark scores but also a deeper understanding of the coupling between attention, sparse experts, long-horizon training, and agent alignment. As hardware and training frameworks continue to evolve, such models are expected to deliver greater practical value in long-document analysis, codebase-level understanding, multimodal retrieval, and complex agentic tasks.