Trillion-Parameter Sparse Multimodal LLMs: Key Architecture and Engineering Techniques
This article provides a technical overview of the architecture and engineering methods behind trillion-parameter sparse multimodal models. It covers sparse MoE design, staged context extension to the 1M-token level, hybrid attention, lower-bounded decay parameterization for linear attention, auxiliary-loss-free load balancing, cross-layer attention residuals, and agentic RL post-training. It also summarizes practical lessons for developers working on LLM training, inference optimization, and agent applications.
In recent years, large language models have evolved rapidly along three main axes: larger scale, longer context, and stronger agentic capabilities. A new class of flagship models recently released by the industry generally adopts a sparse mixture-of-experts (MoE) architecture, with total parameter counts reaching the trillions, around one hundred billion parameters activated per forward pass, native support for visual multimodal input, and context windows extended to the million-token level. This article does not evaluate any specific product. Instead, it examines the core architecture designs and engineering methodologies behind such models from a general technical perspective, providing a reference for engineers working on LLM development, inference optimization, and agent application development.
1. Overall Architecture: Co-Design of Sparse Activation and Ultra-Long Context
The overall scale of such models can be summarized with a few numbers: about 2.8 trillion total parameters, about 104 billion parameters activated per token, and a sparse activation rate of about 3.7%. The point of sparsification is to exchange relatively controllable computation for enormous model capacity. Inference cost is mainly determined by activated parameters, while the model’s capacity to carry knowledge is determined by total parameters.
For context capability, pretraining uses a multi-stage progressive strategy: starting from 8K, then extending to 64K and 256K, and finally advancing to 1M tokens. This curriculum-style length extension has become a mainstream industry practice for a direct reason: the attention computation cost of long-sequence training grows approximately quadratically with length, so training directly from 1M tokens is computationally infeasible. Gradual extension allows the model to first learn language ability on short sequences and then progressively learn long-range dependencies.
The attention module uses a hybrid design: an incremental variant of linear attention, which can be generalized as gated incremental attention, is mixed layer by layer with gated multi-head latent attention, a low-rank KV-cache-compressed attention variant, and a cross-layer attention residual mechanism enables information sharing across layers. The motivation for this hybrid architecture is to balance two goals. Linear attention has O(n) computational complexity on long sequences and can significantly reduce inference overhead for million-level context. Full attention layers retain precise global retrieval capability and avoid the degradation of pure linear attention on needle-in-a-haystack tasks.
On the MoE side, the model uses an extremely sparse configuration with hundreds of routed experts, such as 896, together with an improved SwiGLU-like activation function and an auxiliary-loss-free load balancing mechanism to ensure training stability under large-scale expert parallelism.
2. Core Technique 1: Lower-Bounded Decay Parameterization for Linear Attention
This is the design with the highest value for engineering reuse.
Pain point analysis: When linear attention / Delta Rule-style models use chunked computation, they need to handle the product of cumulative decay factors. When the decay factor is parameterized in an unbounded way, for example through arbitrarily large log-decay produced by softplus, the inverse of the decay matrix can easily underflow or overflow under finite precision such as BF16. The direct consequence is that computation within diagonal tiles cannot be converted into regular matrix multiplication forms and must follow an explicit position-pair path. As a result, GPU Tensor Core matrix multiplication acceleration cannot be used, and operator efficiency drops sharply.
Solution mechanism: Change the log-decay parameterization from an unbounded form to a lower-bounded scaled Sigmoid form. For example, set a fixed decay lower bound, such as -5 in the log domain, so that decay values within each tile are constrained to the closed interval from the lower bound to 1. In this way, the cumulative decay and its reciprocal within any fixed-length diagonal block, such as 16 tokens, are locked within a numerically safe range.
Engineering benefit: Diagonal block computation can be uniformly expressed as standard GEMM, and the entire chunked linear attention operator can achieve order-of-magnitude throughput improvement at the CUDA level, while numerical stability is significantly improved. The code change is minimal. Essentially, it only changes the parameterization of an activation function, but the benefit is large. It is a typical example of achieving disproportionate impact with a small change. Teams building their own linear attention operators can directly reuse this idea:
# Unbounded parameterization (high numerical risk)log_decay = -softplus(raw) # range (-inf, 0]# Lower-bounded parameterization (safe reciprocal within a tile)log_decay = log_floor * sigmoid(raw) # range [log_floor, 0]
3. Core Technique 2: Auxiliary-Loss-Free MoE Load Balancing
A classic challenge in large-scale MoE training is load balancing. If routing is unbalanced, a small number of experts receive excessive routing, causing computation long tails in expert parallelism, uneven memory usage, and even training collapse. The traditional approach introduces an auxiliary load balancing loss, but auxiliary loss conflicts with the main task objective in terms of gradients, damaging model quality and being sensitive to hyperparameters.
The idea behind the new mechanism: Instead of relying on a loss term, apply dynamic correction directly during the routing scoring stage. According to the recent load distribution of each expert, adjust the routing bias by quantile. Experts with higher load automatically have their selection priority reduced, while experts with lower load receive the opposite treatment. Because the correction acts only on routing selection rather than the gradient path, main-task optimization is not disturbed.
Engineering significance: In extremely sparse scenarios with hundreds or even thousands of routed experts, this auxiliary-loss-free balancing strategy significantly reduces tuning cost and improves training throughput and stability for very large-scale expert parallelism. This direction has become an important trend in industry MoE training and is worth prioritizing when evaluating in-house MoE frameworks.
4. Core Technique 3: Cross-Layer Attention Residuals
Deep Transformers have a well-known problem: as layers deepen, representations in the residual stream tend to become homogeneous, and the effective contribution of deep modules diminishes. This is the deep representation bottleneck.
Mechanism design: The output of an attention module is not only written into the residual stream of the current layer. It is also normalized and weighted to share intermediate attention computation results with several subsequent layers, forming a block-level attention residual. Intuitively, attention patterns already computed in shallow layers can be directly reused by deep layers, so deep layers do not need to rebuild them from scratch.
Dual benefits: On one hand, this alleviates deep-layer degradation and improves model quality at the same depth. On the other hand, it reduces repeated computation and cross-device communication, lowering both memory usage and cluster communication overhead during long-context training. For very large model training with pipeline parallelism, the reduction in communication volume is especially critical.
5. Post-Training: Million-Token-Level Agentic Reinforcement Learning
Pretraining addresses whether the model can read long documents. Post-training addresses whether it can use long documents to accomplish tasks. In the post-training stage, such models introduce agentic reinforcement learning under ultra-long context. The model performs policy optimization in task environments that require multi-turn tool calls, long-horizon planning, and intermediate state memory.
There are three main technical challenges. First, rewards are sparse and credit assignment is difficult over ultra-long trajectories. Second, long rollouts create memory and throughput pressure, requiring an architecture that separates inference and training. Third, environment interaction stability requires asynchronous sampling and failure retry mechanisms. Common practices include hierarchical reward design, combining process rewards with outcome rewards, and segmented advantage estimation for long trajectories.
6. Practical Implications for Developers
Overall, such models leave engineering teams with three transferable lessons. First, numerical safety comes first. Any parameterization design should first verify its value-range boundaries under BF16. Second, balancing mechanisms should move away from losses. For discrete decisions such as routing and sampling, online statistical correction should be preferred over gradient constraints for balancing. Third, length extension should be staged. Whether in pretraining or RL, progressive length extension is a key means of controlling cost.
At the implementation level, enterprises can build long-text understanding and agent applications on general cloud infrastructure, such as container platforms, object storage, and distributed training and inference services. They can use elastic compute to optimize the cost of ultra-long-context inference without building a complete underlying infrastructure stack themselves.
Conclusion
The development of trillion-parameter sparse models shows that competition in large models has shifted from simply stacking parameters to fine-grained architecture engineering. Attention numerical design, MoE balancing strategies, cross-layer information flow, and agentic post-training are all redefining the boundary between performance and cost. Understanding these general technical paradigms has more long-term value than chasing a specific model version.
