From 2.8 Trillion Parameters to Efficient Inference: Key Design Choices in a Next-Generation MoE Multimodal Model

1. Introduction: What Does the Next Phase of the LLM Race Come Down To?

As large models enter the trillion-parameter era, the industry’s focus has shifted from raw capability emergence to a dual challenge: efficiency and stability. A MoE multimodal model released in July 2026 (hereafter “the model”) drew widespread attention with its architectural choices. The model has 2.8 trillion total parameters, yet thanks to its MoE (Mixture-of-Experts) mechanism, only 104 billion parameters are activated per inference. It natively supports visual multimodality and a 1-million-token context window, and it demonstrates top-tier results on open-source benchmarks.

Parameter counts, however, are only the tip of the iceberg. The real technical depth lies in questions such as: How do you maintain training stability with a massive expert network of 896 routed experts? How do you make linear attention both fast and accurate over ultra-long contexts? How do you enable genuine long-horizon agentic reasoning through post-training? Drawing on publicly available technical reports, this article dissects the model’s core innovations and explores their implications for engineering large models into production.

2. Architecture Overview: Balancing Extreme Sparsity with Full-Modality Coverage

The model’s base architecture follows a demanding path: a standard dense Transformer backbone deeply integrated with multiple improved attention mechanisms and MoE layers.

• Scale: 2.8T total parameters, only 104B activated, a sparsity ratio of roughly 26.9:1. The model’s theoretical knowledge capacity is enormous, while the compute cost of a single forward pass is held to the level of a mid-sized model.
• Modality support: Native visual and text multimodal input. Features from the vision encoder are concatenated directly with the text token sequence and participate jointly in all subsequent attention computation and expert routing, rather than relying on a simple image-captioning approach.
• Context window: Through multi-stage pretraining, the context length is extended progressively from 8K to 64K, then 256K, and finally 1M tokens. This requires extreme optimization across position encoding, attention compute complexity, and memory management.
• Training pipeline: A three-stage process of pretraining, long-context extension, and post-training. The post-training stage introduces million-token-scale Agentic RL (reinforcement learning) specifically to strengthen long-horizon planning and tool-use capabilities.

3. Core Innovation #1: KDA Lower-Bounded Decay Parameterization — Making Linear Attention Fly on Tensor Cores

3.1 The Pain Point: Numerical Pitfalls of Linear Attention in Chunked Computation

Linear attention mechanisms (including various Linear Attention variants and derivatives) can theoretically reduce sequence-length complexity from O(N²) to O(N), which is the foundation for ultra-long contexts. In practice, however, fully utilizing GPU Tensor Core matrix-multiply throughput usually requires splitting the sequence into fixed-size chunks (e.g., 16 tokens). Each chunk contains one diagonal block (causal tile) and several off-diagonal blocks (non-causal tiles).

To preserve causality, the diagonal block computation must correct the historical state using the reciprocal of a cumulative decay factor Γ. Traditional approaches such as Delta Net parameterize the log-decay with a negative Softplus function, which makes Γ prone to underflow (toward zero) or overflow (toward infinity) under BF16 precision. Once numerical stability is lost, the compiler is forced to fall back to an explicit position-pair path for the diagonal block—effectively degrading to a slow element-wise loop instead of a unified GEMM operation. Tensor Core utilization drops sharply, and long-context throughput becomes unattainable.

3.2 The Solution: A Lower Bound Guaranteed by Scaled Sigmoid

KDA (Kimi Delta Attention) proposes an elegant parameterization fix: replace the unbounded “negative Softplus” log-decay computation with a bounded “scaled Sigmoid.” Mathematically, the log of the decay factor is constrained to:

g_theta = g_min · Sigmoid(e_theta^T z_theta)

where g_min is a fixed negative value (e.g., -5). This seemingly minor change yields qualitative gains:

• Numerical stability: Because Sigmoid output is strictly between (0, 1), the log-decay is locked within (-5, 0), ensuring the reciprocal of Γ always stays within the safe range of BF16 with no overflow.
• Operator unification: With values no longer diverging, compiler and kernel developers can remove all special-case branches for diagonal blocks. All causal tiles—diagonal or not—can call the dense GEMM kernel directly, enabling a unified pipelined computation. Measurements show that this change improves the efficiency of the linear attention operator implemented on the CUTLASS library by several times.

3.3 Engineering Benefit: Enabling 1M Extrapolation Under NoPE

More profoundly, KDA parameterization is naturally compatible with NoPE (no positional encoding). Because the decay factor itself carries relative-distance information, the model no longer needs explicit positional encoding to perceive token order. This greatly reduces the distribution-shift risk when extrapolating to ultra-long contexts, making it possible to extrapolate directly from an 8K pretraining length to a 1M context.

4. Core Innovation #2: Quantile Balancing — An Auxiliary-Loss-Free Stabilizer for Large-Scale MoE

4.1 The Challenge: Hot Spots and Dead Spots in Sparse Expert Networks

In MoE architectures, the router assigns tokens to the top-K experts. With 896 experts, a vicious cycle easily emerges: some experts are selected frequently due to initial bias or data preference (hot experts), overloading their compute, while others sit idle for long periods (dead spots), contributing nothing despite their parameters. The conventional fix is an auxiliary loss to enforce load balancing, but this interferes with the main task’s gradient signal, and tuning its hyperparameters is extremely difficult—at the scale of nearly a thousand experts, it can even cause training divergence.

4.2 The Mechanism: Dynamic Bias Adjustment Based on Quantiles

Quantile Balancing (QB) offers an auxiliary-loss-free alternative. Its core operations are as follows:

• On each forward pass, the router generates a Softmax score distribution over all experts for each token.
• For each expert, the system continuously tracks the historical distribution of routing scores (logits) it receives.
• When computing the actual routing gate value, the raw score is no longer used directly; instead, a quantile of that expert’s score distribution (e.g., P80/P90) is subtracted.

The philosophy behind this dynamic bias adjustment is that it imposes no constraint at the loss level but instead corrects routing bias directly inside the inference computation graph. If an expert’s scores are generally high (a hot expert), the quantile subtracted is correspondingly larger, reducing its probability of being “absolutely selected”; conversely, for an expert whose scores are persistently low, the calibrated score rises relatively. The whole process is implemented purely with tensor operations, with near-zero compute overhead and no manual tuning of auxiliary-loss weights, allowing model capacity to be fully unleashed.

5. Core Innovation #3: Block Attention Residuals — Rebuilding the Information Highway in Deep Transformers

5.1 Representation Degradation in Deep Networks

As Transformer depth increases (the model has hundreds of layers), deep networks often suffer from representation degradation: low-level detail is gradually “smoothed out” through layer upon layer of nonlinear transformation, making it hard for deeper modules to learn new, complementary features and causing performance to plateau. Standard residual connections alleviate vanishing gradients, but information-transfer efficiency still decays with distance.

5.2 The Approach: Attention Residual Skips at the Block Level

The model introduces Block-level Attention Residuals (AttnRes). The network is divided into several large blocks, each containing multiple Transformer layers. Within each block, standard layer-by-layer connections are maintained; between blocks, a “highway” connection adds the attention representation output by the previous block directly onto the output of the next block.

This design yields three direct benefits:

• A memory anchor: Deep modules can directly access low-level visual or semantic features extracted by shallow layers, avoiding forced forgetting of original information.
• Faster convergence: Gradients can flow directly back to shallow layers through the skip connection, greatly alleviating gradient discontinuities in deep networks and accelerating training convergence.
• Lower training memory and communication overhead: In long-context training, this residual path reduces the volume of data communicated across devices. Because information transfer no longer depends entirely on sequential layer-by-layer propagation, the communication topology for tensor parallelism and sequence parallelism can be simplified, significantly improving scaling efficiency on large GPU clusters.

6. Training Strategy: A Staged Climb from 8K to 1M and Agentic RL

6.1 Multi-Stage Pretraining: Steady Context Extension

The model’s ultra-long context capability is not achieved overnight. Training follows a strict progressive strategy: first, large-scale corpus pretraining at 8K context to establish foundational knowledge; then, gradually increasing the RoPE base frequency or adjusting decay parameters to move to 64K; then extending to 256K; and finally cleaning data and extending to 1M. Each stage adjusts the data mix, emphasizing long documents, multi-document collections, and repository-level code corpora to stimulate the model’s ability to capture long-range dependencies.

6.2 Agentic RL in Post-Training: From Conversational Model to Task Executor

To improve practical utility as an agent, the post-training stage introduces Agentic RL. This stage trains not on simple question-answer pairs but by having the model autonomously complete multi-step tasks in simulated environments. For example, given a complex goal, the model must call external tools (such as search engines and code interpreters), read results, reflect on errors, and adjust strategy until the goal is achieved.

The RL reward signal comes not only from final task success rate but also from assessments of intermediate-step reasonableness. Across reasoning trajectories spanning up to a million tokens, the model continuously optimizes its reasoning-path planning through policy-gradient methods. This gives the model a coherence in behavior on tasks requiring long logical chains and memorization of key historical facts that traditional SFT (supervised fine-tuning) models cannot match.

7. Conclusion and Implications: The Technical Deep Water of Engineering Large Models

Looking across the model’s three core innovations, the current competitive high ground for large models becomes clear:

1. Extreme operator optimization: Finding the balance between numerical precision and hardware instruction sets—adapting formulas to the hardware—is often more cost-effective than simply adding compute.
2. A priori design for training stability: Working inside the “computational flow” rather than at the loss-function level (as with the QB mechanism) is a more robust path to preventing training collapse in very large sparse models.
3. From model R&D to system co-design: Attention residuals and progressive context extension are, at their core, system-engineering considerations aimed at achieving more efficient parallel communication and utilization on tens of thousands of GPUs.

For developers and AI engineers, replicating the training of a trillion-scale model is not feasible—but the engineering philosophy behind it—“do not chase complex structures blindly; instead, use fine-grained parameterization control to reduce operator implementation complexity”—is arguably the most valuable mindset to study in AI today. With compute costs high, this kind of software-hardware co-design thinking may well be the key engine on the road to general artificial intelligence.