Architecture Deep Dive: Trillion-Parameter Multimodal LLMs from Sparse Training to Long-Horizon Reasoning

1. Key Capabilities at a Glance: Design Trade-offs at Trillion-Parameter Scale

Kimi K3, a multimodal model released in July 2026, has a publicly disclosed parameter scale of 2.8 trillion total parameters and 104 billion activated parameters. These numbers alone reveal a core design orientation: at an extremely large total parameter scale, drastically reducing the floating-point operations per inference step through extremely sparse expert activation, while balancing capacity, memory footprint, and response latency.

From an engineering perspective, the large gap between total and activated parameters is not about chasing “parameter numbers” for their own sake, but about solving three practical problems:

1. The tension between knowledge capacity and compute cost: A larger total parameter count can carry broader world knowledge and multimodal mapping capabilities, but if all parameters are activated on every inference, latency and energy consumption rise sharply.
2. Unified processing of multimodal inputs: Visual and text inputs need to be aligned in a shared representation space. The model must have native visual encoding capability rather than simply attaching vision tasks as an external module to a text backbone.
3. Usability of ultra-long context: A 1-million-token context is not simply a matter of increasing sequence length. It requires coordinated adaptation of attention computation, positional encoding, state caching, and training strategy.

Kimi K3 can therefore be viewed as a “sparse multimodal model designed for deployability”: it increases model capacity through MoE, alleviates long-sequence compute pressure through a low-overhead attention structure, and uses dedicated load balancing and training stability mechanisms to prevent sparse experts from degrading or becoming imbalanced during ultra-large-scale training.

2. Attention Mechanism: Fusing Incremental Attention with Gated Latent Attention

In long-context scenarios, the compute and cache overhead of traditional self-attention grows rapidly with sequence length. For inputs at the 1-million-token level, the memory footprint of the key-value cache alone — even with standard attention caching — can exceed what a single GPU or even a multi-GPU cluster can bear.

Kimi K3’s attention module combines “Kimi Delta Attention (KDA)” with “Gated Multi-head Latent Attention (Gated MLA),” essentially establishing a compromise between two design goals:

2.1 Incremental Attention for Long-Sequence State Updates

The core idea of KDA is to treat the attention distribution as a state that can be incrementally updated, rather than recomputing the full attention matrix from scratch at every step. For previously processed historical segments, the model can store a compressed attention state; when a new token arrives, it only needs to perform an incremental update on the current state.

This approach yields two direct benefits:

• More stable inference latency: Because historical information does not need to be repeatedly involved in full recomputation, the per-step cost does not accumulate indefinitely with total length when processing long documents or long conversations.
• Compressible cache: After abstracting historical information into a state representation, a substantial proportion of fine-grained key-value pairs can be discarded, significantly reducing KV cache occupancy.

To make this intuitive, the logic of incremental attention can be represented with highly abstract pseudocode:

state = compress_initial_context(tokens[:window])
for new_chunk in stream_chunks(tokens[window:]):
    state, output = incremental_attention(state, new_chunk)
    append_output(output)

Note that “incremental” here does not mean simply discarding historical tokens. Instead, necessary information that influences future attention is retained through a learnable state representation. This is especially important for document QA, repository-level code understanding, and long-running multi-turn conversations.

2.2 Gated Latent Attention Reduces Multi-Head Cache Pressure

The core idea of Gated MLA is to project each head’s key and value into a shared low-dimensional latent space, then modulate the information flow of different heads through a gating function. Unlike explicitly storing the full key-value tensors for every attention head, latent attention only needs to cache low-dimensional latent vectors and reconstruct the corresponding key-value representations when needed.

This can be compared to “lossy but controllable attention compression”:

latent_k, latent_v, gate = project_with_shared_space(x)
k = reconstruct_k(latent_k)
v = reconstruct_v(latent_v)
gated_v = v * gate

The gating term is essentially the model learning “which latent dimensions are more useful for the current layer and current head.” In multi-head attention, different heads focus on quite different information patterns, and a uniformly shared latent space may sacrifice some expressive capacity. The gating mechanism compensates for this loss.

Compared with purely low-rank approximation latent attention approaches, Gated MLA achieves a better balance between cache efficiency and expressive power. Combined with KDA, the model can compress both historical states and multi-head key-value representations, thereby supporting context processing at the 1-million-token level.

2.3 Attention Residuals Enable Cross-Layer Information Sharing

Kimi K3 also introduces “Attention Residuals (AttnRes)” between attention modules. Deep Transformers typically ensure gradient flow through residual connections, while AttnRes goes further by enabling explicit combination between attention outputs from different layers.

A standard Transformer residual connection can be written as:

h_{l+1} = h_l + attn_l(h_l) + ffn_l(h_l + attn_l(h_l))

With attention residuals, the attention output of a given layer is not restricted to flowing only into the next layer — it can also be passed as a cross-layer signal to deeper attention layers:

attn_res = combine(attn_l, attn_{l-1}, ..., attn_{l-k})
h_{l+1} = h_l + attn_res + ffn_l(h_l + attn_res)

The value of this approach: local patterns captured by shallow attention — such as syntactic relationships between adjacent tokens, code indentation structure, or local image textures — do not have to rely entirely on layer-by-layer propagation to be directly utilized by deeper attention layers. For ultra-long documents that require understanding both fine details and global semantics, this cross-layer sharing can alleviate deep-layer information loss to a certain degree.

3. MoE Sparse Experts and Training Stability

On the MoE side, Kimi K3 is equipped with 896 routed experts and adopts the Stable LatentMoE architecture, along with the SiTU-GLU activation function and the Quantile Balancing (QB) mechanism. These designs do not exist in isolation; they jointly serve a key objective: keeping an extremely sparse model with nearly a thousand experts stable and efficient during large-scale training.

3.1 Stable LatentMoE: Expert Routing for Extreme Sparsity

Traditional MoE models typically have 8–64 experts, whereas 896 routed experts means only a very small number of experts are activated per inference. This high degree of sparsity further reduces per-step compute, but it also brings several engineering challenges:

• Routing fluctuation: If routing scores are unstable, the same or similar inputs may be directed to different experts, increasing training gradient noise.
• Coexistence of under-trained and over-trained experts: Popular experts are updated frequently, while cold experts may not receive effective gradients for extended periods.
• Load imbalance: A small number of experts handle too many tokens while the rest are underutilized, reducing effective compute parallelism.

The core idea of Stable LatentMoE is not to perform hard routing directly across all 896 experts for every token, but to first perform stable matching in a low-dimensional latent space and then map to specific experts. This partially decouples the process of “choosing who processes the current token” from high-noise routing scores, enhancing the stability of routing decisions.

In actual training, this design can reduce the optimization difficulty caused by frequent routing switches. When a token’s representation maintains a consistent tendency across training steps, its gradients flow more stably to the corresponding expert, alleviating the “routing oscillation” problem common in sparse MoE.

3.2 SiTU-GLU: An Activation Function Optimized for Sparse Experts

The GLU family of activation functions has been widely adopted in large models, typically in the form:

GLU(x, W, V) = (xW) ⊙ sigmoid(xV)

or with other gating functions replacing sigmoid. The SiTU-GLU used by Kimi K3 can be understood as adjusting the nonlinear form of the GLU gating branch to make it better suited to large-scale sparse expert networks.

In extremely sparse MoE, the same expert may process tokens from different distributions and semantic regions. If the activation function saturates too easily, some experts may exhibit excessively small output magnitudes or vanishing gradients during training. If the activation function lacks sufficient nonlinear boundaries, it weakens the differentiation capability between experts.

SiTU-GLU more precisely matches the gating function with the output scale of the main projection branch, preserving the advantages of GLU gating while reducing activation saturation and gradient instability in sparse expert training. For developers, the implementation can still maintain a combination of linear layers and element-wise multiplication:

output = linear_gate_situ(x)
result = linear_value(x) ⊙ output

This structure is convenient to adapt within existing Transformer training frameworks while providing better numerical stability for large-scale MoE training.

3.3 Quantile Balancing (QB): Load Balancing Without Fixed Capacity

Traditional MoE often limits the number of tokens each expert processes through a capacity factor, then encourages balanced routing distributions through auxiliary losses. However, in scenarios with extremely large expert counts and mixed long sequences, fixed capacity strategies often introduce unnecessary truncation or routing failures.

The Quantile Balancing (QB) mechanism re-examines the load balancing problem from a statistical perspective. It does not require every expert to process exactly the same number of tokens in every training batch. Instead, it dynamically adjusts expert load based on quantiles of routing scores, allowing a certain degree of imbalance under long-tail distributions while preventing extreme skew.

It can be viewed as a “soft constraint”:

For each expert e:
    Count the proportion of tokens routed to e in the current batch
    Compute an adjustment signal based on quantile differences
    Inject the adjustment signal into routing scores or expert outputs

Compared with hard auxiliary losses, QB is better suited to the extreme sparsity scenario of K3’s 896 experts. On one hand, it reduces expert capability homogenization caused by forced balanced training. On the other, it can continuously correct the overload of a minority of experts during training, preventing resource hotspots from becoming system bottlenecks.

4. Multi-Stage Pretraining: Gradual Expansion from Short Sequences to a Million Tokens

Kimi K3 adopts a four-stage pretraining strategy: 8K, 64K, 256K, and 1M. This design directly addresses the problems of “unstable length extension and high training cost” in ultra-long context training.

In the 8K stage, the model first learns basic syntax, common knowledge structures, and short-range dependencies. In the 64K stage, it begins processing longer documents and learns longer-distance coreference and context integration. In the 256K stage, the model needs to adapt to repository-level tasks, long report summarization, and multi-document QA. Finally, in the 1M stage, the model handles inputs approaching one million tokens, such as complete software repositories, long meeting transcripts, or multimodal long video frame sequences.

A clear benefit of gradual expansion is training stability. Training directly with full-length sequences causes positional encoding, attention states, and memory allocation strategies to undergo drastic changes simultaneously, which can easily trigger loss spikes in early training. Staged expansion allows learning rate, batch size, and sequence length to be adjusted gradually, making the optimization trajectory smoother.

At the same time, the data mix across length stages is also critical. In real-world applications, the vast majority of tasks fall in the short-to-medium length range, with only a small fraction requiring 256K or 1M. Therefore, pretraining data is not simply evenly divided across the four length intervals; shorter sequences dominate, with longer samples gradually mixed in. This covers long-tail needs without sacrificing language modeling quality at common lengths.

Multimodal capability also intervenes naturally during pretraining. Native visual multimodality means visual signals are not forcibly connected through post-hoc alignment, but are jointly trained with text during pretraining. Image patch embeddings output by the visual encoder share the same representation space with text tokens and participate together in routing and attention computation. This approach helps the model retain finer-grained visual information in cross-modal tasks, such as chart interpretation, screenshot QA, and UI understanding in automation.

5. Agentic RL Post-Training: Toward Long-Horizon Reasoning and Tool Use

In the post-training stage, Kimi K3 adds million-token-level Agentic RL. This is an important differentiator from ordinary instruction-tuned models.

Ordinary post-training often focuses on single-turn QA quality, output formatting, and instruction-following ability, whereas Agentic RL places more emphasis on the model’s planning, verification, error correction, and tool-calling capabilities in long-horizon tasks. Its training samples may include the following structure:

Goal: Complete a multi-step task
Environment: Executable actions
Process: Model observes state → outputs action → receives result → continues reasoning
Final reward: Depends on task completion, number of steps, and correctness of results

In this setting, the model learns not just “how to answer an isolated question” but “how to adjust strategy during a continuous interaction process.” For example, the model may need to retrieve information first, discover that the retrieval results are insufficient, then rewrite the query; or generate an initial plan, execute it, and correct based on errors; or break a long task into multiple subtasks and verify whether conditions are met at each node.

This training is highly coupled with million-token context capability. Intermediate states, historical actions, and results during long-horizon reasoning need to be continuously retained; otherwise, the model can easily lose the context chain. The efficient state caching brought by KDA and Gated MLA allows the model to maintain longer interaction trajectories under limited memory, while Agentic RL further trains the model to use this long-context information to make more reasonable decisions.

From a capability perspective, the model after Agentic RL has advantages in the following scenarios:

• Multi-turn tool calling and error recovery
• Step-by-step modification and testing for complex code tasks
• Retrieval-augmented generation driven by long documents
• Cross-modal information integration and decision planning

6. Implications and Practical Considerations for Developers

For development teams that do not have the resources to train trillion-parameter models, Kimi K3’s architecture does not need to be reproduced exactly, but several of its design principles offer strong reference value.

6.1 Use Sparse Experts to Improve Capacity and Cost Efficiency

If a business needs to significantly increase model knowledge capacity while unable to accept the inference cost of a dense model, the MoE route is worth considering. However, more experts is not always better; the number must match routing stability, load balancing, and data scale. Kimi K3’s approach shows that when expert count reaches the hundreds or near-thousand level, mechanisms like Stable LatentMoE and Quantile Balancing become critical, or the model may fall into training instability or insufficient expert utilization.

6.2 Long-Context Optimization Focuses on Caching and State Management

Achieving long context is not just about whether the model can “see” more tokens — it is about whether the system can support long-sequence serving with acceptable memory and latency. For teams building their own models, consider rewriting attention output from full key-value caching to latent representation caching and designing incremental update logic. Even when adapting existing open-source frameworks, prioritizing compression and merging of historical states can yield substantial gains.

In engineering implementation, long documents can be split into chunks with a compact state maintained for each:

current_state = None
for chunk in split_document(document):
    current_state, chunk_output = attention_with_state(chunk, current_state)
    store_relevant_chunk_output(chunk_output)

This design not only reduces memory footprint but also allows online inference services to maintain predictable resource consumption when facing long inputs.

6.3 Multi-Stage Length Training Is More Practical Than Single-Stage Full-Length Training

For enterprises looking to enhance long-context capability, directly continuing pretraining with full-length data carries high risk. A more reasonable path is to set up multiple length stages and determine the data proportion for each stage based on business distribution. At each stage, positional encoding or attention state mechanisms should also be adjusted, rather than only increasing sequence length. Gradual expansion can significantly reduce the risk of training instability and degradation of short-sequence capability.

6.4 Agentic Capabilities Need to Be Built in Sync with Long-Context Capabilities

Tool calling, planning, and error correction are not capabilities that can be reliably achieved through prompting alone, especially when tasks span multiple steps, multiple tools, and multiple state changes. To achieve reliable agent behavior, the model needs sufficiently long effective context memory, and the decision-making process needs continuous optimization through reinforcement learning with environmental feedback or similar training methods.

For application developers, a practical starting point is to begin with fixed-workflow Agent pipelines, managing tool calls and state records in a structured way. As model capabilities improve, they can then move toward more open autonomous planning. Regardless of the approach, logging, trajectory replay, and observability are essential foundations for deploying long-horizon Agent capabilities.

7. Summary

Kimi K3 demonstrates several frontier practices in architecture and training methods for trillion-parameter multimodal models. It balances ultra-large capacity and ultra-long context through a sparse MoE expert structure and low-overhead attention modules. It addresses the training stability challenges brought by nearly a thousand experts through Stable LatentMoE, SiTU-GLU, and Quantile Balancing. It mitigates length-extension instability through staged pretraining from 8K to 1M. And it transforms long-context capability into practical long-horizon reasoning and tool-calling ability through million-token-level Agentic RL.

This technical路线 also has reference value for model design aimed at enterprise deployment. Even without pursuing trillion-parameter scale, core design principles can be adopted: use sparsity to control cost, use state compression to support long sequences, use gradual training to ensure stability, and use environmental interaction to improve agent decision-making. What truly determines the engineering value of a large model is often not just the parameter scale itself, but whether these underlying designs can be correctly implemented in a deployable, maintainable, and scalable system.