Trillion-Parameter Open Model Released: Architecture Innovations and Quantized Deployment Practices

Model Overview: From Open Weights to Fulfilled Technical Promises

Recently, a leading AI lab fulfilled its commitment by publicly hosting the complete weights of its new large language model on mainstream open-source platforms. This move satisfies the community’s long-standing demand for auditability and reproducibility, providing a solid foundation for academic research, private deployment, and domain-specific fine-tuning.

The model utilizes a typical Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8 trillion. Thanks to a sparse activation mechanism, it only activates approximately 104 billion parameters per token. This design significantly expands model capacity while effectively controlling computational overhead during inference. The model supports a context window of up to 1 million tokens and natively supports multimodal image-text inputs, covering scenarios from long-document parsing to complex visual question answering.

Architecture Deep Dive: Synergistic Design of Sparse Attention and Expert Routing

Layer Structure: Hierarchical Fusion of Full and Sparse Attention

According to the public model configuration, the model contains 93 Transformer layers. Among them, 69 layers employ KDA (Key-Value Decomposed Attention), while the remaining 24 layers use Gated DSA (Gated DeepSeek Sparse Attention). This design choice reflects strong engineering considerations: introducing sparse attention in deeper layers significantly reduces computational complexity and memory usage when processing ultra-long sequences. Meanwhile, the gating mechanism dynamically determines which token pairs need to participate in attention calculation, balancing information recall with computational efficiency.

Compared to the previous generation’s full utilization of Gated MLA (Multi-head Latent Attention), replacing some layers with Gated DSA highlights a clear focus on optimizing inference costs for long-context scenarios. At the million-token input length, the memory footprint of standard full attention grows quadratically. In contrast, sparse attention reduces complexity to near-linear by combining local windows with global anchor tokens, making it possible to process ultra-long documents within a single node.

Expert Network: Large-Scale Sparse Routing Strategy

The model is configured with 896 Expert modules. During computation, each token dynamically selects 16 experts for forward propagation, supplemented by 2 shared experts designed to capture cross-domain general knowledge. This combination of “fine-grained experts + shared experts” draws on proven practices in recent MoE training: shared experts ensure that foundational language capabilities remain balanced, while sparse routing promotes specialized division of labor among experts.

In terms of parameter distribution, the vast majority of the 2.8 trillion parameters are concentrated in the expert network. During training, this scale imposes extremely high requirements on the coordinated scheduling of data, tensor, and expert parallelism. During inference, it relies on efficient GPU memory management strategies to load expert weights on demand or keep them resident in memory.

Vision Encoder: Lightweight Fusion for Multimodal Inputs

The model is equipped with a vision encoder containing approximately 400 million parameters, responsible for converting image inputs into semantic representations aligned with text tokens. Although the vision module accounts for less than 0.2% of the total parameters, its design quality directly determines the upper limit of image-text understanding tasks. Multimodal alignment typically occurs in two stages: first, contrastive learning pre-trains the vision tower and text projection layer; second, during instruction fine-tuning, the vision tower is frozen or adapted using Low-Rank Adaptation (LoRA) to achieve deep cross-modal semantic fusion.

Quantization Storage and Inference Practices: Dual-Precision Strategy with MXFP4 and MXFP8

Quantization-Aware Training: End-to-End Low-Bit Quantization

Official technical documentation reveals that model weights are stored in MXFP4 format, while activations use MXFP8 format. Crucially, the entire process employs Quantization-Aware Training (QAT) rather than Post-Training Quantization (PTQ). The core difference lies in the fact that QAT simulates the numerical errors of low-bit forward propagation during training, allowing model parameters to adaptively adjust to a distribution robust against quantization noise. This avoids the common accuracy cliff-drop associated with PTQ during high-bit-width compression.

Specifically, the mantissa and exponent bit allocation for MXFP4 (4-bit floating point) is specially designed to preserve dynamic range even within an extremely narrow bit width. Using MXFP8 for activations represents a compromise between precision and speed for compute-intensive operators (such as matrix multiplication). On GPUs supporting FP8 acceleration, this dual-precision strategy can achieve nearly a 2x throughput improvement.

Storage Footprint and Parameter Statistics

Public repository information indicates that the entire repository occupies about 1.56TB of space, while the total parameter count based on the safetensors format is approximately 2.78 trillion. This minor discrepancy stems from redundant metadata storage and serialization overhead. For teams planning local deployment, the 1.56TB download and storage cost is a hard resource constraint that must be evaluated in advance.

Inference Framework Compatibility

Currently, three mainstream open-source inference engines have provided deployment recipes for this model, including the widely used vLLM, SGLang, and the emerging TokenSpeed. This means developers can quickly spin up services using the high-level APIs of these frameworks without manually writing underlying CUDA kernels or complex parallel scheduling logic.

In vLLM, enabling continuous batching and PagedAttention can significantly improve GPU utilization in long-context scenarios. SGLang automatically caches prefix KV states via RadixAttention, making it particularly suitable for repeated prompt processing in multi-turn conversations or Agent tasks. TokenSpeed leans towards extreme low-latency optimization, ideal for interactive scenarios with stringent requirements for Time-To-First-Token (TTFT).

Local Deployment and Hardware Threshold Assessment

VRAM and Compute Requirements Estimation

Although weights are stored in MXFP4, they still need to be decompressed to compute precision (typically FP16/BF16 or FP8) during inference to execute matrix operations. Calculated at BF16 precision, the 2.8 trillion parameters require approximately 5.6TB of VRAM, far exceeding the capacity of a single GPU. Therefore, multi-node or multi-GPU parallelism is the only realistic path.

Community members have provided rough estimates based on current flagship GPU configurations: using 8 professional compute cards with 288GB of VRAM each (e.g., a certain B300 model) yields a total of 2.3TB of VRAM, which is still insufficient to fully resident the BF16 weights. In this scenario, memory optimization technologies such as KV Cache offloading and on-demand expert weight loading are required. The hardware procurement cost for this configuration is estimated to be between $550,000 and $650,000, not including the costs of high-speed interconnect networks (like InfiniBand) and storage arrays.

Memory and I/O Bottlenecks

It is worth noting that even if VRAM is sufficient, the memory bandwidth required to read the 2.8T parameters during each forward propagation is astonishing. Assuming that generating one token requires reading all activated expert weights, activating 104B parameters per token means a single forward pass needs to read about 208GB (BF16) of data from HBM. This requires GPU HBM bandwidth to reach at least the 8TB/s level to keep single-token generation latency in the tens of milliseconds. Therefore, the inference performance of MoE models depends not only on compute power but is also heavily constrained by memory bandwidth and the optimization of data loading paths.

Real-World Agent Task Performance and Evaluation Signals

Evaluation Methodology Based on Real Conversations

Unlike traditional static benchmarks (such as MMLU or HumanEval), this model’s performance evaluation is based on an online platform named Arena, which collects logs of real users engaging in multi-turn Agent interactions with various models. The evaluation system extracts signals across four dimensions: whether the user explicitly marks the task as complete, the user’s positive or negative sentiment expression towards the model’s reply, whether the model effectively adjusts its strategy after user correction, and the success rate of background Shell command execution.

Based on these signals, the platform calculates a “net uplift” score to quantify the value increment the model provides to users in real-world tasks. The advanced nature of this methodology lies in its ability to bypass the saturation effects of static datasets, dynamically reflecting the model’s generalization capabilities in complex, long-tail, multi-step tasks.

Key Evaluation Data Interpretation

On the day of the weight release, the platform published evaluation results based on over 17,500 real long-context sessions. The new model achieved a net uplift of +5.1% under default parameter settings, ranking first among open-weight models and thirteenth on the overall leaderboard including all models. It topped the open-weight model charts in three sub-categories: Work, Code, and Chat. In contrast, the previous generation model only achieved a net uplift of +0.91%, ranking sixth.

Itemized data shows that the model performs exceptionally well in Bash command-line task reliability, with a net uplift of +7.9%, whereas the previous generation recorded a negative value of -4.5% in this category. This indicates a quantum leap in the model’s tool invocation and instruction-following capabilities. The engineering significance of this improvement is clear: when AI Agents execute complex DevOps, data processing, or automated scripting tasks, their ability to accurately generate Shell commands and recover from errors is critical to task success.

API Usage and Engineering Implementation Recommendations

Invocation Interface and Parameter Configuration

For teams without their own compute resources, an official cloud API compatible with mainstream API styles is available. The model service is named kimi-k3.1-pro, and a high-speed version is also provided. The official documentation claims its processing speed is 2 to 3 times faster than the standard version, making it suitable for high-throughput batch processing scenarios.

Regarding inference parameters, the system supports the reasoning_effort parameter with options low, high, and max (default is max). This parameter controls the depth of the Chain-of-Thought (CoT) reasoning the model performs before answering. For scenarios requiring quick responses, setting it to low is recommended; for solving complex logic problems or multi-step mathematical reasoning, the max value can significantly improve answer accuracy. However, it is important to note that deep CoT will linearly increase the number of output tokens, thereby affecting overall latency and cost.

Multi-Turn Tool Invocation State Management

In multi-turn tool invocation scenarios, developers are explicitly required to pass the reasoning_content and tool_calls parameters from the previous round back into the context. This design maintains the coherence of the reasoning chain, preventing the model from “losing its way” in the next dialogue turn due to missing intermediate reasoning processes. In engineering implementation, developers must completely record the original request and response structure of each interaction in the session state management module, rather than only retaining the streamlined, user-visible text.

Conclusion and Future Outlook

The open-sourcing of this 2.8T-parameter MoE model marks a new phase in the large model open-source ecosystem: model scale has moved from the hundred-billion level to the trillion level, quantization training has transitioned from experimental exploration to production-grade application, and evaluation systems have shifted from static benchmarks to dynamic real-world scenarios. For technical teams, rationally evaluating their own hardware constraints and task types to choose an appropriate deployment strategy (local weights + inference framework vs. cloud API) will determine whether they can maximize the productivity of this industrial-grade model. In the future, as more efficient sparse attention variants and low-bit compute frameworks become widespread, the local deployment threshold for trillion-parameter models is expected to decrease further, fostering more innovative Agent applications.