2.8T MoE Weights Open-Sourced: Architecture Innovations and a Local Deployment Guide

An Open-Source Milestone: The 2.8T MoE Weights Are Fully Available

A major event recently shook the AI world: on November 4, a leading cloud vendor made good on its promise by uploading the complete weights of its in-house 2.8T-parameter mixture-of-experts (MoE) large model to a mainstream international open-source community. The move marks a new era for open-weight models in both scale and capability. With approximately 2.78 trillion (about 2.8T) total parameters, the model sets a new open-source record, and it has topped the open-weight category of a well-known agent capability leaderboard, ranking 13th on the global list — clear evidence of its strong tool-calling and task-planning abilities.

For engineering teams, this release is more than a downloadable set of weight files: it is a valuable reference for studying the architecture and training methods of frontier large models. This article examines the release across four dimensions: technical architecture, quantization and storage, deployment practice, and cost considerations.

Architecture Deep Dive: Hybrid Attention Meets Sparse Activation

Layer Design: KDA and DSA Working Together

The model adopts a deliberately hybrid layer design: 93 layers in total, of which 69 are KDA (Kimi Dense Attention, i.e., full attention) layers and 24 are Gated DSA (Gated DeepSeek Sparse Attention) layers. The philosophy here is to use full-attention layers to capture local detail and deep semantic relationships within short sequences, ensuring lossless information flow, while gated sparse attention (DSA) handles long-sequence expansion and sidesteps the quadratic computational complexity that plagues traditional full attention at very long context lengths. The gating mechanism lets the model dynamically decide how much each attention head participates at specific positions, adaptively allocating compute according to task difficulty.

Compared with the previous generation’s full-attention structure, this model achieves a significant efficiency leap in long-text processing. Sparse attention uses chunked processing and hash-based sampling strategies to bring computational complexity down from O(n²) to approximately O(n), so a 1-million-token context window no longer depends on massive brute-force compute. This coarse-to-fine strategy delivers architectural advantages for long-document understanding, complex codebase analysis, and multi-turn deep conversations.

Expert Configuration and Routing

On the expert side, the model uses 896 expert networks: each token dynamically activates 16 experts, plus 2 always-on shared experts. This design follows the same lineage as the mainstream DeepSeek-MoE architecture. The shared experts capture cross-domain general knowledge, while the hundreds of specialized experts each handle specific domains or knowledge clusters. A routing mechanism classifies tokens through a shallow encoder and dispatches inputs precisely to the relevant experts, while a load-balancing loss keeps expert utilization even and prevents some experts from becoming “lazy.”

The direct payoff of this sparse activation: although total parameters reach 2.8T, only about 104B parameters are activated per token during forward propagation. In practice, the model’s inference cost is only slightly higher than a ~100B dense model, yet it carries a knowledge capacity approaching that of a trillion-scale dense model.

Vision Capability: High-Resolution Image-Text Understanding

On the visual side, the model ships with MoonViT-V2, a 401M-parameter vision encoder that supports mixed image-text input. Through multi-scale feature fusion and dynamic token compression, the encoder preserves fine-grained detail even with high-resolution image inputs while keeping the visual token count under control, so vision tokens don’t crowd out the text context window. This design gives the model out-of-the-box capability for document parsing, chart analysis, and visual question answering.

Storage and Quantization: 1.56TB in MXFP Format

Remarkably, this 2.8T-parameter behemoth occupies only about 1.56TB in the open-source repository. That is thanks to the official quantization-aware training strategy: weights are stored uniformly in MXFP4 (4-bit floating point) format, while activations use MXFP8 (8-bit floating point) during forward propagation. This asymmetric quantization scheme is not a crude post-training truncation — quantization noise is simulated during the pre-training stage itself, so the model’s parameters retain high-precision expressiveness even at low bit widths.

MXFP4 preserves the dynamic range of floating-point numbers at extremely low bit widths through shared scaling factors and block-wise exponent encoding. Compared with traditional INT4 integer quantization, MXFP4 reduces precision loss by roughly 30% on very small values and sharply peaked distributions. Because the quantization was baked in during training, the final FP4 weights can be deployed directly after download without any additional calibration, which greatly simplifies the workflow.

Local Deployment: From Framework Selection to Hardware Requirements

Inference Framework Support

The open-source community moved fast, and three mainstream inference frameworks have already adapted the architecture and published official deployment guides:

• vLLM: The veteran throughput-optimized framework. Its PagedAttention and continuous batching maximize GPU utilization, making it well suited to high-concurrency serving scenarios.
• SGLang: Uses a RadixAttention backend to accelerate prefix reuse in multi-turn conversations, cutting inference latency by roughly 40% in agent-style multi-turn tool-calling scenarios.
• TokenSpeed: A newer framework focused on aggressive operator fusion and memory optimization, with competitive performance in single-node multi-GPU setups.

VRAM and Hardware Estimates

Community developers have estimated a minimum viable configuration based on the model’s size: a server with 2.3TB of GPU memory, initially configured with 8 NVIDIA B300-class GPUs. The hardware cost for such a server is estimated in the $550,000–$650,000 range. Note that this is not an official recommendation — it is a community simulation of memory usage under tensor parallelism. In real deployments, FP8 dynamic quantization can further compress activation memory, or CPU offloading can move some shared-expert layers to system memory, though at some cost to inference speed.

A Lighter Alternative: API Access

Given the steep hardware barrier to local deployment, the cloud API is the more realistic path for most application teams. The current API model is kimi-k3.1-pro, fully compatible with both OpenAI and Anthropic interface specifications, which makes migration seamless. For latency-sensitive workloads, a -highspeed variant is available, which the vendor says responds 2–3 times faster than the standard version. The API’s reasoning_effort parameter is adjustable (low/high/max, defaulting to max), giving developers the flexibility to balance reasoning depth against response speed.

One pitfall deserves special attention: state management in multi-turn tool calling. When the model generates reasoning_content (the reasoning process) and tool_calls (tool invocation instructions) in earlier turns, subsequent requests must pass both fields back verbatim in the message history. If you send back only the final conversational content, the model’s context will break and it will fail to chain tool calls correctly. This is the most common integration mistake in agent application development, and developers must strictly align with the request protocol.

Conclusion and Outlook

The full release of this 2.8T-class MoE model is not just a release of technical resources — it is a meaningful push for AI democratization. It gives academia a complete specimen for studying extremely large sparse architectures, and industry a powerful base model ready for direct fine-tuning without further pre-training. Although the hardware barrier to local deployment remains high, the mature cloud API and multi-framework support have lowered the usage threshold as much as possible. For teams aiming to push the boundaries of AI applications, now is an excellent time to build the next generation of intelligent applications on this open ecosystem.