Open-Source 2.8T-Parameter MoE Model: Weights Release, Architecture, and Agent Capabilities Explained

Model Overview: From Parameter Scale to Architectural Innovation

An AI lab has fulfilled its commitment to open-source the complete weights of its flagship large language model to the community. This move attracted widespread attention in the technical community, primarily due to its unprecedented parameter scale and architectural design. The model has a total parameter count of 2.8 trillion (2.78T) and uses a typical Mixture-of-Experts (MoE) architecture. During inference, only 104 billion parameters (104B) are activated per token — a sparse activation design intended to balance model capacity with inference cost.

In terms of model structure, the total number of layers is 93, of which 69 layers use KDA (Key-Value Decomposed Attention) and the remaining 24 layers use Gated DSA (Gated DeepSeek Sparse Attention). Notably, compared with the previous generation, the research team replaced Gated MLA (Multi-head Latent Attention) in the full-attention layers with a sparse attention mechanism. This change primarily serves long-context scenarios, significantly reducing computational overhead while maintaining model capability. For the expert layer configuration, the model contains 896 experts in total, dynamically selecting 16 experts per inference pass, plus 2 additional shared experts to ensure stability of foundational capabilities. On the multimodal side, the model includes a built-in vision encoder with 401 million parameters, supporting mixed image-text input.

Weight Storage and Quantization Strategy: MXFP Format in Practice

The storage format of weight files directly determines model distribution efficiency and deployment cost. The model stores weights in MXFP4 (Microscaling Floating Point 4-bit) format, while using MXFP8 (8-bit) precision for activation computation. The official team states that this approach benefits from sufficient pretraining with quantization-aware training. In terms of actual data, the Hugging Face repository occupies approximately 1.56TB, and the parameter count reported in safetensors format is approximately 2.78 trillion.

The core idea of quantization-aware training is to introduce simulated quantization errors during the training phase, so that model parameters remain highly robust under low-precision representation. The MXFP format is essentially a representation method that combines a floating-point scaling factor with a low-bit-width mantissa. Compared with traditional INT8 or FP8, MXFP achieves a better balance between dynamic range and precision. Using MXFP4 storage means each weight occupies only 4 bits. With a total parameter count of 2.78 trillion, the total storage is compressed to approximately 1.56TB, providing a practically feasible distribution solution for organizations with multi-GPU clusters.

However, the hardware barrier for deploying this model remains significant. According to estimates from third-party technical enthusiasts, one feasible local inference configuration requires 8 NVIDIA B300 GPUs, with a total VRAM capacity of approximately 2.3TB and a hardware procurement cost of roughly $550,000 to $650,000. This is clearly not a solution for individual developers, but rather a high-end configuration intended for enterprise applications or research institutions. It is worth noting that Moonshot AI has not officially published a recommended hardware list; the above figures are community estimates only.

Inference Deployment in Practice: Framework Adaptation and Optimization

Regarding deployment concerns from the community, three mainstream inference frameworks have already completed adaptation for this model: vLLM, SGLang, and TokenSpeed. These three frameworks each represent different optimization approaches: vLLM emphasizes high throughput and continuous batching; SGLang focuses on structured generation and an efficient RadixAttention mechanism; TokenSpeed prioritizes reducing time-to-first-token and optimizing VRAM reuse.

From an engineering practice perspective, deploying a model of this scale requires attention to the following key configurations:

1. VRAM Planning: Since MXFP4 weight storage is used, the model occupies approximately 1.56TB of VRAM after loading. Tensor parallelism and pipeline parallelism combinations must be planned according to the actual number of GPUs.
2. KV Cache Strategy: For a 1-million-token context window, KV Cache VRAM usage cannot be ignored. The introduction of the sparse attention mechanism significantly alleviates VRAM pressure under long sequences, but enabling VRAM management mechanisms such as PagedAttention is still recommended in actual deployment.
3. Batching Strategy: Under the MoE architecture, different experts may experience load imbalance. Attention must be paid to Expert Parallelism settings to avoid some GPUs becoming bottlenecks.

The following is a simplified deployment example based on vLLM:

from vllm import LLM, SamplingParams
# Initialize the model and set tensor parallelism degree
llm = LLM(
    model="path/to/model",
    tensor_parallel_size=8,  # Adjust based on the number of GPUs
    max_model_len=1000000,
    dtype="float8",  # Activations use MXFP8
    quantization="mxfp4",  # Weights use MXFP4 format
)
# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=4096,
)
# Run inference
outputs = llm.generate("Your prompt here", sampling_params)
print(outputs[0].outputs[0].text)

Agent Capability Evaluation: Performance in Real Conversations

Model reasoning evaluation is no longer limited to static benchmarks but has shifted toward real-world agent task performance. A well-known Arena platform conducted quantitative evaluation of the model across multiple dimensions based on more than 17,500 real long-horizon conversation sessions. Evaluation signals include: whether the user explicitly confirms task success, praise and complaint text about the model, the model’s response quality after user corrections, and the success rate of shell command execution.

The latest data shows that the model achieves a Net Lift of +5.1% on Arena, ranking first among open-weight models and 13th on the overall leaderboard. By comparison, the previous generation model achieved a Net Lift of only +0.91% under the same evaluation system, ranking sixth. Breaking down by category, the model ranks first among open-weight models in all three categories — Work, Code, and Chat — demonstrating comprehensive capability improvement.

Particularly noteworthy is the reliability of Bash command execution. Among open-weight models, this model achieves a Bash reliability score of +7.9%, compared with -4.5% for the previous generation, which ranked 13th. This significant improvement means the model has achieved qualitative leaps in tool-calling capability, command generation accuracy, and error self-correction in real environments — precisely the most critical engineering metrics for agent deployment.

API Service and Call Optimization

For developers who cannot deploy locally, the model also offers API access. The API model name is kimi-k3.1-pro, and its interface is compatible with both OpenAI and Anthropic calling conventions, reducing integration costs. A kimi-k3.1-pro-highspeed version is also available, which the official team claims delivers inference speeds 2 to 3 times faster than the standard version. For reasoning parameter configuration, reasoning_effort supports three levels: low, high, and max, with max as the default, directly affecting the model’s depth of reasoning on complex tasks. For multi-turn tool-calling scenarios, developers should pass back the previous turn’s reasoning_content and tool_calls parameters together to maintain context consistency.

The following is an example of multi-turn tool calling via the API:

from openai import OpenAI
client = OpenAI(
    api_key="your_api_key",
    base_url="https://api.example.com/v1"  # Use a compatible endpoint
)
# First turn: request the model to call a tool
response = client.chat.completions.create(
    model="kimi-k3.1-pro",
    messages=[{"role": "user", "content": "Check the weather in New York and calculate the temperature difference"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "parameters": {"city": {"type": "string"}}
        }
    }],
    reasoning_effort="high"
)
# Extract the tool call result
tool_call = response.choices[0].message.tool_calls[0]
tool_result = get_weather(tool_call.function.arguments)
# Second turn: pass in the previous turn's reasoning_content and tool_calls
response2 = client.chat.completions.create(
    model="kimi-k3.1-pro",
    messages=[
        {"role": "user", "content": "Check the weather in New York and calculate the temperature difference"},
        response.choices[0].message,  # Includes reasoning_content and tool_calls
        {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": tool_result
        }
    ],
    reasoning_effort="high"
)

Summary and Outlook

The open-weight release of this model marks the entry of the open-source large language model community into the 2.8T-parameter era. Its technical approach — the deep combination of sparse attention and MoE, and the engineering application of the MXFP quantization format — provides valuable reference for future model design. From the evaluation data, the model significantly outperforms its predecessor in real agent tasks, particularly in tool calling and command execution, demonstrating practical value. However, the ultra-large-scale hardware requirements and inference costs remain the primary obstacles to broader adoption. How to find a better balance between model capability and deployment feasibility will be the core challenge for the next stage of technical evolution. For technical decision-makers, when selecting models of similar scale, they should comprehensively consider task complexity, hardware investment, and API costs to formulate flexible hybrid deployment strategies.