Kimi K3.1 Open Weights: Deploying a 2.8T-Parameter Sparse MoE Model, from Quantized Storage to 1M-Token Long Context

Rethinking Model Scale: Total Parameters, Active Parameters, and Expert Routing

Before discussing deployment, one key concept needs clarifying: a model’s “total parameter count” and the “parameters actually activated per inference” are not the same thing — especially for sparse expert (MoE) models.

The released model has about 2.78 trillion total parameters, but only about 104 billion are activated in the forward pass for a single token. This design means:

• Model capacity can support a much richer store of knowledge and capabilities;
• Per-token compute cost does not scale linearly with total parameters;
• Inference throughput depends more on expert routing, caching strategy, and batch scheduling than on simply stacking more parameters.

From an engineering perspective, this architecture suits workloads that need “broad coverage with deep reasoning”: long documents, complex codebases, and multi-turn tool calling. For deployment teams, hardware sizing should not be based on total parameters alone — active parameters, KV cache size, and the VRAM growth caused by context length all matter.

1.1 Why “2.8T Parameters” Doesn’t Mean “2.8T of VRAM”

The weights repository shows the model is stored in MXFP4 format, with the repo occupying roughly 1.56TB. Activation compute uses MXFP8. This separation between storage and compute directly affects deployment cost:

• MXFP4 significantly reduces the weight storage footprint;
• MXFP8 preserves higher numerical precision at inference time, mitigating the output quality degradation that low-precision quantization can cause;
• Repo size and runtime VRAM are not the same thing — you also need room for activations, optimizer states (if fine-tuning), and the KV cache.

When planning resources, teams should estimate “weight storage,” “runtime activation memory,” and “long-context KV cache memory” as three separate line items, rather than purchasing hardware based on repo size alone.

1.2 What Quantization-Aware Training Means

According to the official release, the model was trained with quantization awareness. This means quantization was not a simple post-training conversion: quantization error simulation was introduced during training so that the model’s parameters adapt to low-precision representations.

For users, the benefits are:

• Compared with post-training quantization, the model is more likely to remain stable at low precision;
• On numerically sensitive tasks such as code generation and math reasoning, precision loss is relatively contained;
• That said, the quantized model may still differ subtly from a full-precision model. For scenarios with strict output-consistency requirements, run systematic regression tests.

Architecture Changes: From Full Attention to Sparse Attention

The model has 93 layers: 69 use the KDA structure and 24 use Gated DSA. Compared with the previous generation, a key change is replacing some full-attention Gated MLA layers with Gated DeepSeek Sparse Attention (DSA), with the goal of handling long context more efficiently.

2.1 Why Hybrid Attention Layers Matter

In long-context scenarios, full attention’s compute grows rapidly with sequence length, and the KV cache becomes a VRAM bottleneck. By introducing sparse attention layers in part of the network, the model can reduce attention compute there, which:

• Cuts VRAM usage for long sequences;
• Speeds up inference on long documents;
• Lowers resource consumption while preserving the model’s expressiveness.

However, hybrid attention also brings new engineering challenges. The inference framework must support different attention implementations for different layers and schedule them correctly under tensor, pipeline, and expert parallelism.

2.2 Expert Layers and the Vision Module

The model’s expert layers contain 896 experts, with 16 activated per token plus 2 shared experts. The vision side uses MoonViT-V2 with 401M parameters, giving the model image-and-text input capability.

The expert routing design implies:

• Different inputs activate different expert combinations, so deployment must reserve enough VRAM to load all expert weights;
• Shared experts handle cross-domain, general-purpose features, while routed experts carry finer-grained knowledge;
• The inference framework must balance expert parallelism against data parallelism to avoid communication bottlenecks.

For the vision module, deployment additionally involves preprocessing image inputs, generating visual tokens, and fusing image and text modalities — all of which increase input-stage latency and VRAM usage.

VRAM and Hardware Planning for Self-Hosting

For teams that want to self-host, the hardware bar is unavoidable. One community-derived estimate describes a workable configuration: a server with 8 NVIDIA B300 accelerators and roughly 2.3TB of total VRAM, with hardware cost likely in the $550,000–$650,000 range. Note that this estimate comes from the community, not from official guidance.

3.1 Breaking Down VRAM Requirements

Deploying a 2.8T-parameter, 104B-active sparse MoE model requires VRAM for:

1. Weights: about 1.56TB in MXFP4, which still occupies a corresponding amount of space once loaded into VRAM;
2. Activations: intermediate tensors from each forward pass, driven by batch size and sequence length;
3. KV cache: a function of context length, batch size, layer count, and attention head dimensions;
4. Framework overhead and communication buffers: multi-GPU parallelism also consumes VRAM for buffers.

At a 1M-token context, the KV cache can become the dominant consumer. Before deploying, build a VRAM estimation model based on your target batch size and sequence length — don’t judge by weight size alone.

3.2 Framework Support and Deployment Recipes

Several mainstream inference frameworks already ship deployment recipes, including vLLM, SGLang, and TokenSpeed. Teams don’t have to write inference code from scratch, but should still check:

• How tensor parallelism and expert parallelism are partitioned;
• Operator support for the hybrid attention layers;
• Whether the MXFP4 and MXFP8 dequantization path is efficient;
• Whether paged attention implementations remain stable at long context.

In practice, validate the end-to-end inference path at a smaller configuration first, then scale up to full context and target throughput.

API Usage and Tool-Calling Constraints

Teams that don’t want to maintain their own hardware and serving stack can use the API instead. The model name is kimi-k3.1-pro, compatible with both OpenAI and Anthropic API styles; a high-speed variant is also available, officially stated to be 2–3x faster than the standard version.

4.1 Reasoning Effort and Returned Content

The model exposes a reasoning_effort parameter with values low, high, and max, defaulting to max. Under the default configuration, the model may produce lengthy reasoning traces, increasing both response time and token consumption. For latency-sensitive scenarios, try lowering the reasoning effort.

In multi-turn tool calling, you must pass the previous turn’s reasoning_content back along with tool_calls. This is a critical constraint:

• If you return only the final text and discard the reasoning, the model may lose conversational coherence;
• If you omit tool_calls, the model may fail to correctly interpret the previous turn’s tool execution results;
• Session state must be maintained carefully, and key fields must not be truncated across turns.

A good practice is to persist reasoning content and tool-call records as part of session state — useful not only for the current conversation but also for post-hoc analysis and quality evaluation.

4.2 Combining Long Context with Tool Calling

The model supports 1M-token context and tool calling. Combining the two raises several common issues:

• Overly long context leads to missed or duplicated tool calls;
• Early information gets compressed or forgotten after many tool-call rounds;
• High reasoning effort settings significantly increase the total wall time of tool-call chains.

To address these, design prompts at the application layer that clearly tell the model when to call tools and how to organize returned results; additionally, chunk long documents semantically and combine them with retrieval or summarization strategies.

Agent Evaluation Signals and How to Read the Results

On the day the weights were released, an evaluation based on Agent Arena was published. This evaluation draws on real user agent sessions rather than offline benchmarks, extracting model performance from several signal types:

• Whether the user explicitly confirmed task success;
• Whether the user expressed praise or complaints;
• Whether the model adjusted promptly after user corrections;
• Whether shell commands executed successfully.

These signals are combined into a “net uplift” metric. Based on more than 17,500 real long-horizon sessions, the model achieved +5.1% net uplift in Agent Arena — ranked #1 among open-weight models and #13 on the overall leaderboard. It ranked #1 among open models in all three categories: Work, Code, and Chat.

By comparison, the previous generation scored +0.91% net uplift in the same evaluation, ranking #6 among open models. On the Bash reliability submetric, this model reached +7.9% versus -4.5% for the previous generation, moving from #13 to #1 among open models.

5.1 Why This Kind of Evaluation Matters

Traditional offline benchmarks measure capability on fixed datasets, but real agent work typically involves:

• Multi-step planning and execution;
• Environmental feedback and error correction;
• State maintenance over long-running sessions;
• Actual pass/fail judgment of command execution.

Evaluation based on real sessions reflects how stable a model is in complex, dynamic environments. For teams planning to use the model to drive automation scripts, code fixes, or data analysis, these signals are more informative than generation quality alone.

5.2 Engineering Takeaways from the Results

The data shows clear improvements in command reliability and long-horizon task completion. In engineering terms, this means:

• Lower failure rates are plausible in automated pipelines;
• Better ability to continue a task after user corrections;
• More reliable state maintenance across multi-turn interactions.

That said, evaluation results still depend on how the model is actually used. Build an internal evaluation set for your own business scenarios, focusing on task completion rate, command execution success rate, error recovery capability, and latency distribution — rather than relying solely on public rankings.

Summary and Recommendations

The open-weights release gives teams with sufficient compute the option to self-deploy, fine-tune, and audit the model. Its core characteristics can be summarized as:

• About 2.8T total parameters and about 104B active parameters — a large-scale sparse MoE architecture;
• Weights stored in MXFP4, activations computed in MXFP8, with quantization-aware training;
• Hybrid KDA and Gated DSA layers for more efficient long-context processing;
• 1M-token context and image-text input support;
• Deployment recipes already available from inference frameworks, but hardware cost and VRAM requirements are high;
• Strong results in agent long-horizon evaluations, with notable gains in command reliability and error recovery.

Recommendations by role:

• Algorithm teams: focus on the model architecture and training strategy, and assess whether domain adaptation based on the open weights is worthwhile;
• Infrastructure teams: build a VRAM estimation model early, choose appropriate parallelism strategies and an inference framework, and run end-to-end load tests;
• Application teams: pay attention to API constraints, reasoning effort settings, and multi-turn tool-call state management to keep production systems stable.

Finally, it’s worth emphasizing that model capability is only one part of the system. Context management, tool interface design, error handling, and observability are what determine whether an agent application lands reliably in production. When facing large-scale sparse models and long-context workloads, engineering rigor and patience often matter more than the model itself.