Kimi K3 Local Deployment: VRAM Requirements, GPU Count, and Cost Estimates
This guide estimates the memory requirements, GPU configurations, performance targets, and deployment costs for running Kimi K3 locally. It covers MXFP4 and MXFP8 storage calculations, compares K3 with a 671B MoE model, and outlines three deployment tiers ranging from basic development to production service. All figures are planning estimates based on publicly cited information and should be validated against official specifications, current hardware prices, and real-world benchmarks.
Data and Assumptions
This article is based on publicly available information cited at the time of writing. The parameter count, MoE architecture, and quantization formats have not been independently verified. Cost figures are estimated using assumed market prices. Confirm the official specifications and current quotations before purchasing hardware.
1. Calculating the Minimum Memory Requirement
Kimi K3 is reported to have 2.8 trillion total parameters and 896 experts, with 16 experts activated for each token. The activation ratio is approximately 1.79%:
16 / 896 ≈ 1.79%
This ratio affects the amount of computation performed per token, but not the storage requirement. All 896 experts must be stored in system memory or GPU memory, so memory capacity must be calculated from the total parameter count.
Assuming the weights use MXFP4, each element requires 4 bits, while every group of 32 elements shares an 8-bit scale. The effective storage per element is therefore:
4 + 8 / 32 = 4.25 bits
The estimated weight storage is:
2.8 × 10^12 × 4.25 ÷ 8≈ 1.4875 × 10^12 bytes≈ 1,487.5 GB
The raw 4-bit data accounts for approximately 1,400 GB, while the shared scales add about 87.5 GB.
In practice, the deployment also requires non-4-bit tensors—such as embeddings and normalization layers—along with metadata, alignment padding, runtime workspace, and KV cache. The actual minimum will therefore be higher than 1,487.5 GB. The estimates below use a planning baseline of more than 1,500 GB.
If MXFP8 is used instead, each element requires:
8 + 8 / 32 = 8.25 bits
The weight storage would then be approximately 2,887.5 GB, or close to 2.9 TB. The quantization format has a major effect on hardware requirements and cost.
2. Rough Comparison with a 671B Model
The source material cites an example from a similar MoE model. A 671B-parameter model reportedly requires approximately 685 GB for its main parameters and additional modules in FP8. With 4-bit quantization, the weights require about 335 GB, while activation memory is estimated at approximately 37 GB, for a total of around 400 GB. That configuration can reportedly run on eight 80 GB accelerator cards.
However, the measurement scope is incomplete. The original source does not specify whether the 37 GB activation figure includes KV cache, context length, or inference-framework overhead. Scaling by parameter count can provide only an order-of-magnitude reference.
K3 has approximately 4.2 times as many parameters:
2,800 ÷ 671 ≈ 4.2
Multiplying 335 GB by 4.2 gives roughly 1,400 GB, which is in the same range as the 1,487.5 GB estimate above.
3. Comparison of Deployment Tiers
| Deployment tier | Minimum memory requirement | Recommended configuration | Estimated cost | Suitable use case |
|---|---|---|---|---|
| Tier 1: Basic operation | Approximately 1,490 GB for weights in CPU memory, plus about 24 GB of GPU memory for Attention | One 24 GB consumer GPU, a dual-socket server, and 2 TB of DDR5 ECC memory | Approximately CNY 100,000–200,000 | Personal development and functional validation |
| Tier 2: 25 tokens/s for one user | At least 19 × 80 GB GPUs if the weights and KV cache are placed entirely on GPUs | Option A: one 8-GPU server with more than 850 GB offloaded to CPU memory; Option B: three 8-GPU NVLink servers, for 24 GPUs total | Option A: approximately CNY 1–1.6 million; Option B: approximately CNY 3–4.5 million | Internal tools for small teams |
| Tier 3: 10 concurrent users at 25 tokens/s each | Approximately 1,487.5 GB for weights, plus an estimated 40–80 GB for KV cache | Four 8-GPU servers, for 32 GPUs total, connected with 400G RDMA | Approximately CNY 4–6 million | Public-facing services operated by a team |
These estimates assume CNY 100,000–150,000 per GPU, based on the source material. System memory is estimated at CNY 30–60 per GB. Servers, networking, storage, and other infrastructure are listed separately.
4. Tier 1: Basic Operation
The goal of this tier is simply to generate tokens, even if generation is very slow. The GPU handles the Attention component, while the 896 experts remain in CPU memory and are processed by the CPU.
Decoding speed will be constrained by memory bandwidth. Although a dual-socket server can provide substantial memory bandwidth, expert computation still requires large volumes of data to be read from memory, making performance difficult to improve. If the prefill phase also runs on the CPU, time to first token will be high.
The source material cites an experience-based result of more than 10 tokens per second for a similar configuration. This figure has not been reproduced here and must be measured on the target hardware. This tier is suitable for development, debugging, and functional validation, but not for public service.
5. Tier 2: 25 Tokens per Second for One User
The performance target must first be defined precisely:
- Output speed: 25 tokens/s
- Batch size: 1
- Context lengths: preferably tested separately at 2K and 8K
- Quantization format: MXFP4
Without a consistent measurement methodology, performance figures cannot be compared reliably. A rate of 25 tokens/s corresponds to approximately 40 ms per decoding step.
Option A: CPU Offload
This configuration places more than 850 GB of expert weights in CPU memory. Decoding speed will be limited by CPU memory bandwidth.
The amount of expert data read at each step depends on the model’s total activated parameter count. Because the source material does not provide that figure, it is not possible to calculate a theoretical upper bound for decoding speed. Until benchmark data is available, there is no basis for confirming that this configuration can achieve 25 tokens/s.
Option B: Store All Weights on GPUs
With 24 GPUs, the average weight allocation is:
1,487.5 ÷ 24 ≈ 62 GB per GPU
Each 80 GB GPU would have approximately 18 GB remaining for KV cache and activation values. This margin is tight. An 8K context may be possible, but it requires real-world testing.
Cross-server communication will add latency at every step. Whether the system can reach 25 tokens/s depends on the interconnect bandwidth and communication scheduling.
The original 8-GPU proposal claimed that 640 GB of VRAM could hold 1,200 GB of weights. That conclusion is not valid. An 8-GPU system can run K3 only through substantial CPU offloading, or by increasing the deployment to at least 24 GPUs.
6. Tier 3: Ten Concurrent Users
If 10 users each require 25 tokens/s, the system must sustain approximately 250 tokens/s in aggregate.
In a 32-GPU configuration, the average weight allocation is:
1,487.5 ÷ 32 ≈ 46.5 GB per GPU
Each GPU would have roughly 30 GB remaining for KV cache and runtime workspace, providing a more comfortable memory margin.
KV cache usage depends on the model architecture. Ten users with 8K contexts represent approximately 80,000 tokens in total. The source material estimates KV cache usage at 40–80 GB, but it does not provide the architectural parameters needed to verify this figure. Actual usage must be measured.
The 32 GPUs would be distributed across four servers. Expert All-to-All communication would therefore cross server boundaries. The source material states that in-server NVLink latency is approximately one-fifth to one-tenth of cross-node latency, but it does not provide a source for this comparison. The actual communication performance should be validated through benchmarks.
Speculative decoding would require additional GPU memory. Its benefit depends on how well the draft model matches K3 and on the resulting acceptance rate. This article does not provide a specific performance estimate.
For power planning, assuming approximately 10 kW per 8-GPU server, four servers would require about 40 kW in total. Before deployment, verify data-center power capacity, cooling, and floor-load limits.
7. Common Mistakes
- Assuming that a small number of activated parameters means a small memory requirement. Memory capacity is determined by the total parameter count.
- Assuming that eight 80 GB GPUs can directly hold K3. Their combined 640 GB is far below the estimated 1,487.5 GB requirement.
- Treating the FP8 version as a 1.4 TB model. The estimated FP8 weight size is approximately 2.9 TB.
- Looking only at hardware prices while ignoring facilities, electricity, operations, and debugging time. The source material estimates that software debugging may take two to four weeks.
- Applying deployment figures from other models directly to K3.
8. Factors That Still Require Validation
- Whether the official weights use MXFP4 and whether other versions will be available.
- Actual KV cache usage at different context lengths.
- Whether the inference framework natively supports MXFP4. If dequantization is required, storage usage and performance may change.
- Current GPU and memory prices, which can vary significantly. Obtain current quotations before purchasing.
9. Budget Planning Recommendations
- Basic operation: Approximately CNY 100,000–200,000. Suitable for individual developers and experimental environments.
- 25 tokens/s for one user: Approximately CNY 1–1.6 million for the CPU-offload option, or CNY 3–4.5 million for the 24-GPU option. Both configurations require benchmark validation.
- Production service for 10 concurrent users: Approximately CNY 4–6 million or more, with a professional operations team likely required.
