Local Deployment of a Trillion-Parameter MoE Model: A Tiered Configuration and Cost Guide

As of this writing, the complete weights for a large MoE model with 2.8 trillion total parameters have not been publicly released; only architectural details and API pricing have been disclosed. Based on the publicly available architecture, this article breaks down the deployment requirements into three progressive tiers, covering memory, hardware, and communication costs, and provides a reusable estimation methodology. Once the weights are released, readers can verify these estimates with benchmark tests. All speed, memory, and hardware figures are estimates based on public information, not official measurements.

1. Memory First: The First Principle of MoE Deployment

This model uses an MoE architecture with 2.8 trillion total parameters and 896 experts. Each token activates 16 of them, giving an expert activation ratio of about 1.79%. A common misconception is assuming that the expert activation ratio equals the parameter activation ratio. In MoE deployment, memory usage must be calculated based on the total parameter count. The activation ratio affects compute, not storage. You must fit all 896 experts into addressable memory; at inference, routing logic only fetches the 16 activated experts per token.

The core architecture is KDA (Kimi Delta Attention) with Attention Residuals, of which Gated MLA is a component. The model was trained with quantization-aware training starting from the SFT phase, using MXFP4 for weights and MXFP8 for activations. Converting 2.8 trillion parameters to 4-bit gives a raw lower bound of about 1400 GB. However, MXFP4 includes an additional 8-bit scale per 32 4-bit elements. Per the OCP specification:

Weights + scales = 2.8T × 4.25 bits ÷ 8 ≈ 1487.5 GB

This excludes non-4-bit tensors, metadata, alignment overhead, runtime workspace, and KV cache. For reference, a 671B-parameter MoE model requires about 685 GB for main and MTP parameters in FP8. When quantized to 4-bit, it still needs ~335 GB for weights plus 37 GB for activations—roughly 400 GB total to run on eight 80 GB GPUs (640 GB total). Since this 2.8T model is more than four times larger, its memory floor dictates the full deployment baseline.

2. Communication: Why 64+ GPU Supernodes Are Recommended

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts spread across multiple GPUs, each token’s activations must be routed to the GPU hosting its targeted experts, then sent back after computation. Communication latency directly reduces compute efficiency. More GPUs and finer expert sharding increase data movement frequency.

The official recommendation to use supernodes with 64+ accelerators is driven by the need to keep expert traffic within a single high-speed communication domain, such as NVLink or equivalent. This is not about stacking GPUs for prestige; it’s a physical constraint of MoE sparse activation. As expert parallelism increases, the interconnect topology must keep all-to-all hop counts low; otherwise, compute utilization drops sharply.

3. Three Tiers of Configuration and Cost Estimates

Tier 1: Getting It to Run, Regardless of Speed

The only goal is to load nearly 1500 GB of weights into addressable storage and get the model to generate tokens, even at single-digit tokens per second.

The most straightforward approach is GPU + CPU heterogeneous inference, using frameworks like KTransformers. In MoE, each layer has a shared attention component (here, KDA with Gated MLA) that is compute-intensive and stays on the GPU for high-bandwidth processing. The 896 sparse experts occupy most of the storage but only 16 are activated per token; they can be fully offloaded to CPU memory, fed from a large-memory server.

Below is an illustrative configuration sketch (pseudocode, not runnable, expressing the offloading logic):

# Pseudocode: MoE heterogeneous offloading strategy
config = {
    "gpu": {
        "modules": ["kda_attention", "gated_mla", "shared_experts"],
        "dtype": "mxfp4",
    },
    "cpu": {
        "modules": ["routed_experts"],  # All 896 experts reside in RAM
        "dtype": "mxfp4",
        "numa_aware": True,
    },
    "io": {"prefetch_depth": 2, "pin_memory": True},
}

Cost estimate: This requires a large-memory server—2 TB of RAM is recommended as a starting point (1500 GB weights plus runtime headroom)—paired with 1–2 GPUs with 80 GB of memory for attention computation. The total system cost ranges from tens of thousands to a few hundred thousand RMB, depending on memory generation, channel count, and GPU selection. Throughput is not guaranteed at this tier; it serves verification, debugging, and feature demonstration purposes only.

Tier 2: 25 tokens/s for a Single User

25 tokens/s is the baseline for smooth single-user interaction. At this point, the bottleneck shifts from “does it fit” to “can we feed it fast enough.” CPU memory bandwidth becomes the critical factor. Although only 1.79% of expert weights are activated per token, each token still requires fetching the corresponding expert shards from memory. Insufficient memory bandwidth will directly throttle decode speed.

Configuration recommendations:

• Keep attention and shared experts on the GPU side, choosing high-bandwidth memory GPUs.
• On the CPU side, prioritize high-memory-bandwidth platforms (multiple channels, high frequency) over sheer capacity.
• Use NUMA-aware expert placement and prefetching to reduce cross-node access.
• If a single machine’s bandwidth is still insufficient, split into a small GPU cluster of 2–4 cards, sharding experts across GPUs and using high-speed interconnect for all-to-all traffic.

Cost estimate: A complete system typically falls in the range of a few hundred thousand to several hundred thousand RMB. The dominant costs are high-speed memory and multi-GPU interconnect. This tier suits individual developers or small teams building prototypes.

Tier 3: 10 Concurrent Users with 25+ tokens/s per User, Quantization ≥ 4-bit

Ten concurrent users roughly doubles KV cache and batch scheduling overhead while maintaining 25 tokens/s per stream. This demands surplus in both compute and communication. Heterogeneous offloading is effectively ruled out at this scale; you need a GPU cluster with supernode-grade interconnect.

Configuration recommendations:

• 64+ accelerator supernode with expert parallelism matched to the interconnect domain.
• Quantization no lower than 4-bit (MXFP4 weights, MXFP8 activations) to balance precision and memory.
• KV cache must be budgeted separately based on 10 concurrent users and long-context requirements.
• Deploy request scheduling and expert-affinity orchestration to prevent hot experts from saturating a single GPU.

Cost estimate: This tier moves into cluster acquisition or rental territory. Full system costs can reach several million RMB. For rental, pricing is typically based on accelerator hours plus high-speed interconnect bandwidth. For most teams, the more practical path is to first stress-test on cloud-based supernode instances, then decide whether to build on-premises.

4. Four Common Cost-Estimation Pitfalls

1. Calculating memory from activated parameters only: MoE storage must be sized by total parameters. Activation ratio affects compute, not memory.
2. Ignoring scales and metadata: MXFP4 scales push weights from 1400 GB to ~1487.5 GB. Add runtime overhead and leave 15–20% headroom.
3. Underestimating all-to-all communication: More experts mean more fragmented traffic. Interconnect bandwidth must scale with the GPU count.
4. Forgetting the KV cache: As concurrency rises, KV cache can consume more memory than expected. Model it separately.

5. Implementation Recommendations

Before the weights are released, run small-scale MoE models through both heterogeneous offloading and expert-parallel pipelines. Set up scheduling, metering, and monitoring instrumentation. Once the full weights arrive, benchmark each tier against the framework above and calibrate memory and bandwidth figures. For budget-sensitive teams, validate Tier 2 and Tier 3 feasibility on cloud-based on-demand instances before committing to hardware purchases.

Summary: Tier 1 starts at tens of thousands of RMB and is validation-focused; Tier 2 ranges from a few hundred thousand to several hundred thousand RMB for smooth single-user operation; Tier 3 reaches the million-RMB scale for production-grade concurrent use. The real cost driver is not the GPU unit price but whether memory bandwidth and interconnect topology match the sparse activation characteristics of MoE.