Cost to Self-Host a Trillion-Parameter MoE Model: Hardware and Budget Breakdown Across Three Deployment Tiers

As of this writing, the full weights and technical report for a certain 2.8-trillion-parameter MoE model have not been publicly released. The official disclosures cover only the architectural direction and API pricing. Therefore, the speed, VRAM footprint, and hardware configuration figures below are extrapolated from publicly available architecture information and general technical knowledge. They must be verified item by item with benchmarks once the weights are officially released. The goal here is not to provide a precise price quote but to offer a reusable evaluation framework: break “can it run” into three quantifiable dimensions—storage, bandwidth, and compute—and then discuss cost tier by tier.

1. Understanding the Resource Characteristics of This Model Class

This model uses a MoE architecture with 2.8 trillion total parameters, 896 experts, and 16 activated experts per token, giving an expert activation ratio of roughly 1.79%. A critical distinction: expert activation ratio is not the same as parameter activation ratio. VRAM footprint for MoE deployment is calculated based on total parameters; the activation ratio only affects compute consumption. You must first fit all 896 experts into addressable storage, then at inference time route each token to the 16 relevant experts and bring their weights into the computation.

The architecture backbone is KDA (Kimi Delta Attention) plus Attention Residuals, with Gated MLA as a component. This hybrid attention design directly affects KV cache size and access patterns, making it a variable that cannot be ignored when estimating workspace VRAM later.

On quantization, the model underwent quantization-aware training starting from the SFT stage, with weights in MXFP4 and activations in MXFP8. Converting 2.8 trillion parameters to 4-bit gives a raw data lower bound of approximately 1400 GB. However, MXFP4 carries an 8-bit scale for every 32 4-bit elements. Per OCP specification, weight elements plus scales total approximately 2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB. This does not yet account for non-4-bit tensors, metadata, alignment padding, runtime workspace, and KV cache. In other words, 1500 GB is only the theoretical lower bound for weights; actual deployment requires substantial headroom on top of this.

For reference: a 671B-parameter MoE model in FP8 format requires roughly 685 GB for main parameters plus MTP parameters, and even after 4-bit quantization still needs about 335 GB for weights plus 37 GB for activations—approximately 400 GB total to run on 8 high-VRAM GPUs (640 GB combined). The model discussed here is more than 4× that size, which determines the lower bound for full deployment.

2. Official Recommendations and the Interconnect Bottleneck

The official recommendation for this model class is a supernode deployment with 64 or more accelerators. The reason is not compute but communication: large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple cards, each token after routing must have its activations moved to the card hosting the target expert, computed, and moved back. Communication latency directly erodes compute throughput. A supernode expands the communication domain through high-bandwidth interconnect, keeping this movement within a single domain as much as possible.

This also explains a common misconception: stacking GPUs does not equal being able to run the model. If interconnect bandwidth is insufficient, adding cards only makes all-to-all slower. When evaluating cost, interconnect topology and bandwidth must be calculated alongside VRAM capacity.

3. Tier 1: Just Make It Run, Speed Irrelevant

The only goal: fit nearly 1500 GB of weights into addressable storage and get the model to emit tokens, even at single-digit tokens per second.

The most straightforward approach is heterogeneous GPU-CPU inference. Each MoE layer has a shared attention component (in this model, KDA plus Gated MLA) that is compute-intensive and stays on the GPU for high-bandwidth computation. The 896 sparse experts account for the bulk of storage but only 16 are activated at a time, so they can all be offloaded to CPU memory and fed by a large-memory server. A common industry practice is to place expert weights in host memory or even local NVMe, with the GPU retaining only attention and routing-related hot data.

The cost structure for this tier is clear:

• GPU side: No top-tier cards needed. One or two mid-to-high-end GPUs handle attention computation and routing scheduling; the key requirement is enough VRAM for activations, KV cache, and runtime workspace.
• CPU and memory side: This is the main cost. 1500 GB of weights require at least 1.5–2 TB of host memory for comfortable loading. Adding OS and runtime overhead, 2 TB is the recommended starting point. If memory is insufficient, NVMe can serve as second-tier storage, but this further sacrifices speed.
• Storage side: The weight files alone are about 1.5 TB. Reserve space for decompression, conversion, and caching—4 TB NVMe is recommended.
• Interconnect: Single-machine PCIe is sufficient; no supernode needed.

A typical configuration for this tier is one large-memory server plus a small number of GPUs, with cost concentrated in memory modules and NVMe. Based on general hardware price ranges, the total system cost typically falls in the range of several hundred thousand RMB, depending on memory generation, capacity, and GPU tier. Expected speed is single-digit tokens/s, suitable only for validation, debugging, and demos—not for any production use.

4. Tier 2: Single-User 25 tokens/s

25 tokens/s is the usability threshold for single-user interaction. Moving from Tier 1 to Tier 2, the bottleneck shifts from “fitting it in” to “moving it fast enough.”

The key change: each token generation activates 16 experts. If these experts all reside in CPU memory, every forward pass must move tens of GB of expert weights to the GPU over PCIe, and bandwidth immediately becomes the ceiling. To reach 25 tokens/s, frequently activated experts must reside permanently in GPU VRAM, with only cold experts left on the host side.

This leads to several engineering considerations:

1. Expert hotness analysis: Use a small calibration dataset to profile the routing distribution and identify the high-frequency expert set. MoE routing typically exhibits a pronounced long-tail distribution, with a minority of experts handling the majority of activations.
2. Tiered placement strategy: Place hot experts on GPU, warm experts in host memory, and cold experts on NVMe. The placement policy must be configurable for tuning against actual workloads.
3. VRAM budget: GPU VRAM must simultaneously hold attention weights, resident experts, KV cache, and workspace. KV cache size depends on context length and concurrency. Single-user short-context scenarios place less pressure, but long contexts quickly consume all available VRAM.
4. Quantization level: Weights no lower than MXFP4, activations in MXFP8. If weight precision is compressed further, confirm that quantization-aware training covered that precision; otherwise, accuracy loss is uncontrollable.

Single-card VRAM is limited, so multi-card partitioning is typically required. With 4–8 high-VRAM GPUs and high-speed inter-card interconnect, partitioning resident experts by layer across cards can significantly reduce cross-card transfers. At this point, cost shifts from “large memory” to “multiple GPUs plus high-speed interconnect,” with total system cost typically rising to the million-RMB range. Whether 25 tokens/s can be sustained depends on expert hit rate, inter-card bandwidth, and batch scheduling—this must be verified with actual benchmarks, not theoretical bandwidth alone.

5. Tier 3: 10 Concurrent Sessions at 25+ tokens/s Each

Tier 3 is the real production threshold: 10 concurrent sessions, each with output speed no lower than 25 tokens/s, and quantization no lower than MXFP4. This means aggregate throughput must exceed 250 tokens/s, with no significant latency degradation under concurrency.

The core tension at this tier is expert activation divergence within a batch. In single-user inference, each token activates 16 experts. At 10 concurrent sessions, tokens from different requests in the same batch may route to entirely different expert sets, dramatically increasing the total number of activated experts and multiplying the volume of transfers between GPU and host. There are three categories of countermeasures:

• Increase the resident expert ratio: Place more experts in GPU VRAM at the cost of more GPUs. Based on 1500 GB of total weights, even keeping just half resident requires close to 750 GB of GPU VRAM—equivalent to a dozen or more high-VRAM cards.
• Expand the high-speed interconnect domain: Use a supernode to bring dozens of cards into a single communication domain so that all-to-all traffic stays within it. This is precisely why 64+ accelerators are officially recommended. Interconnect cost accounts for a high share at this tier and cannot be estimated from GPU unit price alone.
• Request scheduling and batching optimization: Use general-purpose inference optimizations such as continuous batching and chunked prefill to stagger expert activation over time and improve VRAM and bandwidth utilization. KV cache must also be carefully managed, as long-context concurrency is the primary VRAM consumer.

The hardware profile at this tier approaches a small cluster: dozens of high-VRAM accelerators, supernode-class interconnect, and supporting power and cooling. Cost typically falls in the multi-million-RMB range and varies widely with interconnect solution and VRAM capacity. Factoring in redundancy, operations, and data center conditions, total cost of ownership rises further.

It must be emphasized that performance targets at this tier cannot be promised on paper. Actual MoE inference throughput depends heavily on routing distribution, sequence length distribution, and scheduling policy. Stress testing with workloads that resemble real business traffic is essential, verifying tokens/s and tail latency item by item.

6. A General Method for Cost Estimation

Abstracting the three tiers into an estimation procedure makes it applicable to other very large MoE models:

1. Calculate weight lower bound: total parameters × bits per element ÷ 8
   (include scale and other overhead bits, added back per quantization spec)
2. Add runtime overhead: non-quantized tensors, metadata, alignment, workspace, KV cache
3. Define placement strategy: GPU-resident / host memory / NVMe three-tier
4. Calculate bandwidth demand: bytes of activated experts per token × target tokens/s
   Compare against PCIe and inter-card interconnect bandwidth to find the bottleneck
5. Calculate concurrency scaling: concurrency × experts activated per token, assess in-batch divergence
6. Select interconnect topology: whether a supernode-class communication domain is needed
7. Verify with benchmarks: measured tokens/s, tail latency, peak VRAM

Step 4 is where mistakes are most common: many people calculate only VRAM capacity and ignore bandwidth. The bottleneck in MoE inference is often not “can’t fit it” but “can’t move it fast enough.” Step 5 determines why Tier 3 costs jump—concurrency does not scale compute linearly; it scales communication non-linearly.

7. Summary

The essential difference across the three tiers is that the bottleneck shifts from storage capacity to memory access bandwidth, then to interconnect and scheduling:

• Tier 1 buys memory: several hundred thousand RMB, single-digit tokens/s, validation only.
• Tier 2 buys VRAM plus inter-card bandwidth: million-RMB range, using hotness-based tiered expert placement to reach 25 tokens/s.
• Tier 3 buys supernode and scheduling capability: multi-million-RMB range, relying on a large communication domain and batching optimization to sustain 10 concurrent sessions.

Two final reminders: First, all figures above are estimates based on publicly disclosed architecture information and must be verified item by item with benchmarks once weights are released. Second, in the self-hosting cost of very large MoE models, hardware is only one part—data center space, power, cooling, and operations are equally significant and should be included in total cost of ownership when budgeting. For most teams, the safer path is to first validate the pipeline with heterogeneous inference during the proof-of-concept stage, then decide whether to upgrade to a cluster based on actual workload.