How Much Does It Cost to Self-Host a Trillion-Parameter MoE Model?

The question “how much does it cost” cannot be answered with a single number, because “can run,” “smooth single-user experience,” and “multi-user production” represent three fundamentally different hardware philosophies, with cost differences spanning more than two orders of magnitude. Let’s first define the requirements clearly:

• Tier 1: Just needs to run, speed irrelevant. The goal is to validate the pipeline, compatibility, and compliance. Even fractions of a token per second are acceptable.
• Tier 2: Single user, output speed at 25 tokens/s (t/s). Personal productivity level, with an experience close to mainstream cloud services.
• Tier 3: 10 concurrent users, each above 25 t/s, with quantization no lower than 4-bit floating point. Production grade, requiring concurrent throughput, low latency, and precision simultaneously.

This article uses the typical configuration of a publicly disclosed ultra-large-scale MoE model as the estimation baseline: approximately 2.8 trillion total parameters, 896 experts, 16 experts activated per token, weights quantized via quantization-aware training (QAT) from the SFT stage using 4-bit micro-scaled floating point, with 8-bit activations. It is important to emphasize: before the complete weights are publicly released, all memory, speed, and hardware figures below are estimates based on public architectural information. Once the weights are released, every number must be verified with benchmark tests.

1. The First Calculation: Memory Floor Is Determined by Total Parameters, Not Activation Ratio

1. MoE memory is calculated by total parameters.

Each token only activates 16/896 ≈ 1.79% of the experts (note: this is the “expert activation ratio,” not strictly the parameter activation ratio, since attention and shared layers participate in computation for every token). However, routing is a runtime behavior: before inference, you must load all 896 experts into fast-accessible storage. Therefore, the activation ratio determines the compute and data-read volume per token—i.e., speed—but what determines whether the model fits is always the total parameter count.

2. The real cost of 4-bit quantization.

The industry-standard micro-scaled 4-bit floating-point format (often called MXFP4) is not pure 4-bit: every 32 4-bit elements share an 8-bit scaling factor, resulting in an effective bit-width of 4 + 8/32 = 4.25 bits. Thus:

• Pure 4-bit data lower bound: 2.8 trillion × 4 ÷ 8 ≈ 1,400 GB.
• Including the scaling factor: 2.8 trillion × 4.25 ÷ 8 ≈ 1,487.5 GB.

This is only the “bare weights.” Embedding tables, the output layer, routing gates, normalization layers (non-quantized tensors), framework metadata and alignment, communication buffers, runtime workspace, and KV cache must all be accounted for separately. During planning, it is recommended to reserve 1.7–2.0 TB of total memory.

Since this class of model was trained with quantization-aware training from the SFT stage, 4-bit is its “native design point”: FP4 deployment incurs very little precision loss. Pushing it down to 2/3-bit, without retraining with QAT, would cause a significant quality collapse. Going up to FP8 would double the weight memory to roughly 3 TB. This is precisely why Tier 3 requires “quantization no lower than 4-bit floating point”—it is not a stringent requirement but a reasonable floor aligned with the model’s training objective.

3. Calibrating with an open-source 671B-class MoE model.

Using an industrially open-sourced 671B-parameter MoE model as a reference: in FP8 deployment, the main parameters plus the Multi-Token Prediction (MTP) module total roughly 685 GB; switching to mainstream grouped 4-bit quantization gives about 335 GB of weights plus 37 GB of activations, totaling roughly 400 GB, which can run with reasonable headroom on 8 flagship inference GPUs with 80 GB memory each (640 GB total). A 2.8-trillion-parameter model is approximately 4.2× the size of the 671B model, and memory requirements scale roughly linearly: the full deployment lower bound is approximately 1.5 TB (4-bit) / approximately 3 TB (FP8). This determines the minimum number of GPUs for all subsequent scenarios.

2. The Second Calculation: Three Tiers of Requirements, Three Hardware Solutions and Cost Ranges

Tier 1: Just needs to run—trade activation ratio for cost. Approximately ¥200,000–¥400,000 (≈$30,000–$55,000).

The core approach is expert offloading: place all weights in large system memory, and for each token only send the activated parameters (approximately 500 billion, or 2.8T × 1.79% ≈ 50B) to the compute unit. This translates to touching approximately 26.5 GB of 4-bit data per token.

• Option A: Dual-socket server + 2 TB DDR5 memory, weights resident in RAM, CPU inference. Aggregate memory bandwidth is hundreds of GB/s, theoretically allowing a dozen or so tokens per second, but after accounting for attention computation, routing scatter/gather, and framework overhead, the practical rate is likely single-digit tokens per second—and lower with long contexts.
• Option B: 1–2 large-memory GPUs + memory/SSD tiering. Shared layers and attention reside in GPU memory; expert weights are fetched from RAM or even NVMe on demand. Lower cost, but speed drops by another order of magnitude—only suitable for “getting it running.”

Option A costs approximately ¥200,000–¥400,000 for a full system; Option B is around ¥100,000+ (≈$14,000). These are suitable for privatization compliance validation, integration-layer development, and small-batch offline tasks.

Tier 2: Single user at 25 t/s—essentially fitting 1.5 TB of weights into GPU memory. Approximately ¥3 million–¥9 million (≈$420,000–$1.25 million).

Decoding is fundamentally a bandwidth problem: each token requires reading approximately 26.5 GB of activated weights, so 25 t/s means roughly 660 GB/s of effective bandwidth just for weight reads. A single 80 GB flagship GPU has 2–3 TB/s of memory bandwidth and could easily handle this in isolation—the problem is that 1.5 TB of weights cannot fit on one GPU.

• Minimum GPU count: 1,487.5 GB ÷ 80 GB ≈ 19 GPUs; with KV cache and runtime overhead, plan for 20–24 GPUs. If using GPUs with 140–190 GB of memory per card, this can be reduced to 9–13 GPUs, fitting in one or two chassis.
• The communication domain must converge: For single concurrency at batch=1, every token requires cross-GPU expert routing (moving activations to the GPU hosting the target experts, computing, then moving results back). This all-to-all traffic must use high-speed inter-GPU interconnect rather than cross-node networking; otherwise, communication latency will directly degrade decoding speed.
• Cost: Three 8-GPU servers with inter-node interconnect, roughly estimated at ¥6 million–¥9 million at current market prices; large-memory GPU solutions can compress this to ¥3 million–¥5 million, subject to vendor quotes.

Tier 3: 10 concurrent users × 25 t/s, quantization no lower than 4-bit floating point. This is the super-node domain. Approximately ¥20 million–¥30 million (≈$2.8 million–$4.2 million).

Ten concurrent users introduce three simultaneous pressures:

• Memory pressure: In addition to 1.5 TB of weights, you must accommodate 10 concurrent KV caches. With long contexts, the KV cache becomes non-negligible. Fortunately, this class of model uses latent attention and similar structures that significantly compress KV, but you must still reserve for “concurrency × context length,” plus prefill workspace.
• Communication pressure: 896 experts spread across dozens of GPUs; every token must move activations across GPUs after routing. With only 10 concurrent requests, communication cannot be amortized over large batches, so latency directly manifests as per-user speed degradation. For large-scale MoE inference, all-to-all communication bandwidth is the primary bottleneck.
• Compute pressure: Decoding compute itself is modest (250 t/s × roughly 100 GFLOP per token ≈ 25 TFLOPS, which a few dozen GPUs can handle with ease). However, 10 long-prompt prefill requests arriving simultaneously—while keeping first-token latency low—requires genuine dense compute and careful prefill/decode resource orchestration.

This is precisely why “super-nodes” exist: using high-speed inter-GPU interconnect to form a single communication domain across dozens of GPUs, keeping expert-routing traffic within the domain, and pushing transfer latency to sub-millisecond levels. Expert parallelism naturally spreads all 896 experts across the entire domain, with each GPU holding only a few experts. Publicly available information suggests the recommended deployment for a model of this scale is a super-node of 64 or more accelerators: 64 × 80 GB = 5.12 TB of aggregate memory, with weights occupying about 1.5 TB and the remainder reserved for KV cache, workspace, and headroom. In terms of cost, 8 servers with 8 GPUs each + high-speed interconnect switches + shared storage, roughly estimated at ¥20 million–¥30 million. There are also ongoing costs: at 40–70 kW under full load, annual electricity bills reach the hundreds of thousands of RMB, plus high-density power and cooling retrofits for the data center, inference-framework optimization, and dedicated operations staff.

3. Turning Estimates into a Method: Code and a Verification Checklist

def weight_mem_gb(n_params, data_bits, scale_bits=8, group_size=32):
    """Estimate MoE weight memory: total parameters × effective bit-width / 8."""
    eff_bits = data_bits + scale_bits / group_size
    return n_params * eff_bits / 8 / 1e9
# 2.8 trillion parameters, 4-bit data + one 8-bit scale per 32 elements
print(weight_mem_gb(2.8e12, 4))   # ≈ 1487.5 GB (bare weights, excluding KV cache and runtime)
def kv_cache_gb(layers, kv_bytes_per_layer, ctx_len, concurrency):
    """Estimate KV cache: bytes per layer × context length × concurrency (use compressed dimensions for latent attention)."""
    return layers * kv_bytes_per_layer * ctx_len * concurrency / 1e9

Verification checklist after the weights are released:

1. Measure bare weights: Load the model and read the GPU memory footprint; compare against the estimate. If the error exceeds 15%, investigate the proportion of non-quantized tensors.
2. Single-concurrency throughput: Measure t/s at 1K, 8K, and 32K context lengths to confirm the long-context degradation curve.
3. Concurrency stress test: Sustained load at 10, 20, and 50 concurrent users; check whether throughput scales linearly and whether the bottleneck falls on the interconnect.
4. Time to first token (TTFT): Measure TTFT with long prompts to verify whether prefill compute is sufficient.
5. Precision spot checks: Compare perplexity between the 4-bit deployment and high-precision reference output, with manual sample evaluation.
6. Load balancing: Observe whether expert routing is skewed; hot experts on specific GPUs are often hidden bottlenecks.

4. Hidden Costs and Alternative Paths

• Power and cooling: A 64-GPU cluster consumes 40–70 kW under full load, with annual electricity costs starting at hundreds of thousands of RMB. The data center requires high-density racks and possibly liquid cooling.
• Optimization talent: Large-scale MoE expert-parallel scheduling, batching strategies, and inference-framework tuning require dedicated engineers working continuously.
• Depreciation and iteration: Hardware depreciation is typically calculated over 3–5 years, while model architectures, quantization formats, and inference frameworks evolve faster. Leave expansion headroom when purchasing.
• Alternative paths: If all three budget tiers exceed your limits, first validate business value through official APIs or managed inference services in the cloud. Cloud GPU providers offer pay-as-you-go options, converting one-time capital expenditure into elastic operating costs—until your scale is clear enough to justify self-hosting.

Conclusion: To answer the title question in one sentence—Tier 1, “just needs to run,” costs approximately ¥200,000–¥400,000 (large-memory server + expert offloading); Tier 2, “single user at 25 t/s,” costs approximately ¥3 million–¥9 million (a cluster of roughly 20 GPUs); Tier 3, “10 concurrent production users,” costs approximately ¥20 million–¥30 million (a 64-GPU super-node plus supporting infrastructure). The memory calculation is deterministic multiplication; speed and GPU count are estimates requiring load testing; prices are variables requiring vendor quotes. Until the official weights are released and benchmark data is available, treat this article as a methodology for capacity planning, not a procurement checklist.