Trillion-Parameter MoE Model Local Deployment: Memory and Cost Estimation

1. Two Separate Budgets: Memory Follows Total Parameters, Compute Follows Active Parameters

The most common misreading of large-scale MoE models is to treat the “activation ratio” as the “memory ratio.” Take a 2.8-trillion-parameter model: it has 896 experts, and a router activates only 16 of them per token — an expert activation ratio of roughly 1.79%. That figure describes compute, not memory.

The reason is straightforward: routing is per-token and dynamic. You cannot predict which experts the next token will hit, so all 896 experts must reside in memory at all times, with only 16 called per inference step. In short:

• Total parameters × quantization bit-width → minimum deployment memory (all weights must be loaded)
• Active parameters × token count → FLOPs per token and theoretical latency

One detail often missed: 1.79% is the expert activation ratio, not the strict parameter activation ratio. The attention backbone, shared layers, router, normalization, and output head are on the critical path for every token and remain active throughout. These models typically use Delta-style linear attention variants with attention residual connections, and gated multi-head latent attention for low KV overhead — together forming a “always-on, always-active” base that further widens the gap between the memory budget and the compute budget.

2. Weight Storage Down to the Bit: The 4-bit Floor Exceeds 1,400 GB

Models of this scale usually undergo quantization-aware training (QAT) from the SFT stage. Weights are natively in MXFP4 format with MXFP8 activations, so no post-training quantization is needed at inference — accuracy loss is kept minimal from the training source. This means you can reliably calculate memory at 4 bits.

Step 1 — raw data floor:

2.8 trillion parameters × 4 bit ÷ 8 = 1,400 GB

Step 2 — add group-scaling overhead. Under the widely adopted micro-scaling (MX) format, every 32 four-bit elements share one 8-bit scale factor, averaging 4.25 bits per parameter:

2.8 trillion × 4.25 bit ÷ 8 ≈ 1,487.5 GB

Step 3 — 1,487.5 GB is “weights only.” Real deployment adds four categories of overhead:

1. Non-quantized tensors (embedding, output head, router weights, normalization parameters)
2. Metadata and alignment (expert boundary info, routing tables, allocator fragmentation)
3. Runtime workspace (activation buffers, communication staging, intermediate results)
4. KV cache (grows linearly with concurrency and context length)

In practice, plan for 10–20% headroom on top of pure weights — target 1.6–1.8 TB of total memory when sizing hardware.

3. A Reference Point: Deployment Floor for Hundreds-of-Billions-Parameter Models

Absolute numbers are hard to intuit, so use a widely deployed 700B-parameter open-source MoE model as a baseline. At FP8 it weighs about 685 GB; even with 4-bit weight quantization it needs roughly 335 GB of weights plus 37 GB for activations — about 400 GB total. The industry typically uses 8 × 80 GB flagship GPUs (640 GB combined) for this.

A 2.8-trillion-parameter model is over 4× that size. Memory scales nearly linearly with parameter count, so the pure-weight floor alone approaches 1.5 TB, and with all overhead lands in the 1.6–1.8 TB range. This is the hard hardware constraint: anything below that is not “deployment” — it is offloading, swapping experts to CPU RAM on demand at steep latency cost, covered later.

4. Why 64 Accelerators Is the Starting Point: All-to-All Is the Hidden Gate

A naive memory division — 1.7 TB ÷ 80 GB ≈ 22 GPUs — suggests a smaller cluster. Yet both the model’s publishers and industry practice recommend 64+ accelerators as a super-node, and the reason is not memory but communication.

Large-scale MoE inference is extremely sensitive to all-to-all bandwidth. Expert Parallelism shards the 896 experts across GPUs; after routing, each token’s activations must be dispatched to the target expert’s GPU and collected back after computation:

token → router → all-to-all dispatch → expert compute → all-to-all collect → next layer

This happens in every MoE layer, per token. If traffic traverses a generic inter-node network, round-trip latency and bandwidth bottlenecks starve the GPUs — they wait for data rather than compute. A super-node uses high-bandwidth chip-to-chip interconnects (hundreds of GB/s) to keep dispatch and collect within a single communication domain, never crossing slow links.

The correct way to estimate scale is the maximum of two constraints:

• Memory constraint: ceil(total memory required ÷ per-GPU memory)
• Communication constraint: number of GPUs needed to fit all experts in one high-speed domain (often the larger number in practice)

5. KV Cache and Concurrency: Don’t Let Throughput Eat Your Headroom

The general KV cache formula:

Per-token KV bytes ≈ 2 × num_layers × KV_dim × precision_bytes
Total KV = per-token KV × sequence_length × concurrent_requests

Delta-style linear attention and latent attention keep the effective KV dimension small, so per-token cost is far lower than in standard multi-head attention. But it is a triple product: longer context and higher concurrency both multiply fast. If per-token KV is on the order of tens of KB (verify against the actual architecture), a 128K-context, 100-concurrent service can need hundreds of GB — equivalent to several or a dozen extra GPUs. Treat KV cache as a separate line item: compute it for your target concurrency and context, then add to the weight total.

6. Cost Framework: Purchase, Rental, and Cloud Inference

With the hardware numbers sorted, the money side follows the same structure:

GPU count N = max( ceil(weight_memory ÷ usable_memory_per_GPU), communication_domain_requirement )
Purchase TCO ≈ N × per-GPU price + chassis & high-speed interconnect + facility power + 3–5 year depreciation
Rental cost ≈ N × per-GPU hourly rate × monthly effective hours

Using a 64-GPU scale for ballpark judgment (actual prices depend on procurement or rental channels):

• Self-built purchase: a one-time investment in the tens of millions of RMB, plus high-bandwidth interconnect chassis, rack power and cooling, dedicated ops staff, and electricity — amortized over 3–5 years.
• Hourly rental: converts capex to opex, good for peaky or experimental workloads; but if daily inference hours are low, per-unit cost can be higher.
• Managed cloud inference or direct API calls: if data residency, sub-millisecond latency, or extreme throughput are not requirements, managed services often give the best TCO; alternatively, evaluate GPU compute, container platforms, and monitoring capabilities offered by cloud providers on a pay-as-you-go basis, run the workload first, then decide whether to migrate on-prem.

Whether self-building makes sense boils down to a decision checklist:

1. Must data never leave your premises (hard compliance)?
2. Is millisecond-level, long-term stable latency required?
3. Is daily token volume high enough to amortize fixed costs?
4. Does your team have the skills to operate large-scale distributed inference?

If either of the first two is “yes,” on-prem is necessary; the last two decide whether self-build or rental is cheaper.

7. Cost-Reduction Paths: You Don’t Have to Go “Full Spec”

1. Accept native 4-bit: QAT makes 4-bit the design precision; pushing to lower bit-widths is aggressive post-training quantization with steep quality loss — not recommended for flagship models.
2. Choose a smaller model: for many use cases, models in the tens-to-low-hundreds-of-billions range are sufficient, with one to two orders of magnitude less cost.
3. Offload experts to CPU RAM: swap rarely used experts on demand — large latency penalty; suitable only for offline evaluation or very low QPS, not online serving.
4. Tiered routing: serve easy requests with a small model and escalate complex ones to the large model — a few full-power calls for a lower overall cost.
5. Inference framework optimization: continuous batching, prefill/decode separation, operator fusion, and similar techniques significantly boost per-GPU throughput, equivalent to lowering per-token cost — exhaust software optimization before adding hardware.

8. Summary

Back to the original question — how much does it cost to deploy a trillion-parameter MoE model on-prem? Three steps:

1. Memory floor: total parameters × quantization bit-width (including scale factors) starts around 1.5 TB; with runtime overhead and KV cache, plan for 1.7 TB+.
2. GPU floor: memory division yields a theoretical 20+ GPUs, but the communication-domain constraint pushes the engineering minimum to a 64-accelerator super-node.
3. Cost magnitude: self-build is a tens-of-millions-RMB capex item plus ongoing ops; rental and cloud are pay-as-you-go. Whether it is worth it hinges on compliance, latency, and throughput.

Lay all three numbers on the table before deciding — far safer than buying GPUs first and calculating later.