How Much VRAM and Cost Are Needed to Deploy a Trillion-Parameter MoE Model Locally

1. Defining the Problem: Three Levels of Performance Targets

The cost of locally deploying a trillion-parameter MoE model cannot be answered with a single number. Hardware cost is almost entirely determined by the performance target. Requirements can be divided into three tiers:

Tier 1: It runs at all. The only goal is to fit roughly 1.5TB of weights into addressable storage so the model can produce tokens, even at single-digit tokens per second. This tier is about whether the model can be loaded at all, not how fast it runs.

Tier 2: Usable by a single user. The requirement is at least 25 tokens/s output for a single user. This is the comfort threshold for human reading; below this speed, the interactive experience degrades noticeably.

Tier 3: Multi-user concurrency. The requirement is 10 concurrent streams, each maintaining at least 25 tokens/s, with quantization no lower than a certain threshold. This tier places simultaneous demands on memory bandwidth, total compute, and communication topology.

The cost gap between tiers can be an order of magnitude. Let’s break down each level.

2. The VRAM Lower Bound: MoE Deployment Scales with Total Parameters

A common misconception is that since a MoE model only activates a subset of experts per token, memory usage also scales with the activation ratio. This is incorrect.

The activation ratio affects compute consumption, not memory usage. During inference, all experts must reside in addressable storage. The router decides which experts to invoke based on the token. Memory planning must therefore be based on total parameter count, not activated parameter count.

Consider a MoE architecture with 2.8 trillion total parameters, 896 experts, and 16 experts activated per token. The expert activation ratio is approximately 1.79%, but all expert weights must be loaded into storage.

2.1 How Quantization Format Affects VRAM

Assume the model underwent quantization-aware training from the supervised fine-tuning stage, with weights in 4-bit floating-point format and activations in 8-bit floating-point format. The raw data lower bound:

2.8T parameters × 4 bit ÷ 8 = 1400 GB

Actual usage exceeds this figure. In 4-bit floating-point formats, every 32 4-bit elements carry an additional 8-bit scale factor. Following the OCP specification:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This still excludes non-4-bit tensors (such as normalization layers and embedding layers that may retain higher precision), metadata, memory alignment, runtime workspace, and KV cache. In practice, overhead beyond weights typically requires an additional 10%–20% headroom.

2.2 Reference Point: How Much a Smaller MoE Model Needs

Take a MoE model with approximately 670B total parameters as a reference. In FP8 format, the main parameters plus multi-token prediction parameters total about 685GB. Even with 4-bit weight quantization, it requires roughly 335GB of weights plus 37GB of activations — about 400GB total — to run on 8 accelerator cards with 80GB each (640GB total VRAM).

A 2.8-trillion-parameter model is more than 4 times this size. The VRAM lower bound for full deployment is thus determined: at least approximately 1.5TB of accelerator VRAM is required, with practical planning headroom of 1.8TB–2TB.

3. Why the Official Recommendation Is 64+ Accelerators in a Supernode

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth.

With 896 experts distributed across multiple cards, each token’s activations must be moved to the cards hosting the target experts for computation, then moved back. This transfer follows an all-to-all pattern, with communication volume dependent on expert distribution, batch size, and sequence length.

If the communication domain is too small, transfers must cross nodes over general-purpose networks, and latency will directly stall computation — GPUs sit idle waiting for data, and utilization drops. The value of a supernode is that high-bandwidth interconnect (such as NVLink or equivalent high-speed interconnect) enlarges the communication domain so that expert transfers stay within a single domain, keeping all-to-all latency within an acceptable range.

This is why “it runs” and “it runs fast” are two different problems. The former can be addressed through heterogeneous inference that works around communication bottlenecks; the latter must be solved head-on.

4. Tier 1 Approach: GPU + CPU Heterogeneous Inference

If the goal is simply “it runs,” the most direct approach is heterogeneous inference: keep compute-intensive components on the GPU, and offload the storage-heavy sparse experts to CPU memory.

The division of labor:

• Shared attention components (compute-intensive): kept on the GPU for high-bandwidth computation.
• Sparse expert components (storage-heavy, but only a few activated per token): offloaded entirely to CPU memory, served by large-memory servers.

The representative technical approach for this is a CPU-GPU heterogeneous inference framework. Public tutorials report high-performance numbers from a dual-socket server CPU paired with a consumer flagship GPU running a ~670B-parameter MoE model (4-bit quantization, ~377GB), achieving over 200 tokens/s for prefill and about 14 tokens/s for decode — though these figures assume expert selection strategy constraints.

For a 2.8-trillion-parameter model, the same approach requires:

• CPU memory: at least 1.5TB, with 2TB recommended. Dual-socket server platforms typically support 4TB–6TB of memory, so capacity is not the bottleneck, but memory bandwidth becomes the limiting factor for decode speed.
• GPU: one or more high-VRAM cards to host the attention components and KV cache.
• Storage: weight loading requires fast local storage, ideally an NVMe SSD array; otherwise load times can be measured in hours.

For this tier, the primary cost is CPU memory and the server platform; the GPU is actually the smaller expense. Expected speed: decode may reach only single-digit tokens/s and will decline as context length grows.

5. Tier 2 and Tier 3: From Single-User Usability to Multi-User Concurrency

5.1 Single-User 25 tokens/s

To achieve 25 tokens/s for a single user, CPU memory bandwidth is usually insufficient. MoE decode is memory-bandwidth-bound: each generated token requires reading the weights of activated experts. Even with only 1.79% of experts activated and 4-bit quantization, each token requires reading approximately 26GB of weight data.

1487.5 GB × 1.79% ≈ 26.6 GB/token

Generating 25 tokens in one second requires approximately 665 GB/s of effective bandwidth. This approaches the upper limit of high-end server memory bandwidth, and real-world effective bandwidth is far below theoretical values. Tier 2 therefore typically requires placing experts on accelerator cards as well, or adopting a hybrid approach: hot experts resident on GPU, cold experts on CPU.

If everything is placed on accelerators, 1.5TB of weights plus KV cache and workspace requires approximately 1.8TB of VRAM. At 80GB per card, this means 24+ cards; at 141GB per card, 14+ cards. Accounting for tensor parallelism and expert parallelism communication overhead, actual configurations are typically higher.

5.2 10 Concurrent Streams at 25 tokens/s Each

Tier 3 requires total compute at least 10 times that of Tier 2. At this level:

• VRAM: weights can be shared, but KV cache grows linearly with concurrency. In 10-stream, long-context scenarios, KV cache can reach hundreds of GB.
• Compute: total bandwidth demand during decode reaches the 6.6 TB/s level, requiring accelerator HBM.
• Communication: as expert parallelism increases, all-to-all communication volume multiplies. The supernode shifts from “recommended” to “required.”

The hardware configuration for this tier is essentially the officially recommended 64+ accelerator supernode. Costs enter the tens of millions RMB range.

6. Cost Estimation Framework

The analysis above can be organized into an actionable estimation framework:

Target Tier	Weight Storage	Accelerator VRAM	System Memory	Interconnect	Cost Range
Runs at all	1.5TB+	1–2 cards	2TB+	No high-speed interconnect required	~$100K level
Single-user 25 tokens/s	1.5TB+	1.8TB+	512GB+	Intra-node high-speed interconnect	~$1M level
10-stream 25 tokens/s	1.5TB+	2TB+	1TB+	Supernode-class interconnect	~$10M level

Cost items that are easily overlooked:

1. Storage cost: 1.5TB of weights requires fast local storage, with additional cache space potentially needed during inference.
2. Power and cooling: a 64-card supernode draws tens of kilowatts; electricity and cooling are ongoing costs.
3. Operational cost: large-scale heterogeneous clusters involve failure rates, driver compatibility, and scheduling systems, all requiring staffing.
4. Quantization calibration cost: if official quantized weights are not provided, self-quantization requires calibration datasets and validation pipelines.

7. Practical Recommendations

First, wait for weight releases before verification. All speed, VRAM, and hardware configuration figures in this article are estimates based on public information. After weights and technical reports are released, each item must be verified through benchmarks, particularly actual VRAM usage, decode speed, and communication overhead.

Second, start with Tier 1 validation. Do not plan a 64-card cluster from the outset. First use heterogeneous inference to verify the model runs and output quality is acceptable, then gradually scale up.

Third, pay attention to actual quantization overhead. 4-bit is not simply “parameter count divided by 2.” Scale factors, non-quantized tensors, and alignment padding all push actual usage above theoretical values. Leave 20% headroom when planning.

Fourth, communication topology determines the concurrency ceiling. If the goal is multi-user concurrency, a supernode is not optional. Within budget constraints, prioritize a sufficiently large communication domain over simply stacking compute.

Fifth, consider elastic cloud options. For most teams, a one-time investment of tens of millions in hardware carries excessive risk. Using on-demand cloud accelerator resources for validation and elastic scaling is a more pragmatic path. Container platforms, object storage, monitoring, and alerting capabilities provided by cloud vendors can significantly reduce deployment and operational complexity.

8. Summary

Locally deploying a trillion-parameter MoE model comes down to three core constraints: the VRAM lower bound is determined by total parameter count, the speed ceiling is determined by memory bandwidth, and the concurrency ceiling is determined by communication topology.

“It runs at all” can be achieved with heterogeneous inference at a cost in the $100K range. “Single-user 25 tokens/s” requires investment in the $1M range. “10 concurrent streams at 25 tokens/s” is a $10M-class engineering project. There are no shortcuts between tiers — only clear recognition of the constraints and systematic verification.