Deploying Trillion-Parameter MoE Models Locally: A Guide to Compute, VRAM, and Cost Estimation

1. Background: Why Trillion-Scale MoE Models Are Hard to Deploy Locally

Large language models have increasingly moved toward sparse “mixture-of-experts” architectures. Unlike dense models, which activate every parameter in each forward pass, MoE models split most parameters into multiple expert networks and activate only a small subset during inference. This significantly reduces the floating-point compute per inference step, but it introduces a critical engineering problem: the inference node must still keep all expert weights in addressable storage, waiting for routing decisions.

For a typical trillion-scale sparse model, the total parameter count can exceed two trillion. Even with 4-bit quantization, the weight volume alone approaches 1.5 TB. Add activations, runtime workspace, key-value cache, communication buffers, and model metadata, and the actual storage requirement grows further. The difficulty of local deployment therefore is not merely “can you buy enough GPUs”—it is how to design a heterogeneous compute system that is cost-controlled, communication-efficient, and delivers acceptable inference speed.

This article does not discuss the official recommended setup for any specific model. Instead, it approaches the problem from a general engineering perspective, splitting deployment goals into three tiers—getting it to run, achieving smooth single-user throughput, and supporting multi-user concurrency—and derives hardware lower bounds and cost structure for each.

2. Three Sets of Key Parameters to Understand Before Deployment

2.1 Total Parameters, Activated Parameters, and Expert Activation Ratio

In a mixture-of-experts architecture, “activated parameters” are not equal to “total parameters.” Suppose a model contains roughly 2.8 trillion total parameters, with close to 900 experts, and activates only a dozen or so experts per forward pass. The expert activation ratio is then about 1.8%.

Note, however: the expert activation ratio affects only the compute per forward pass, not VRAM usage. The inference node must store all expert weights in full, because the router dynamically selects different experts for each input. If an expert is not resident in local VRAM, the computation cannot complete.

2.2 Weight Quantization Format and Effective Bit Width

Weight quantization is the primary means of compressing model size. Common 4-bit quantization is not literally “4 bits per parameter.” For example, with a block-scaling floating-point quantization format, every 32 4-bit elements carry an 8-bit scaling factor. Under this scheme, a 2.8-trillion-parameter model in 4-bit quantization totals roughly 1487 GB to 1500 GB for weights plus scaling factors.

This figure is the “bare weight lower bound,” not the final VRAM requirement. Actual deployment must also account for the following overheads:

• Non-quantized tensors such as embedding layers, normalization layers, and output heads;
• Memory alignment and sharding redundancy;
• Runtime workspace and activation cache;
• Key-value cache generated by concurrent requests;
• Communication library buffers and model-parallel partitioning overhead.

VRAM planning should therefore use “1.4x to 2x the bare weight volume” as the estimation range, depending on framework implementation and whether tensor parallelism, pipeline parallelism, or other strategies are used.

2.3 Communication Domain and All-to-All Bandwidth

Sparse expert models involve substantial “all-to-all” communication during inference. In each forward pass, the hidden state of every input token must be sent to the device hosting the selected expert; after expert computation, outputs are gathered back to the originating device. If experts are spread across multiple servers connected by ordinary Ethernet, communication latency will significantly drag down overall throughput.

Trillion-scale sparse models therefore typically require deployment within a high-speed-interconnect supernode. Such supernodes organize dozens or even hundreds of accelerators into a single communication domain via high-bandwidth, low-latency links, keeping expert scheduling from crossing slow network boundaries.

3. Tiered Deployment Options

Tier 1: Just Make It Run, Speed Irrelevant

The goal of this tier is to let the model complete inference and produce output, even if only a few tokens per second. It suits technical validation, weight verification, academic research, and low-cost stress testing.

The minimum deployment threshold is determined by two factors: total weight volume and addressable storage space. Since bare weights approach 1.5 TB, a single-machine multi-GPU setup must provision sufficient VRAM or unified memory. If each GPU has 80 GB of VRAM, at least about 20 GPUs are needed to hold all expert weights. If CPU memory offloading is used, a large-memory server is required, with memory capacity preferably no less than 1.8 TB to 2 TB.

One common approach is heterogeneous inference: keep the compute-intensive shared attention modules on GPU and offload the massive but sparsely activated expert modules to CPU memory. This significantly reduces GPU count, but every expert computation requires cross-device data movement, which is slow.

Another approach uses large-capacity enterprise SSDs as cold storage, dynamically loading weights by layer or by expert. This introduces severe I/O bottlenecks and is suitable only for batch processing or offline evaluation.

From a cost perspective, the minimum configuration for this tier can be kept in the range of several hundred thousand to over a million RMB, depending on whether second-hand enterprise equipment is purchased and whether extremely low throughput is acceptable.

Tier 2: Single-User Output at 25 tokens/s

When the deployment goal shifts from “it runs” to “it is usable,” single-user interactive generation speed must be guaranteed. 25 tokens/s means total latency per decoding step must not exceed 40 ms, placing higher demands on compute, communication, and memory access.

At this point, CPU offloading can no longer serve as the primary compute path. Weights should reside in accelerator VRAM as much as possible, with the CPU used only for auxiliary scheduling. Assuming 80 GB of usable VRAM per GPU, an estimated 32 to 64 accelerators are needed to simultaneously hold weights, activations, and KV cache while leaving communication headroom.

The key to smooth single-user performance is shortening the critical path. Possible optimizations include:

• Placing shared attention components and high-frequency experts within a closer communication domain;
• Using tensor parallelism to solve single-layer weights exceeding VRAM;
• Using pipeline parallelism to reduce idle waiting;
• Quantizing activations to lower precision, such as 8-bit floating point;
• Pre-allocating fixed-size KV cache to avoid dynamic allocation overhead.

If the hardware platform itself offers high-speed intra-server multi-GPU interconnect, prioritize compressing the model into a single server. If a single server cannot hold all weights, remote direct memory access networking must be introduced across servers, with per-step communication time kept within a few milliseconds.

Tier 3: 10 Concurrent Users at 25 tokens/s Each

Concurrency differs fundamentally from the single-user scenario. It not only multiplies compute but also causes KV cache to grow linearly. If each request has an average cache length of several thousand tokens, 10 concurrent requests may consume tens of additional GB of VRAM.

The bigger challenge comes from batch scheduling and expert load balancing. Although the sparse structure reduces per-token compute, when multiple users infer simultaneously, different inputs may activate different expert combinations, causing queues at some experts. If routing distribution is uneven or intra-batch activation is concentrated, overall throughput drops noticeably.

Tier 3 therefore typically requires a true supernode deployment. A starting point of 64 accelerators is recommended, ideally scaling to 128 or more. All devices should reside in the same high-bandwidth communication domain to avoid cross-rack expert scheduling.

The inference framework must also support continuous batching and dynamic routing. For applications with large shared prefixes, prefix caching can be introduced; for long-text scenarios, paged KV cache can reduce memory fragmentation.

Hardware costs at this tier rise significantly. Excluding racks, power, cooling, and operations, accelerator purchase cost alone may reach several million to over ten million RMB. If renting equivalent cloud compute, long-term inference cost should be evaluated on an hourly or monthly basis, with attention to communication performance, VRAM specifications, and inter-instance bandwidth.

4. Hardware Configuration Derivation Table

The following estimates assume 80 GB VRAM accelerators, ~1.5 TB of 4-bit weights, and multi-GPU parallel inference. Results are order-of-magnitude references only.

Deployment Tier	Recommended Accelerator Count	Memory/VRAM Requirement	Approximate Hardware Cost Range	CPU Offloading Required
Just make it run	20+	~2 TB	Several hundred thousand to over 1 million RMB	Allowed
Single user at 25 tokens/s	32–64	~3 TB to 6 TB	Several million RMB	Not recommended
10 concurrent at 25 tokens/s	64–128+	~6 TB to 12 TB	Several million to over 10 million RMB	No

5. Common Misconceptions and Engineering Advice

Misconception 1: Looking only at activated parameters and ignoring total parameters. Routing activates only a few experts, but storage must cover all experts. VRAM planning must be based on total parameters.

Misconception 2: Treating bare weights as the full VRAM requirement. Actual deployment also needs activations, KV cache, communication buffers, and runtime workspace. Reserve at least 40% headroom.

Misconception 3: Ignoring the communication domain. Stacking more GPUs without considering interconnect topology will not translate into effective throughput. The core of supernode design is keeping expert scheduling on the shortest path.

Misconception 4: Assuming quantization is free. Quantization reduces VRAM usage but may affect model accuracy and routing stability. If the model was not trained with quantization awareness, aggressive post-hoc compression requires additional validation.

Engineering advice: Before committing to a purchase, complete weight volume estimation, request concurrency modeling, maximum batch size determination, and latency budget modeling. Validate single-layer inference performance on a small number of devices first, then extrapolate to the full model configuration. Without an in-house team and data center facilities, short-term validation on cloud high-performance compute instances is often the more prudent choice.