Deploying a Trillion-Parameter MoE Model Locally: VRAM, Hardware, and Cost Guide
This article provides a practical framework for estimating the memory, hardware, and cost requirements of running a 2.8-trillion-parameter mixture-of-experts (MoE) model with 896 experts and 16 active experts per token on local infrastructure. It explains why a naive VRAM division underestimates real needs, covers weight storage, runtime workspace, KV cache, communication overhead, and presents three performance tiers with indicative hardware and cost ranges. The guide also discusses quantization considerations, total cost of ownership, and a validation checklist for post-release benchmarking.
Why Simple VRAM Division Is Misleading
When estimating the cost of deploying a trillion-parameter MoE model locally, the most common mistake is to perform a single VRAM division: multiply total parameters by quantization bit width, then divide by per-GPU memory. That calculation only yields a theoretical lower bound. It fails to account for runtime workspace, KV cache, metadata, fragmentation, and the communication overhead introduced by expert parallelism in MoE architectures.
This article uses a publicly known architecture with 2.8 trillion parameters, 896 experts, and 16 active experts per token as a case study to build a reusable estimation framework. Note: before the full weights and technical report are released, all speed, VRAM, and hardware figures below are estimates based on public information. Once weights are available, every estimate must be validated through benchmarking.
Step 1: Build the VRAM Ledger — Total Parameters Drive Storage, Activation Ratio Drives Compute
For MoE architectures, VRAM usage must be calculated using the total parameter count, not the activated parameter count. Among the 896 experts, only 16 are activated per token — an activation ratio of about 1.79%. While this reduces compute during inference, it does not reduce weight storage. All experts must be loaded into storage or VRAM before inference; routing then selects 16 experts per token. Therefore, the first line item in the VRAM ledger is the full set of expert weights.
Using 4-bit weight quantization, the lower-bound raw data size for 2.8 trillion parameters is approximately:
2.8e12 × 4 bit ÷ 8 = 1.4e12 Byte ≈ 1400 GB
But 4-bit weights typically also include scaling factors. Assuming one 8-bit scale per 32 4-bit elements, the average is about 4.25 bits per element:
def weight_gb(total_params, weight_bits=4, scale_bits=8, elems_per_scale=32):bits_per_elem = weight_bits + scale_bits / elems_per_scalereturn total_params * bits_per_elem / 8 / 1e9print(weight_gb(2.8e12)) # ~1487.5 GB, decimal
This 1487.5 GB still does not account for non-4-bit tensors, metadata, alignment, runtime workspace, KV cache, VRAM fragmentation, or multiple replicas. In practice, it is wise to reserve an additional 15–40% of runtime headroom on top of the raw weight size. If long contexts, high concurrency, or multiple KV cache replicas are needed, the headroom must be increased further. In other words, a full 4-bit deployment should target roughly 2.0–2.5 TB of VRAM, not the bare 1.5 TB figure.
For scale comparison: a MoE model of roughly 700 billion parameters needs about 685 GB with 8-bit weights, 335 GB with 4-bit weights, and approximately 400 GB total when activations and workspace are included. This can run on 8 × 80 GB GPUs (640 GB total). The 2.8-trillion-parameter model is roughly four times larger, which sets the lower bound for a full deployment. With 80 GB GPUs, the raw weights alone require about 19 cards. Factoring in expert parallelism, communication domains, KV cache, and runtime headroom, realistic configurations often involve 32–64 GPUs or more. Never treat the result of a simple VRAM division as a runnable configuration.
Step 2: Three Performance Tiers and Corresponding Hardware and Cost
Below are three target tiers. Cost figures are illustrative assumptions, not quotes or measurements. They are based on roughly 30,000–60,000 RMB per 80 GB accelerator, a system and network/storage multiplier of 1.5–2.0, electricity at 0.8 RMB/kWh, a PUE of 1.3, and annual operations at 10–20% of hardware cost.
Tier 1: Just Run It, Without Caring About Speed
The goal is to completely load the weights and finish inference, allowing low throughput, longer queues, and limited concurrency. The bottleneck here is still VRAM capacity and expert-parallel communication. A recommended setup is 64 × 80 GB accelerators forming a single large communication domain, with about 5 TB total VRAM, 10–20 TB local NVMe storage, 1–2 TB RAM, and a high-speed interconnect. If you attempt CPU offloading or use low-bandwidth interconnects, all-to-all communication will severely slow down inference. The model may “run,” but interactive experience will be unpredictable.
Cost estimate: accelerator hardware alone is roughly 1.92–3.84 million RMB. Adding servers, switching fabric, storage, and facility infrastructure brings one-time hardware costs to approximately 3–9 million RMB. Annual electricity, assuming 60 kW total system power and PUE 1.3, is about 550,000 RMB. Adding maintenance, spare parts, and bandwidth, annual operating cost is roughly 0.5–1.5 million RMB.
Tier 2: 25 tokens/s for Single-User Output
25 tokens/s for a single user does not demand high throughput, but it is sensitive to time-to-first-token and all-to-all communication latency. The routing traffic across 896 experts must be kept within a single high-speed communication domain, reducing cross-switch hops, while preserving sufficient KV cache space. Configurations can range from 64 to 128 × 80 GB GPUs, depending on interconnect bandwidth, expert parallelism strategy, and batching implementation.
Cost estimate: one-time hardware costs are roughly 6–20 million RMB, with annual operations around 1–3 million RMB. At this tier, do not focus only on GPU count; network topology and communication overhead matter. If communication overhead is high, adding GPUs may not linearly improve tokens per second.
Tier 3: 10 Concurrent Users, Each Above 25 tokens/s, Quantization No Lower Than 4-bit
The total output requirement is about 250 tokens/s, while maintaining KV cache for 10 concurrent streams and continuous batching. Compute, memory bandwidth, and communication domains must all be scaled up. A recommended setup is 128–256 × 80 GB GPUs, with 10–20 TB total VRAM, higher-bandwidth switching, redundant power, and independent storage. If quantization stays at or above 4-bit, you must verify the behavior of non-4-bit tensors, routing precision, and KV cache precision. Avoid the trap of quantizing only weights and then observing accuracy or stability degradation.
Cost estimate: one-time hardware costs are roughly 15–60 million RMB, with annual operations around 3–10 million RMB. The wide range reflects how sensitive large-scale MoE inference is to network bandwidth, topology, and scheduling efficiency. Actual benchmark results are required to converge on a specific configuration.
Step 3: Total Cost of Ownership Goes Beyond Accelerators
Local deployment costs should be split into one-time capital expenditure and annual operating cost:
One-time CAPEX ≈ accelerator_count × unit_price × multiplier + network + storage + facility modificationsAnnual OPEX ≈ system_power × PUE × electricity_price × 8760 + operations staff + spare parts + bandwidth/storage
Beyond accelerators, high-speed switching fabric, optical cables, NICs, storage, racks, power distribution, and cooling often account for 30–50% of the total investment. In large-scale MoE inference, 896 experts are spread across many GPUs. After each token’s routing decision, activations must be moved to the GPUs holding the target experts, computed, and then moved back. This communication latency directly erodes compute efficiency. High-speed interconnect is not optional; it is part of the VRAM ledger.
Step 4: Quantization Levels and Validation Checklist
4-bit weights with 8-bit activations is a common scheme, but “no lower than 4-bit” does not mean every tensor is 4-bit. Before deployment, confirm: the proportion of non-4-bit tensors, scaling factor overhead, KV cache precision, routing network precision, and long-context stability. After the weights are released, at least verify the following metrics:
- Peak VRAM usage: after loading, at first token, with long context, and under 10 concurrent users;
- Output speed: do both 25 tokens/s for a single user and 25 tokens/s per user under 10 concurrent users hold?
- Time-to-first-token and tail latency;
- All-to-all communication ratio and network utilization;
- Power draw, temperature, and PUE;
- Task metrics, perplexity, and long-context consistency.
Conclusion
The cost of running a trillion-parameter MoE model locally is jointly determined by the total-parameter VRAM ledger, communication domains, and concurrent KV cache requirements. Under the assumptions in this article, Tier 1 one-time hardware costs are approximately RMB 3–9 million, Tier 2 RMB 6–20 million, and Tier 3 RMB 15–60 million. Annual operating costs range from several hundred thousand to tens of millions of RMB. For short-term validation or elastic scaling, public cloud GPU instances, container platforms, and object storage can reduce upfront investment. However, for a long-term full-fidelity self-hosted deployment, you must use benchmark tests to verify real tokens per second, peak VRAM, and communication efficiency. Before weights are released, perform sensitivity analyses using the formulas. After release, replace every estimate with measured data.
