Local Deployment of Trillion-Parameter MoE Models: VRAM and GPU Estimation

    Deploying a trillion-parameter Mixture of Experts (MoE) model requires precise VRAM and GPU calculations, primarily based on total parameters rather than active ones. This article provides a comprehensive guide to estimating VRAM budgets, determining the number of GPUs needed for various deployment scenarios (from basic to high concurrency), evaluating communication overhead and interconnect requirements, and comparing local versus cloud deployment costs and architectures.

    The first step in deploying a trillion-parameter MoE model is calculating the VRAM, and it must be based on the total parameter count. The example model in this article has 2.8 trillion total parameters, 896 expert networks, and activates 16 experts per token (an activation ratio of roughly 1.79%). While this ratio reduces the computation and communication volume per token, it does not reduce the resident VRAM required for weights.

    1. Why VRAM is Calculated Based on Total Parameters

    During inference, the router decides which 16 experts to use for each token, and it cannot predict which experts the next token will hit in advance. Therefore, the weights of all 896 experts must reside in VRAM. The activation ratio affects computation and cross-GPU communication but does not decrease the weight footprint.

    2. Components of the VRAM Budget

    A pre-deployment VRAM budget typically includes the following:

    1. Model Weights: Calculated as Parameters × Bits per Parameter ÷ 8.
    2. Scaling Factors and Non-Quantized Tensors: The MXFP4 format includes an 8-bit scaling factor for every 32 4-bit elements, meaning each parameter effectively occupies 4.25 bits. Embedding, normalization layers, and routing parameters usually retain higher precision, depending on the model configuration.
    3. Runtime Overhead: Inference framework, CUDA context, operator workspaces, and VRAM alignment padding.
    4. KV Cache: Grows with context length and concurrent request count, calculated as follows:

    KV Cache ≈ 2 × Number of Layers × Number of KV Heads × Head Dimension × Bytes per Element × Context Length × Concurrent Requests

    Structures like Delta Attention change KV storage. When calculating, use the actual number of layers and heads from the model configuration file rather than default Transformer values.

    For the example model, weights are calculated as: 2.8 trillion × 4.25 bits ÷ 8 ≈ 1487.5 GB. As an initial planning budget, use 1.2 to 1.3 times the weight size and adjust based on actual measurements.

    3. Using a 671B Model as a Reference

    Another open-source model in the industry has roughly 671B total parameters and serves as a good benchmark.

    • FP8 Precision: Weights are about 671GB; the total with main and prediction parameters mentioned is around 685GB.
    • 4-bit Precision: Weights are about 335GB, activations about 37GB, totaling about 372GB. 8 80GB GPUs provide 640GB total, leaving roughly 268GB for short-context, low-concurrency scenarios. For long contexts or high concurrency, the KV Cache must be recalculated.

    The 4-bit weights of the example model are 1487.5GB, which is about 4.4 times the 4-bit weight footprint of the 671B model.

    4. GPU Count Derivation

    The following calculations assume 75GB of usable VRAM per GPU (the actual value after deducting driver and framework overhead from 80GB, which must be tested on target hardware).

    Basic Plan: 4-bit Quantization, Just to Get It Running

    • Weights: 1487.5GB × 1.25 ≈ 1859GB. At 75GB per card, this requires at least 25 GPUs.
    • 24 GPUs could theoretically fit it, but 896 ÷ 24 ≈ 37.3, meaning experts cannot be evenly distributed, leading to load imbalance.
    • 32 GPUs (4 servers with 8 GPUs each) are recommended. 896 ÷ 32 = 28 experts per GPU, requiring about 46.5GB, leaving the remaining space for KV Cache and workspaces.

    Personal Smooth Experience Plan: Single User at 25 TPS

    • You can start with 32 GPUs, but you must measure the single-user latency in practice, not just rely on VRAM estimates.
    • With 32 GPUs spread across 4 servers, inter-expert communication will inevitably cross servers, which is the main performance constraint.

    High Concurrency Plan: FP8 Quantization, 10 Concurrent Streams, 25+ TPS per Stream

    • FP8 weights are about 2800GB, or about 2888GB including scaling factors.
    • 32 GPUs provide 2560GB total, which cannot hold these weights. By total VRAM, at least 37 GPUs are needed, but this leaves no room for KV Cache.
    • 64 GPUs (8 servers with 8 GPUs each) are recommended. Each GPU holds about 45.1GB of weights. After deducting 5 to 8GB of runtime overhead, each GPU has about 27 to 30GB left, providing about 1700 to 1900GB across the cluster for KV Cache. With 10 concurrent streams, each stream gets about 170 to 190GB.

    All plans must confirm whether the hardware natively supports MXFP4 or FP8. Unsupported GPUs will require de-quantization, simultaneously increasing compute time and VRAM usage.

    5. Communication and Interconnect

    Every inference pass in an MoE layer involves two All-to-All communications: one to distribute tokens to the GPUs hosting the experts, and one to retrieve the results.

    The following is a hypothetical example; actual values depend on model config and testing: The model has 60 layers, a hidden dimension of 7168, and activations are transmitted in FP8. Under this assumption, each token sends about 7168 × 16 ≈ 115KB of data per layer, and receives the same amount. Bandwidth is rarely the bottleneck; the latency and synchronization overhead of individual communications are the main constraints.

    At 25 TPS, the time budget per token is 40ms. Assuming compute takes about 15ms, the remaining 25ms must be distributed across 120 All-to-All communications, leaving a budget of about 200 microseconds per communication and sync. Cross-server InfiniBand latency under load can reach hundreds of microseconds, so this must be measured on the target cluster.

    A super-node implies placing 64 or more accelerators into a unified high-speed interconnect domain (e.g., NVLink or equivalent), allowing expert data transfers to bypass slow networks. During deployment, you can also co-locate frequently co-activated experts on the same server based on routing statistics to reduce cross-node communication.

    6. Cost Magnitude

    Hardware costs are estimated at 2 to 3 million RMB (approx. $280K to $420K USD) per 8-GPU server. Market prices fluctuate, so actual procurement should be based on quotes.

    • 32 GPUs (4 servers): Hardware costs about 8 to 12 million RMB.
    • 64 GPUs (8 servers): Hardware costs about 16 to 24 million RMB.

    This excludes InfiniBand switches, optical modules, cables, storage, racks, data center upgrades, and spare parts.

    Power consumption is estimated at 70kW per server at full load, totaling about 613,000 kWh per year. At 0.8 RMB per kWh, annual electricity costs are about 490,000 RMB; if the data center PUE is 1.3, electricity costs are about 640,000 RMB.

    With a 3-year depreciation cycle, the annual hardware depreciation for 64 GPUs is about 5.33 to 8 million RMB. Operations staff, cooling, data center load-bearing capacity, and driver/framework upgrades must also be included in the annual budget.

    7. Local Deployment vs. Cloud Services

    Dimension On-Premises Bare Metal Self-Managed Cloud GPU Instance Managed Inference Service
    Initial Investment High, requires upfront server and network procurement Low, billed hourly Lowest, billed per invocation or usage
    Interconnect Capability Designed by purchaser, can build super-nodes Depends on instance type; verify if high-speed interconnect is provided Decided by provider, details often undisclosed
    Operations Responsibility Fully borne by the user Model and inference framework deployment managed by user Mostly handled by the provider
    Data and Compliance Data stays on-premises Depends on region and contract terms Depends on provider’s data processing terms
    Elastic Scaling Weak, long scaling cycles Strong Strong
    Applicable Scenarios Long-term high load, data sovereignty required Phased validation, medium load Rapid business launch, high load volatility

    When selecting a cloud service, it is recommended to request the following information from at least two providers: GPU model and VRAM capacity, intra- and inter-node interconnect bandwidth, pricing and billing methods, SLA commitments, and data processing terms. Specific SLA terms are subject to the contract.

    8. Frequently Asked Questions

    How much VRAM is needed to locally deploy a trillion-parameter MoE model?
    4-bit weights are about 1488GB, and FP8 weights are about 2888GB. The budget should start at 1.2 to 1.3 times the weight size, plus KV Cache depending on context length and concurrency.

    How many GPUs are needed?
    For 4-bit quantization and a single-user scenario, start with 32 GPUs. For high-concurrency FP8 scenarios, at least 37 GPUs are needed by weight calculation, but 64 GPUs are recommended to leave room for KV Cache.

    How is 25 TPS estimated?
    25 TPS means a single stream outputs 25 tokens per second, or 40ms per token. The sum of compute, communication, and scheduling overhead must be below this value. In testing, fix the input length (e.g., 2K) and output length (e.g., 512), divide the output token count by the generation time to get TPS, and measure the time to first token separately.

    Cloud or local: how to choose?
    For long-term high-load scenarios where data must remain on-premises, self-hosting is suitable. For scenarios with volatile loads or still in the validation phase, cloud services are more appropriate.