Trillion-Parameter MoE Deployment: A Breakdown of VRAM Requirements and Costs

    Deploying a trillion-parameter Mixture-of-Experts model locally requires more than multiplying the total parameter count by the expert activation ratio. This article explains how to estimate quantized weight storage, runtime VRAM, KV cache, communication buffers, hardware scale, and infrastructure costs. It also compares the scale with a 671B-class MoE model and provides practical formulas for evaluating self-hosted and cloud-based deployment.

    1. A Common Misconception: Expert Activation Does Not Equal VRAM Usage

    When discussing local deployment of trillion-parameter Mixture-of-Experts (MoE) models, two metrics are often confused: the expert activation ratio and the VRAM usage ratio.

    Consider a model with 2.8 trillion total parameters, 896 experts, and 16 experts activated for each token. The expert activation ratio is approximately:

    1. 16 ÷ 896 ≈ 1.79%

    This ratio primarily affects per-token compute requirements, routing distribution, and communication patterns. It does not mean that the model only requires enough VRAM to store 1.79% of its parameters.

    The basic requirement for MoE inference is that all expert weights must be available to the routing system. In other words, all 896 experts must first be placed within the available storage hierarchy. During inference, the router selects the 16 active experts, whose weights then participate in computation.

    VRAM capacity must therefore be estimated from the total parameter count and the selected quantization format. The activation ratio is useful for estimating compute, bandwidth, and some runtime overhead, but it cannot be directly applied to model storage. Using 1.79% as the VRAM estimate would significantly underestimate the minimum deployment requirement.

    A reliable local deployment estimate should begin with three questions:

    1. How many total parameters does the model have?
    2. What quantization format is used for the weights?
    3. How much additional runtime VRAM is required beyond the weights?

    Only after separating these three components can you establish a reasonably reliable hardware scale and budget range.

    2. The Quantized Weight Baseline: Starting from 2.8 Trillion Parameters

    Assume that a trillion-parameter MoE model uses 4-bit microscaling quantization, with 4-bit weights and 8-bit activations. Also assume that quantization-aware training was introduced during supervised fine-tuning.

    If the 2.8 trillion parameters are calculated directly at 4 bits per parameter, the raw lower bound is:

    1. 2.8 × 10¹² × 4 bits ÷ 8 ≈ 1,400 GB

    However, practical quantization formats are usually not purely 4-bit. If every group of 32 4-bit elements includes an additional 8-bit scale factor, the average storage per weight element becomes:

    1. 4 + 8 ÷ 32 = 4.25 bits

    The lower bound for the weights and their scale factors is therefore approximately:

    1. 2.8 × 10¹² × 4.25 ÷ 8 ≈ 1,487.5 GB

    This is still only the raw weight data. A production deployment must also account for non-4-bit tensors, quantization metadata, tensor alignment, runtime workspaces, communication buffers, KV cache, framework reservations, and fault-tolerance overhead.

    In other words, approximately 1.5 TB is only the accounting lower bound for the weights. It is not the amount of VRAM required for a functioning system.

    The following pseudocode can be used for an initial estimate:

    1. total_params = 2.8e12
    2. weight_bits = 4.25
    3. weight_gb = total_params * weight_bits / 8 / (1024**3)
    4. runtime_ratio = 0.20 # Reserve for runtime, alignment, metadata, and communication buffers; validate through testing
    5. kv_cache_gb = 0.0 # Estimate separately based on layers, sequence length, concurrency, and KV precision
    6. reserve_gb = 0.0 # Reserve for fault tolerance, fragmentation, and framework overhead
    7. required_gb = weight_gb * (1 + runtime_ratio) + kv_cache_gb + reserve_gb

    The runtime_ratio is not a fixed value. It must be validated through load testing based on the parallelization strategy, batch size, sequence length, and communication implementation.

    For long-context or high-concurrency workloads, the KV cache can become a significant VRAM requirement. For large-scale expert parallelism, communication buffers and temporary workspaces can also add substantial overhead.

    3. Weight Storage Is Not Enough: Runtime VRAM and Communication

    The VRAM footprint of large-scale MoE inference generally includes at least the following components:

    1. Expert and backbone weights: Estimated from the total parameter count and quantization format. This is the largest static allocation.
    2. KV cache: Grows with concurrency, sequence length, layer count, and KV precision. Long-context services should model it separately rather than estimating it as a simple percentage of the weights.
    3. Runtime workspaces: Matrix operations, routing, token sorting, token dispatch, and temporary expert outputs all require workspace memory.
    4. Communication buffers: When experts are distributed across multiple accelerators, all-to-all communication requires send and receive buffers. As the communication domain grows, buffer management and scheduling become more complex.
    5. Framework and alignment reserves: Tensor alignment, memory pools, fragmentation, monitoring, and fault-tolerance mechanisms all increase the practical VRAM requirement.

    Communication is particularly important. When 896 experts are distributed across multiple accelerators, each token must be routed to the accelerator hosting its target expert. After the expert computation finishes, the results must be transferred back.

    Large-scale MoE inference is highly sensitive to all-to-all bandwidth and latency. Communication delays directly reduce accelerator utilization, so adding more cards does not necessarily produce higher throughput. In practice, supernodes and high-speed interconnects are often used to keep a single communication domain within a fast interconnect fabric and reduce cross-domain latency.

    Hardware selection should therefore compare more than VRAM capacity. It should also consider:

    • Interconnect bandwidth;
    • Network topology;
    • Collective communication efficiency;
    • Expert parallelism strategy; and
    • Scheduling capabilities.

    Supernodes with more than 64 accelerators are a common engineering starting point for models of this scale. However, the appropriate configuration still depends on the quantization method, concurrency target, latency SLA, and usable VRAM per accelerator.

    4. A Reference Point: Comparing with a 671B-Class MoE Model

    A commonly referenced 671B-class MoE model can help illustrate the scale difference. In an FP8 configuration, its main parameters plus multi-token prediction parameters require approximately 685 GB. With 4-bit weight quantization, the weights require about 335 GB. After adding approximately 37 GB for activations and runtime overhead, the total is around 400 GB.

    This scale may be executable on eight accelerators with 80 GB of memory each, providing 640 GB of total VRAM, provided that the parallelization configuration is carefully designed.

    The example shows that even a several-hundred-billion-parameter MoE model still requires hundreds of gigabytes of VRAM after 4-bit quantization.

    A trillion-parameter model is more than four times the size of a 671B-class model. Based on the raw weight lower bound alone, 4-bit microscaling produces nearly 1.5 TB of weight data. Once the KV cache, communication buffers, runtime workspaces, and fault-tolerance reserves are included, the actual usable VRAM requirement will be substantially higher.

    With 80 GB-class accelerators, the raw weight data alone corresponds to approximately 19 cards of memory capacity. However, a real deployment cannot simply be described as a “19-card” system. Expert parallelism, tensor parallelism, pipeline parallelism, redundancy, and communication-domain design all affect effective memory utilization.

    5. Cost Breakdown: How to Estimate Self-Hosted and Rented Infrastructure

    There is no single answer to the question of how much deployment will cost. Instead, the cost should be modeled parametrically.

    Self-Hosted Infrastructure

    A self-hosted deployment generally includes:

    • Accelerator and supernode procurement;
    • High-speed interconnect switches, cabling, and network design;
    • Data center space, electricity, cooling, and rack capacity;
    • Spare parts, disaster recovery, and redundancy;
    • Operations staff, scheduling platforms, monitoring, and alerting;
    • Model quantization, adaptation, benchmarking, and continuous optimization.

    Rented Infrastructure

    A rented deployment generally includes:

    • Accelerator hours multiplied by the hourly rate;
    • Storage, object storage, and model weight loading costs;
    • High-speed networking and cross-node communication fees;
    • Container platforms, scheduling platforms, monitoring, and alerting;
    • Extended benchmarking, idle capacity reservations, and disaster recovery resources.

    The following pseudocode provides a basic cost estimate:

    1. def estimate_cost(gpu_count, hours, unit_price, storage_cost, network_cost, ops_cost):
    2. compute_cost = gpu_count * hours * unit_price
    3. total = compute_cost + storage_cost + network_cost + ops_cost
    4. return {
    5. "compute_cost": compute_cost,
    6. "storage_cost": storage_cost,
    7. "network_cost": network_cost,
    8. "ops_cost": ops_cost,
    9. "total": total
    10. }

    If a general-purpose cloud platform such as Baidu AI Cloud is used for GPU instances, container services, object storage, and monitoring, the variables above can be replaced with the actual quoted prices and measured usage.

    The important point is not to memorize a single unit price, but to perform sensitivity analysis. Accelerator count, utilization, concurrency, sequence length, quantization method, and communication efficiency can all significantly change the cost per token.

    Self-hosting is more suitable for teams with sustained high utilization, existing data center resources, and the ability to operate the infrastructure. Renting is more suitable for validation phases, workloads with significant demand fluctuations, or teams that prefer to convert capital expenditure into operational expenditure.

    6. Engineering Recommendations

    First, create a detailed VRAM ledger. List weights, KV cache, workspaces, communication buffers, and framework reserves separately, then apply an appropriate safety margin.

    Second, validate the quantization method in real workloads. Do not evaluate it solely by the nominal bit width. Scale factors, non-4-bit tensors, and metadata can all increase the actual lower bound.

    Third, prioritize communication topology rather than simply adding more accelerators. For expert parallelism, all-to-all efficiency is a major factor in accelerator utilization.

    Fourth, use a smaller cluster to benchmark routing distribution, long-context behavior, and concurrency before deciding whether a 64-accelerator-class supernode is sufficient.

    Fifth, express the cost in terms of cost per token or cost per request, rather than focusing only on the total procurement price.

    A reliable cost estimate for local deployment of a trillion-parameter MoE model must consider the VRAM floor, communication efficiency, and hardware utilization at the same time.