Trillion-Parameter MoE Deployment: Hardware Requirements and Cost Estimates

    A hypothetical 2.8-trillion-parameter mixture-of-experts model illustrates the memory, bandwidth, and interconnect requirements of on-premises inference. This article compares three deployment tiers—from basic CPU–GPU offloading to 10 concurrent streams targeting 25 tokens per second each—and explains how to validate quantization quality, hardware sizing, and costs before committing to a deployment.

    Trillion-Parameter MoE Deployment: Hardware Requirements and Cost Estimates

    1. Scope and Assumptions

    As the open-source community releases increasingly large model weights, more teams are asking whether a trillion-parameter mixture-of-experts (MoE) model can run on local infrastructure.

    This article uses a hypothetical model with the following specifications:

    • Approximately 2.8 trillion total parameters
    • An MoE architecture with roughly 900 experts
    • 16 experts activated per token
    • An official quantization-aware training scheme using 4-bit weights and 8-bit activations

    Until the model weights and technical report are available, all figures for throughput, memory requirements, and hardware configurations remain engineering estimates based on the assumed architecture. They must be verified through benchmarking after release.

    The purpose here is to establish a capacity-planning method, not a final procurement list.

    2. Calculate the Memory Floor First

    A common mistake in MoE deployment planning is estimating memory requirements from the number of active parameters.

    Sparse activation reduces computation, but it does not eliminate the need to store inactive experts. All expert weights must remain in addressable storage so that inference can access whichever experts the router selects. That storage does not necessarily have to be GPU memory.

    For 4-bit weights, the lower-bound calculation is:

    • Raw weights: 2.8 trillion parameters × 4 bits ÷ 8 = approximately 1,400 GB.
    • Quantization overhead: If every 32 four-bit elements share one 8-bit scale factor, the effective storage becomes 4.25 bits per element, or approximately 1,487.5 GB.
    • Additional requirements: Nonquantized tensors, metadata, alignment overhead, runtime workspaces, and the KV cache are not included.

    The practical conclusion is straightforward: storing the complete model at 4-bit precision requires approximately 1.5 TB before runtime overhead.

    For comparison, MoE models with a little over 600 billion parameters can require roughly 400 GB at 4-bit precision and run across eight accelerators with 80 GB of memory each. The hypothetical model discussed here is more than four times larger by parameter count.

    Communication Is Another Hard Constraint

    Large-scale MoE inference is highly sensitive to all-to-all communication performance.

    When experts are distributed across accelerators, token activations must travel to the devices hosting the selected experts and return after computation. Slow communication can consume much of the time saved by sparse computation.

    Deployments at this scale therefore often call for tightly interconnected groups of dozens of accelerators, keeping expert communication within a single high-speed communication domain wherever possible.

    3. Tier 1: Make It Run, Regardless of Speed

    The first objective is simply to load approximately 1.5 TB of weights into addressable memory and generate tokens—even if output speed is only a few tokens per second.

    A direct approach is heterogeneous CPU–GPU inference:

    • Keep the shared attention components on the GPU, where compute-intensive operations benefit from high-bandwidth device memory.
    • Offload the expert weights, which dominate storage, to system RAM.
    • Execute the sparsely activated experts through a large-memory CPU server.

    Reference Configuration

    • One consumer or entry-level data-center GPU with 24–80 GB of memory for attention and routing
    • A large-memory server with approximately 1.5–2 TB of DDR5 RAM for expert weights
    • NVMe SSD storage for weight files and swap buffering

    The lower end of the RAM range leaves very little room beyond the estimated weight footprint, so runtime memory requirements must be checked carefully.

    Expected Performance

    System memory bandwidth is the main bottleneck.

    A dual-socket server with fully populated memory channels may offer approximately 600–800 GB/s of aggregate bandwidth. Under the article’s rough workload assumption, each generated token requires reading weights for the activated experts across the model, amounting to data on the order of 8 GB.

    Allowing for practical execution overhead, the planning estimate is a few to roughly a dozen tokens per second. Actual performance depends on the model implementation and must be benchmarked.

    This tier is best suited to:

    • Offline batch processing
    • Data annotation
    • Model-quality evaluation

    Estimated Cost

    The source estimates are denominated in Chinese yuan and should be treated as regional budget references, not international price quotes:

    • GPU: approximately CNY 10,000–30,000
    • Complete server with 2 TB of RAM: approximately CNY 80,000–150,000
    • Total investment: approximately CNY 100,000–180,000

    4. Tier 2: Target 25 Tokens per Second for One User

    A generation rate of 25 tokens per second allows approximately 40 milliseconds per token for weight access and computation.

    At this target, relying entirely on CPU-resident experts is unlikely to be sufficient. More of the expert workload needs to move into GPU memory.

    There are two main approaches.

    Option 1: Place Frequently Used Experts on GPUs

    Profile the routing distribution and keep frequently activated experts in GPU memory. Leave less frequently selected experts in system RAM.

    For example, eight 80 GB accelerators provide a combined 640 GB of device memory for attention components and frequently used experts. The remaining experts continue to run through the CPU-memory path.

    Performance depends heavily on the GPU-resident expert hit rate. The planning estimate is 20–40 tokens per second for a single stream after tuning, rather than a guaranteed throughput range.

    Option 2: Quantize Expert Weights More Aggressively

    Reducing expert weights to 2–3 bits may make a fully GPU-resident deployment possible, provided the resulting quality remains acceptable.

    For example, 12–16 accelerators with 80 GB each provide approximately 1–1.3 TB of aggregate GPU memory. Quantization near 2 bits could allow the complete model to fit, subject to format overhead and runtime memory requirements.

    Running inference entirely on GPUs could provide more headroom for a 25-token-per-second target. However, the quality impact of aggressive quantization must be evaluated against the actual workload.

    Estimated Cost

    • Eight-accelerator server with 80 GB per device: approximately CNY 1–2 million
    • A 16-accelerator configuration: roughly twice that amount
    • Lower-cost configurations using consumer GPUs and PCIe interconnects: approximately CNY 300,000–600,000

    Consumer-GPU configurations introduce a major trade-off: interconnect bandwidth may become the dominant bottleneck, and the reduced memory budget may require more aggressive quantization. They should not be assumed to match the performance of tightly interconnected data-center accelerators.

    5. Tier 3: Ten Concurrent Streams at 25 Tokens per Second Each

    Concurrency changes the problem from simple bandwidth provisioning to a combination of bandwidth, aggregate throughput, and scheduling.

    KV Cache Capacity

    KV cache requirements grow approximately linearly with concurrent requests at a fixed context length.

    With 10 concurrent streams and long contexts, the cache may consume tens to more than 100 GB of additional GPU memory. This capacity must be reserved rather than allocated to weights.

    Batching Efficiency

    Concurrency can improve expert-weight reuse.

    If tokens in the same batch route to the same expert, the expert weights can be reused across those tokens rather than read independently for every request. A continuous-batching scheduler is needed to exploit this opportunity effectively.

    Interconnect Requirements

    All-to-all traffic grows with batch size. At this tier, a high-speed interconnect becomes a requirement rather than an optional performance enhancement.

    A planning guideline is to keep at least eight tightly interconnected accelerators within one communication domain. Where possible, use cross-node communication for pipeline partitions rather than frequent expert-routing exchanges.

    Reference Configuration and Capacity Check

    The proposed reference configuration is:

    • Two servers, each with eight 80 GB accelerators
    • Approximately 1.28 TB of aggregate GPU memory
    • Expert weights quantized to 3–4 bits
    • Tensor parallelism for attention components
    • Expert parallelism for expert computation
    • At least 150 GB reserved for the KV cache
    • Continuous batching and expert load balancing in the inference framework

    The memory budget needs an important qualification: 1.28 TB cannot hold the approximately 1.5 TB 4-bit weight footprint, even before reserving KV cache space. This configuration therefore depends on quantization toward the lower end of the proposed range and careful accounting for all overhead. It is not a fully GPU-resident 4-bit configuration as stated.

    With an implementation that fits within the memory budget and an appropriate communication topology, 10 concurrent streams at 25 tokens per second each is a target to validate—not an assured outcome.

    Estimated Cost

    • Two eight-accelerator servers: approximately CNY 2.5–4 million
    • Including rack infrastructure, power, cooling, and networking: approximately CNY 3–5 million

    These remain preliminary budget estimates rather than validated price-to-performance figures.

    6. Establish a Quantization Quality Floor

    A deployment should not reduce precision below an acceptable quality threshold simply to make the model fit.

    Use the following validation process:

    1. Establish the baseline. Evaluate the official 4-bit quantization-aware-trained weights on workload-specific benchmarks.
    2. Test each lower precision level. Compare 3-bit and 2-bit variants against the baseline using both perplexity and application metrics. Define an acceptable degradation threshold, such as no more than a 1–2 percentage-point decline in key metrics.
    3. Preserve sensitive components. Keep attention and routing components at higher precision where possible, concentrating aggressive quantization on expert weights.
    4. Load-test realistic conditions. Use actual concurrency levels and context lengths. Evaluate time to first token separately from steady-state generation throughput.

    A configuration that fits in memory but fails quality or latency requirements is not a viable deployment.

    7. On-Premises or Cloud?

    The economics differ across the three tiers.

    Tier 1 is a reasonable candidate for fully self-hosted infrastructure when low throughput is acceptable.

    Tier 2 may justify on-premises investment if the quantization strategy is mature and utilization is sufficient.

    Tier 3 requires a multimillion-yuan capital commitment, along with ongoing work to handle model updates, inference-framework changes, and infrastructure operations.

    For workloads with variable concurrency, on-demand GPU cloud capacity may offer a better total cost of ownership. A cloud deployment can combine:

    • Elastic GPU capacity
    • Object storage for model weights
    • Container orchestration for inference services
    • Monitoring and alerting to support service-level objectives

    General-purpose GPU and inference services from providers, including Baidu AI Cloud where applicable, can serve as comparison points for capacity planning. Service availability and suitability must be checked for the intended deployment region.

    8. Key Takeaways

    The central planning rule is:

    Size weight storage by total parameters, estimate computation from active parameters, and budget for the required concurrency.

    For the hypothetical 2.8-trillion-parameter model:

    • Basic execution: Approximately 1.5 TB of weight storage creates an entry-level hardware budget on the order of CNY 100,000, using CPU–GPU offloading.
    • Single-stream generation at 25 tokens per second: Substantial GPU-resident capacity or aggressive quantization raises the budget from several hundred thousand yuan to around CNY 2 million, with larger configurations costing more.
    • Ten concurrent high-throughput streams: A tightly interconnected multi-accelerator cluster moves the budget into the multimillion-yuan range.

    Memory capacity, quantization overhead, KV cache allocation, and communication topology must all be validated together. Prepare the deployment tiers before the weights are released, then replace every preliminary assumption with benchmark results before purchasing production hardware.