Deploying a 2.8T-Parameter MoE Locally: How to Estimate VRAM, Bandwidth, and Cost

    This article examines three deployment targets for a 2.8-trillion-parameter MoE model—functional validation, single-user 25 tokens per second (TPS), and 10 concurrent users at 25 TPS—and explains how total parameter count, quantization, memory bandwidth, KV cache, interconnect, and redundancy affect hardware sizing and cost. It provides calculation methods, example estimates, risk notes, and a PoC checklist. All figures are planning estimates and must be replaced with the official model configuration and current quotes before procurement.

    Conclusions First

    The three targets map to three hardware paths. They cannot share a single configuration.

    • Functional validation: One large-memory server plus one GPU is enough to get the model running. Throughput is roughly several tokens per second, and the upper bound is determined by memory bandwidth. Estimated cost: about RMB 300,000–600,000.
    • Single-user 25 TPS: The CPU-offload path is not viable. All weights must reside in GPU memory, about 1.8TB. A recommended configuration is 2 nodes, each with 8 192GB-class GPUs. Estimated cost: about RMB 8–15 million, excluding network and data center costs.
    • 10 concurrent users at 25 TPS each: On top of the previous tier, reserve about 2.1TB of VRAM for KV cache and meet an aggregate throughput of about 250 tok/s. Start with a 2-node plan; if the PoC does not meet the target, expand to 4 nodes.

    All numbers above are planning estimates, not verified quotations. Before procurement, replace them with the official model configuration and actual quotes.

    1. Weight Size: Deployment Cost Is Driven by Total Parameters

    The source material gives 896 total experts and 16 activated per token, an activation ratio of about 1.79% (16/896). These numbers should be verified against the official model configuration.

    A low activation ratio does not mean low VRAM requirements. Routing may access any expert, so all expert weights must reside in some storage medium. The cost baseline is therefore the total parameter count, not the activated parameter count.

    Calculation basis:

    • 4-bit raw data: 2.8×10^12 × 0.5 bytes = 1400GB.
    • MXFP4 format: each 32 4-bit elements carry one 8-bit scale factor, equivalent to 4.25 bits per parameter. 2.8×10^12 × 4.25/8 ≈ 1487.5GB, or about 1.49TB.
    • Unit note: This article uses decimal units, where 1GB = 10^9 bytes. Converted to GiB, the same value is about 1385GiB.

    The original plan states that total VRAM should be no less than 1.6TB but does not break down the components. The table below gives one estimated breakdown for a single-user, 32k-context scenario:

    Component Estimate Notes
    Quantized weights (MXFP4) About 1488GB Calculated from total parameters
    Non-quantized tensors (embedding, norm, etc.) About 30–60GB Count item by item from official weight files
    KV Cache About 8GB See the formula in Section 4
    Working memory and framework overhead About 50–100GB Depends on framework and GPU count; measure in practice
    Subtotal About 1.6TB No redundancy
    Redundancy (about 10%) About 160GB For fragmentation and peaks
    Total About 1.8TB Recommended single-user lower bound

    Therefore, the original 1.6TB figure is a lower bound without redundancy. The numbers for “all weights resident in GPU memory” and “CPU offload” cannot be mixed.

    2. Tier 1: Functional Validation (CPU Offload)

    In this design, the GPU handles attention layers and non-expert parts, while expert feed-forward network weights remain in server memory and are computed by the CPU. For each token, activations pass between CPU and GPU over PCIe; expert weights are not moved to the GPU token by token. The speed bottleneck is therefore CPU memory bandwidth—the amount of expert weight data that must be read per token.

    Bandwidth estimate

    • DDR5-4800 provides 38.4GB/s theoretical bandwidth per channel (4800MT/s × 8 bytes).
    • A single CPU with 8 channels has a theoretical value of about 307GB/s; a dual-socket system has a theoretical value of about 614GB/s.
    • The original text uses 90GB/s but does not give a source. That value is about 30% of the single-socket theoretical value. This article treats it as a parameter to be measured, not as a conclusion. Measure it with memory bandwidth tools such as STREAM, and record channel count, DIMM frequency, and NUMA settings.

    Per-token read volume

    • Activated parameters are about 2.8 trillion × 1.79% ≈ 50B (estimated from the expert parameter share, ignoring small items such as attention layers).
    • Under MXFP4, each token reads about 50B × 0.53125 bytes ≈ 26.6GB.

    Speed upper bound

    The speed upper bound is approximately effective bandwidth divided by 26.6GB.

    • Effective bandwidth 90GB/s: about 3.4 tok/s.
    • Effective bandwidth about 215GB/s (70% of the single-socket theoretical value): about 8 tok/s.

    PCIe is usually not the primary bottleneck because the hidden state passed per layer is small. Actual throughput still needs to be measured.

    Configuration checklist

    • Dual-socket server with 2TB or more of DDR5 ECC memory (weights are 1.5TB, so leave headroom).
    • One 24–48GB GPU for attention layers.
    • At least 4TB of NVMe SSD storage for model files.

    3. Tier 2: Single-User 25 TPS

    Check feasibility first

    25 TPS requires each token to complete in about 40ms. The CPU-offload upper bound is at most about 8 tok/s, far below the target. To reach 25 TPS, a single user reading 26.6GB of expert weights per token would require about 665GB/s of effective bandwidth, exceeding the dual-socket DDR5 theoretical peak of 614GB/s. Therefore, the CPU-offload path cannot meet this target.

    All weights resident in GPU memory

    • Total VRAM should be no less than 1.8TB; a recommended configuration is about 3TB, i.e., 2 nodes with 8 192GB-class GPUs each.
    • Use high-speed intra-node interconnect (such as NVLink-class) and InfiniBand or RoCE between nodes, with bandwidth no less than 200Gbps.
    • Expert parallelism or tensor parallelism distributes weights across 16 GPUs. Each GPU reads about 1.7GB of expert weights per token. Based on per-GPU memory bandwidth, the read time is far below 40ms. The real bottleneck is often inter-GPU communication, especially cross-node all-to-all communication.

    Whether 25 TPS can be achieved must be verified by testing, not judged from estimates alone.

    Cost estimate

    • Two 8-GPU high-end servers: about RMB 8–15 million.
    • High-speed switches, cables, racks, power, and cooling: additional, depending on data center conditions.

    4. Tier 3: 10 Concurrent Users at 25 TPS Each

    KV cache estimation formula

    KV cache per token = 2 × layers × KV heads × head dimension × bytes per element.

    The following are example parameters and should be replaced with the real architecture of the official model: 60 layers, 8 KV heads, head dimension 128, BF16 (2 bytes).

    • Per-token usage: 2 × 60 × 8 × 128 × 2 ≈ 0.25MB.
    • 10 users with 128k context each: 1.28M tokens × 0.25MB ≈ 320GB.

    Total VRAM

    Weights 1488GB + non-quantized tensors about 45GB + KV cache about 320GB + working memory about 75GB = about 1.93TB. With 10% redundancy, this is about 2.1TB.

    Two nodes with 16 192GB-class GPUs provide about 3TB in total, leaving VRAM headroom.

    Throughput requirement

    10 users × 25 TPS each gives an aggregate throughput of about 250 tok/s. This figure needs PoC validation. If the 2-node configuration cannot meet it, expand to 3–4 nodes, at an estimated cost of about RMB 16–30 million.

    5. Risk Notes

    • Insufficient memory bandwidth: The speed ceiling of the CPU-offload design is directly determined by bandwidth. Do not promise 25 TPS before measuring it.
    • Insufficient VRAM: Weights, non-quantized tensors, KV cache, and redundancy must all be counted. Buying only by weight size will not fit the actual deployment.
    • Concurrency scales KV cache: After multiplying context length by concurrency, KV cache may exceed the weights themselves.
    • Communication bottleneck: In multi-node deployments, inter-GPU communication often becomes the limit before compute does.
    • Cost volatility: Memory and GPU prices fluctuate significantly. Quotes should state the date, currency, tax status, and whether they cover a complete system or individual parts.
    • Total cost of ownership: Power, cooling, operations, and inference framework engineering effort must be counted separately. A single fully loaded 8-GPU high-end server can draw from several kilowatts to over 10kW.

    6. Environment Record and PoC Checklist

    Before procurement, complete a PoC and record the following items so the plan can be reproduced.

    • Inference framework name and version, launch parameters, and quantization format (MXFP4).
    • CPU model, memory capacity and channel count, DIMM frequency, and NUMA configuration.
    • GPU model, per-GPU VRAM, node count, inter-GPU interconnect type, and inter-node network bandwidth.
    • PCIe generation and lane count.
    • Operating system, driver, and CUDA version.
    • Acceptance metrics: peak VRAM usage, effective memory bandwidth, first-token latency, single-user tok/s, aggregate tok/s at 10 concurrent users, and fallback plan.

    Sign the procurement contract only after the PoC confirms the design.