How Much Does It Cost to Deploy a 2.8 Trillion-Parameter MoE Model On-Premises?

    Deploying a 2.8 trillion-parameter Mixture-of-Experts model requires more than estimating the size of its model files. This article breaks down the capacity, communication, serving, power, and operations costs behind two deployment strategies: a full-scale accelerator cluster and a CPU–GPU hybrid architecture. It also provides practical sizing guidelines and a budgeting model for weights, KV cache, runtime overhead, and hardware.

    How Much Does It Cost to Deploy a 2.8 Trillion-Parameter MoE Model On-Premises?

    Estimating the cost of deploying a 2.8 trillion-parameter Mixture-of-Experts (MoE) model locally requires more than looking at the model file size or the number of experts activated during inference. The real budget is determined by four cost categories:

    • GPU memory and system memory for weights and runtime execution
    • Communication overhead caused by expert parallelism
    • Serving requirements driven by KV cache and concurrency
    • Total cost of ownership (TCO), including power and operations

    This article uses an MoE architecture with 2.8 trillion total parameters, 896 experts, and 16 experts activated per token as a technical example. It compares two deployment paths: a full-accelerator deployment and a CPU–GPU hybrid deployment.

    1. A Common Misconception: Expert Activation Does Not Equal Memory Usage

    This type of model has approximately 2.8 trillion total parameters. With 896 experts and 16 experts selected for each token, the expert activation ratio is:

    1. 16 / 896 ≈ 1.8%

    However, this figure describes the amount of computation performed for each token. It does not mean that only 1.8% of the model weights need to be stored in memory.

    In practical MoE inference, all 896 experts must remain available in a storage tier that can be accessed quickly. The router selects 16 experts for each token, and those experts participate in the computation. As a result, weight capacity must be estimated from the total parameter count. The activation ratio mainly affects compute requirements, memory bandwidth, and routing communication; it does not allow the weight capacity to be divided by the activation ratio.

    The deployment plan must therefore account for the basic reality that weights for all 2.8 trillion parameters need to be stored somewhere.

    MoE models of this type usually contain shared attention components as well as sparse expert components. The shared attention layers are compute-intensive and are well suited to high-bandwidth accelerators. The sparse experts are numerous, but only a small subset is activated for each token, making them the main capacity challenge. This difference is the basis for hybrid offloading strategies.

    2. The Weight Budget: 4-Bit Quantization Still Requires More Than 1,400 GB

    If 2.8 trillion parameters are stored at 4 bits per parameter, the theoretical raw data size is:

    1. 2.8T × 4 bits ÷ 8 ≈ 1,400 GB

    In practice, low-bit formats usually include block-level scaling factors. For example, with 4-bit microscaling floating-point weights and 8-bit activations, every 32 4-bit elements may require an additional 8-bit scale. Based on this type of low-bit format, the estimated size becomes:

    1. 2.8T × 4.25 bits ÷ 8 ≈ 1,487.5 GB

    This is still only the theoretical lower bound for the weight elements and scaling factors. It does not include:

    • Non-4-bit tensors
    • Metadata
    • Tensor alignment
    • Runtime workspaces
    • Communication buffers
    • Framework overhead
    • KV cache

    Some very large models use quantization-aware training to make 4-bit weights with 8-bit activations more stable during supervised fine-tuning. However, this does not eliminate runtime overhead.

    As a point of reference, a 671-billion-parameter MoE model may require approximately 685 GB for its main parameters and multi-token prediction parameters when stored in an 8-bit floating-point format. With 4-bit weight quantization, the weights may occupy about 335 GB, while activations and workspaces add approximately 37 GB. The total is around 400 GB, allowing the model to run on eight 80 GB-class accelerators with 640 GB of combined GPU memory.

    A 2.8 trillion-parameter model is more than four times larger. Its minimum capacity requirement follows directly from that difference.

    A practical planning formula is:

    1. Total GPU/system memory ≈
    2. weights + KV cache + activation workspace
    3. + communication buffers + metadata and alignment
    4. + safety margin

    A safety margin of 15% to 30% of the weight size is recommended. For approximately 1.49 TB of weights, the weights plus the safety margin alone can require close to 1.7–1.9 TB.

    3. Full-Scale Deployment: Why 64 or More Accelerators May Be Required

    For a full-accelerator deployment, capacity is the first constraint. With an 80 GB-class accelerator, the weights alone would require approximately 19–20 cards, leaving almost no room for KV cache or runtime workspaces.

    A low-concurrency validation setup may fit within roughly 24–32 accelerators. However, achieving better throughput and more stable latency generally requires 64 or more accelerators connected through a high-speed communication domain or a tightly integrated supernode.

    The main challenge is not raw compute capacity but communication.

    Large-scale MoE inference is highly sensitive to all-to-all communication. After the router selects experts for each token, the corresponding activations must be transferred to the accelerators hosting those experts. Once the expert computation is complete, the results must be transferred back. Communication latency can consume a significant portion of the available compute performance.

    The value of a high-speed interconnect domain is that it keeps expert traffic within a high-bandwidth communication fabric and reduces the bottlenecks associated with crossing between separate domains.

    The cost of a full-scale deployment includes:

    • The number and cost of accelerators
    • High-speed switches, network adapters, cables, and ports
    • Server CPUs, system memory, and local NVMe storage
    • Redundant power supplies, cooling, and racks
    • Power consumption and operational TCO
    • Inference frameworks, monitoring, alerting, and elastic scheduling

    At a high level, a full-accelerator deployment may require anywhere from several dozen to more than one hundred accelerators. Hardware investment is often in the multi-million-yuan range, depending on the accelerator count, interconnect specifications, redundancy, and purchasing conditions. The cost cannot be estimated accurately by simply multiplying the price of one accelerator by the number of cards.

    4. The Hybrid Approach: Store Experts in CPU Memory and Run Shared Attention on GPUs

    A more practical validation strategy is hybrid CPU–GPU inference.

    The basic idea is to keep the shared attention components on GPUs, where high-bandwidth compute is available, while offloading the large number of sparse experts to CPU memory. Because only 16 of the 896 experts are activated for each token, a large-memory server can feed the selected experts to the accelerator during inference.

    For a 2.8 trillion-parameter model, 4-bit weights require approximately 1.49 TB. CPU memory should not be sized at exactly 1.5 TB because the system also needs room for runtime execution, page cache, expert scheduling, metadata, and alignment.

    A practical starting point is:

    • At least 2–4 TB of system memory
    • Approximately 3–6 TB of NVMe storage for weights, images, and caches
    • One or two high-memory accelerators, or a small number of accelerators, for shared attention, hot experts, and communication buffers

    Publicly reported practice has shown that a dual-socket CPU server paired with a single high-end consumer GPU can run a 671-billion-parameter, 4-bit quantized MoE model. Under a constrained expert-selection strategy, such a setup achieved more than 200 tokens per second during prefill and approximately 14 tokens per second during decode.

    These figures demonstrate that a hybrid architecture can be useful for validation and moderate-throughput workloads. However, they should not be directly extrapolated to high-concurrency production serving for a 2.8 trillion-parameter model. The larger model will introduce substantially higher CPU memory requirements, greater expert-scheduling pressure, and higher time-to-first-token latency.

    Hybrid deployments generally cost less than full accelerator deployments. A typical budget may range from several hundred thousand to around one million yuan, with the main expenses coming from:

    • Large-memory servers
    • NVMe storage
    • A small number of accelerators
    • High-speed internal networking

    For small-scale validation, general-purpose accelerated computing, object storage, and container platform capabilities from providers such as Baidu AI Cloud may also be used. The initial investment can then be controlled according to the selected specifications and billing model.

    5. Cost Estimation Formula and Deployment Checklist

    The following pseudocode can be used to build a basic budget model:

    1. def estimate_moe_cost(total_params_t, card_gb, card_price,
    2. kv_gb, workspace_gb, overhead_ratio=0.25,
    3. replicas=1):
    4. # 4-bit microscaling floating-point format: approximately 4.25 bits per parameter
    5. weight_gb = total_params_t * 1e12 * 4.25 / 8 / 1e9
    6. runtime_gb = weight_gb * overhead_ratio
    7. total_gpu_gb = weight_gb + runtime_gb + kv_gb + workspace_gb
    8. cards = -(-int(total_gpu_gb) // card_gb) * replicas
    9. hardware_cost = cards * card_price
    10. return weight_gb, total_gpu_gb, cards, hardware_cost
    11. def estimate_hybrid_cost(total_params_t, cpu_mem_gb, cpu_mem_price,
    12. nvme_tb, nvme_price, gpu_count, gpu_price):
    13. weight_gb = total_params_t * 1e12 * 4.25 / 8 / 1e9
    14. cpu_mem_recommended = max(cpu_mem_gb, weight_gb * 1.5)
    15. cost = cpu_mem_recommended * cpu_mem_price + nvme_tb * nvme_price \
    16. + gpu_count * gpu_price
    17. return weight_gb, cpu_mem_recommended, cost

    Before selecting a deployment architecture, verify the following:

    1. Are the weights estimated from the total parameter count rather than the activated expert ratio?
    2. Is KV cache estimated separately based on concurrency, sequence length, layer count, and precision?
    3. Will expert parallelism cross high-speed interconnect domains?
    4. Could all-to-all communication become the main bottleneck?
    5. Does the full-scale design reserve 15%–30% of GPU memory as headroom?
    6. Does the hybrid design provide at least 1.5 times the weight size in CPU memory?
    7. Is there enough storage for weights, images, caches, and logs?
    8. Are power, cooling, rack space, and operations included in the TCO?
    9. Can a smaller-scale test measure throughput, latency, and expert-routing behavior before expansion?

    Conclusion

    The local deployment cost of a 2.8 trillion-parameter MoE model starts with approximately 1.49 TB of quantized weights. The upper bound is determined by the communication fabric, concurrent KV cache usage, and required service level.

    A full-accelerator deployment generally requires several dozen or more accelerators and can reach a multi-million-yuan hardware investment. A CPU–GPU hybrid deployment centers on 2–4 TB of system memory and a small number of accelerators, making it more suitable for validation and moderate-throughput workloads.

    Budget planning should begin with the capacity budget, then be adjusted for communication and serving requirements. The expert activation ratio alone is not sufficient to estimate deployment cost.