How Many GPUs Does It Take to Deploy a 2.8T-Parameter MoE Model?

    A 2.8-trillion-parameter Mixture-of-Experts model cannot usually run directly on a single eight-GPU server. This article estimates weight storage and GPU memory requirements, compares quantized and BF16 deployments, explains tensor, expert, and pipeline parallelism, and outlines cold-start, capacity-check, and cloud deployment considerations.

    1. Start by Calculating the Model Size

    A 2.8-trillion-parameter MoE model is far beyond the capacity of a single server. Before evaluating whether deployment is feasible, calculate two key figures: the size of the weight files and the memory required during inference.

    The original specifications indicate that the weight files are approximately 1.56 TB. Dividing this by 2.8 trillion parameters gives an average of about 0.56 bytes per parameter, or approximately 4.5 bits per parameter. This strongly suggests that the released weights are quantized or use a mixed-precision format. The exact precision structure should be confirmed in the official model card.

    If all parameters use BF16, with 2 bytes per parameter, the total weight size would be approximately 5.6 TB. These two estimates differ by more than three times. Choosing the wrong assumption can therefore result in a major error in hardware planning.

    2. Estimating GPU Memory Requirements

    A rough estimate can be expressed as:

    1. GPU memory per card ≈ sharded weights + KV cache + activations + framework overhead

    The weight shard is the total weight size divided by the number of GPUs participating in sharding. Framework overhead typically requires an additional 10% to 20% of total GPU memory. KV-cache usage grows with both concurrency and context length. In long-context workloads, the KV cache can consume more memory than the model weights.

    Using the 1.56 TB weight estimate and decimal units:

    Deployment option Total weight size Minimum number of 80 GB GPUs Notes
    Quantized weights 1.56 TB 20 Weights only; cache and overhead are excluded
    Full BF16 precision 5.6 TB 70 Approximately nine 8-GPU servers

    A single eight-GPU server therefore cannot hold 1.56 TB of weights directly. Eight 80 GB GPUs provide 640 GB in total, which is only about 40% of the weight size. Even with 141 GB GPUs, eight cards provide approximately 1,128 GB, still less than 1.56 TB.

    As a result, whether an eight-GPU deployment is possible depends primarily on the quantization strategy, not only on the GPU model. Running the model on one eight-GPU server would require more aggressive quantization, or placing part of the weights in CPU memory or on local SSD storage. The latter can significantly reduce inference speed and is generally suitable only for offline workloads or infrequent requests.

    3. Choosing a Parallelism Strategy

    Three common parallelism strategies are used for large-model deployment:

    • Tensor parallelism (TP): Each layer’s matrices are split across multiple GPUs. Communication occurs inside every layer, so TP requires high-bandwidth GPU interconnects and is usually kept within a single server.
    • Expert parallelism (EP): Different experts are placed on different GPUs. Each MoE layer must distribute and collect tokens, resulting in frequent all-to-all communication.
    • Pipeline parallelism (PP): The model is divided by layer into multiple stages. Cross-node communication is relatively limited, but pipeline idle time can increase latency.

    In practice, a common combination is TP within a server and EP or PP across servers. The appropriate configuration depends on the capabilities of the selected inference framework. The theoretically optimal configuration may not work well on the available network, so it should be validated with realistic load tests.

    If inter-node bandwidth is insufficient, GPUs may spend substantial time waiting for communication, causing utilization to fall significantly.

    4. Weight Loading and Cold Start

    Weight-loading speed determines how quickly an inference service can start. For a 1.56 TB model:

    • If a single node reads the entire weight set at 2 GB/s, loading takes approximately 13 minutes: 1560 ÷ 2 = 780 seconds.
    • If 20 nodes each load one shard of approximately 78 GB at the same 2 GB/s speed, loading takes approximately 39 seconds.

    Actual performance depends on the storage backend, network bandwidth, and cache hit rate, so these figures must be validated through measurement.

    The following practices can help:

    1. Store the weights in object storage located in the same region as the GPU nodes, and use local NVMe drives as a cache.
    2. Verify the hash of the weight files before loading them to prevent corrupted files from producing abnormal inference results.
    3. Record loading time for every startup and use the data to guide capacity planning and scaling decisions.

    5. Pre-Deployment Checklist

    Before deployment:

    1. Review the official model card to confirm weight precision, the number of experts, the number of active parameters, and license terms. Details such as 896 routed experts, top-16 routing, and two shared experts should be verified against the official documentation.
    2. Estimate per-GPU memory requirements using the formula above and reserve capacity for framework overhead.
    3. Confirm the GPU model, number of nodes, and available inter-node bandwidth.
    4. Verify that the inference framework supports the model architecture and the required parallelism strategies.
    5. First run a single request with a short context, then test concurrency and long-context workloads.
    6. Record time to first token, throughput, peak GPU memory usage, and weight-loading time.
    7. Set scaling thresholds based on load-test results.

    The following script checks available GPU resources before model loading and estimates the required number of GPUs:

    1. import math
    2. import torch
    3. def check_gpu_capacity(required_gb_per_card: float) -> bool:
    4. """Check whether each visible GPU has sufficient memory."""
    5. if not torch.cuda.is_available():
    6. raise RuntimeError("No usable CUDA device detected")
    7. count = torch.cuda.device_count()
    8. ok = True
    9. for i in range(count):
    10. total_gb = torch.cuda.get_device_properties(i).total_memory / 1e9
    11. print(f"GPU {i}: {total_gb:.1f} GB")
    12. if total_gb < required_gb_per_card:
    13. ok = False
    14. print(f"Visible GPUs: {count}, capacity sufficient: {ok}")
    15. return ok
    16. def estimate_card_count(weight_tb: float, card_gb: float, overhead: float = 0.15) -> int:
    17. """Estimate the minimum number of GPUs required.
    18. Includes weights and overhead, but not concurrent KV-cache usage.
    19. """
    20. weight_gb = weight_tb * 1000
    21. needed = weight_gb * (1 + overhead) / card_gb
    22. return math.ceil(needed)
    23. if __name__ == "__main__":
    24. print("Estimated number of 80 GB GPUs:", estimate_card_count(1.56, 80)) # Approximately 23
    25. check_gpu_capacity(required_gb_per_card=80)

    This script only checks resources and provides a rough estimate. It does not load the model. Model loading and inference must be implemented according to the official documentation for the selected framework.

    6. Cloud Deployment Trade-Offs

    Purchasing hardware requires significant upfront investment and also introduces data-center, operations, and depreciation costs. Cloud GPU instances allow teams to request nodes as needed, making them suitable for temporary load testing and handling traffic peaks.

    When using a container platform for elastic scaling, pay close attention to scale-out time. After a new node starts, it must still load the model weights, so the total scaling delay includes weight-loading time. Nodes can be pre-warmed, or a certain number of instances can be kept running continuously.

    Platforms such as Baidu AI Cloud provide GPU instances, object storage, and container services. The specific instance types, bandwidth options, and billing models should be verified against the provider’s current specifications before selection.

    A cost comparison can be calculated as follows:

    1. hourly node price × required node count
    2. + storage cost
    3. + data transfer cost

    This total can then be compared with the depreciation cost of purchasing and operating the equivalent hardware.

    7. Conclusion

    With a weight size of approximately 1.56 TB, a single eight-GPU server usually cannot run the model directly. The exception would be a more aggressive quantization scheme, potentially combined with slower storage or CPU offloading.

    Most teams should first confirm the actual precision information in the model card, then estimate GPU requirements using:

    1. GPU memory per card ≈ weight shard + KV cache + activations + framework overhead

    The final deployment plan should be validated with real workloads, including short and long contexts, different concurrency levels, inter-node communication, startup time, and peak memory usage.