1

    This article breaks down the cost of deploying a trillion-parameter Mixture-of-Experts (MoE) large language model locally. It covers weight storage and quantization overhead, KV cache and memory peaks, all-to-all communication across expert-parallel accelerators, power and cooling, and a three-stage deployment plan with a cost control checklist.

    1. Conclusion First: Full Local Deployment Is a Systems Engineering Problem

    For MoE models with trillions of parameters, the cost of local deployment cannot be estimated from model file size alone. Four factors determine it: storage for weights and quantization metadata, accelerator memory used by the KV cache and runtime workspace during inference, the interconnect bandwidth needed for all-to-all communication across expert parallelism, and the power, cooling, spare parts, and operations staff for the data center. A weakness in any one of these means that a model that loads is not necessarily a service that runs stably.

    Take a 2.8-trillion-parameter MoE model as an example. The theoretical floor for its 4-bit weights is about 1.49 TB. That figure covers only the weights and the block-quantization scaling factors. It does not include non-4-bit tensors, metadata, alignment, runtime workspace, communication buffers, or the KV cache. Assuming roughly 80 GB of usable memory per accelerator, the weight floor alone is close to 19 accelerators. A real deployment must also account for parallel partitioning, expert distribution, redundancy, and peak concurrency. A common approach starts with a validation environment of about 32 accelerators. A full-scale production deployment usually needs 64 or more accelerators in a single high-speed interconnect domain. The cost difference comes mainly from the different goals of validation and production, not from the model file having multiple versions.

    2. Total Parameters Set Storage, and the Activation Ratio Sets Compute

    MoE architectures can lead to a misunderstanding: if each token activates only some experts, does memory also scale only with activated parameters? It does not. In MoE deployment, all experts must reside in storage or accelerator memory. At inference time, the router sends each token to its target experts. The activation ratio affects compute and communication volume. It does not lower the storage floor.

    Suppose the model has 896 experts and each token activates 16. The expert activation ratio is then about 16/896 ≈ 1.79%. This ratio helps estimate compute cost and routing pressure, but it cannot be applied directly to memory estimates. The correct way to estimate the memory floor is to first calculate the weights of all experts from the total parameter count, then add non-expert parameters, quantization scaling factors, runtime buffers, and the KV cache. For 2.8 trillion total parameters, the raw footprint of 4-bit weights plus block-quantization scaling factors can be estimated as follows:

    1. Weight (GB) ≈ total parameters × average bits per element ÷ 8 ÷ 1e9
    2. ≈ 2.8e12 × 4.25 ÷ 8 ÷ 1e9
    3. ≈ 1487.5 GB

    The 4.25 bits come from the common block-quantization overhead of one 8-bit scaling factor per 32 4-bit weights, which is 4 + 8/32 = 4.25 bits. If another quantization granularity is used, recalculate for the actual format. This result still excludes embedding, router, normalization, and output layers, which may keep higher precision. It also excludes metadata, alignment padding, runtime workspace, and the KV cache. The actual accelerator memory requirement is therefore usually well above 1.49 TB.

    As a scale reference, an MoE model of around 600B parameters, quantized to a lower bit width, may need hundreds of GB for weights and activations to run on a small number of high-end accelerators. A 2.8-trillion-parameter model has more than four times the total parameters, so its memory and interconnect requirements for full deployment are considerably higher. The key is not to memorize an absolute number. The key is to apply the split: total parameters determine storage, and the activation ratio determines compute.

    3. Quantization Math: 4-Bit Is Not Simply Divided by 8

    4-bit quantization is often simplified to “parameter count divided by two.” Engineering estimates must include scaling factors, non-4-bit tensors, and runtime overhead. For 2.8 trillion parameters, the theoretical raw 4-