Memory and Cost Requirements for Deploying Trillion-Parameter MoE Models Locally

1. Start with the basics: what parameter counts actually mean for resources

Before discussing how much it costs to deploy a trillion-parameter model locally, it is necessary to separate two concepts that are often confused:

• Total parameter count determines storage requirements. In a Mixture-of-Experts (MoE) architecture, all experts must remain resident in storage or GPU memory, and they are called dynamically based on routing decisions during inference.
• Active parameter count determines compute consumption. Each token passes through only a small number of experts, so the floating-point workload is far smaller than what the total parameter count would suggest.

Take a common class of very large MoE models as an example: roughly 2.8 trillion total parameters, 896 experts, with 16 experts activated per token. That gives an expert activation ratio of about 1.79%. It is important to note that this is an expert activation ratio, not a strict parameter activation ratio — non-expert components such as the attention backbone, shared layers, and embedding layers always participate in computation, so the actual activated parameter share is higher than that number.

The conclusion is straightforward: for MoE deployment, GPU memory is estimated based on total parameters, while the activation ratio only affects the compute budget.

2. Quantization formats: from raw weights to real memory usage

2.1 4-bit is only the starting point

Assume the model uses quantization-aware training from the supervised fine-tuning stage, with weights in 4-bit MXFP4 format and activations in MXFP8. Converting 2.8 trillion parameters to 4-bit gives a theoretical lower bound of:

2.8e12 × 4 bit ÷ 8 = 1400 GB

However, 1400 GB is only the theoretical minimum. MXFP4 is a block floating-point format: for every 32 4-bit elements, an additional 8-bit scale factor is stored. According to the OCP specification, the actual average bit width per weight element is:

4 + 8/32 = 4.25 bit

Therefore, the actual weight footprint becomes:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

2.2 Five categories of overhead that are easy to overlook

1487.5 GB is still not the final answer. Real-world deployment must account for additional overhead:

Overhead type	Description
Non-4-bit tensors	Embedding layers, normalization layers, and routing gates often remain in higher precision
Metadata	Expert indices, routing tables, and tensor descriptors
Memory alignment	Tensors are padded to meet hardware requirements, typically causing 5%–10% waste
Runtime workspace	Temporary buffers for operators and communication buffers
KV cache	Grows linearly with concurrency and context length

In practice, after combining these factors, total system memory requirements typically need to be 20%–40% higher than the theoretical weight value.

2.3 A useful reference case

To establish a sense of scale, consider a medium-sized MoE deployment as a reference: a model with 671B main parameters plus 14B multi-token prediction parameters requires about 685 GB in FP8. After switching to 4-bit weight quantization, it needs roughly 335 GB for weights plus 37 GB for activations, totaling about 400 GB to run on eight 80 GB accelerators with 640 GB of combined memory.

Scaling by size, a full deployment of a 2.8-trillion-parameter model would require at least four times the above case. This means a single-node 8-GPU solution is essentially infeasible; a multi-node superpod approach is required.

3. Communication: a bottleneck more hidden than memory

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. The reason lies in the inference flow itself:

token → routing computation → dispatch to the cards hosting target experts → expert computation → collect activations → proceed to the next layer

With 896 experts distributed across multiple accelerators, every token goes through a “send out — compute — send back” cycle at every layer. If communication latency cannot be hidden behind computation, hardware utilization drops sharply.

This is where the superpod approach proves its value: by using a high-bandwidth, low-latency interconnect domain such as NVLink or an equivalent high-speed fabric, all-to-all communication can be confined within a single communication domain, avoiding long-distance cross-domain transfers. Engineering recommendations:

• Align expert placement with the interconnect topology to reduce cross-switch traffic;
• Use hierarchical all-to-all by aggregating within a node first and then transferring across nodes;
• Apply capacity factor limits on routing to prevent hot experts from causing communication skew;
• Overlap communication with computation by interleaving dispatch with the previous layer’s compute.

4. Cost estimation framework

Deployment costs can be broken down into four parts, best estimated in the following order:

4.1 Number of accelerators

Memory capacity is the primary constraint. With 80 GB per accelerator:

required accelerators ≥ total memory requirement ÷ 80 GB

If the total system requirement is in the 2000 GB range, at least 32 accelerators are needed. Accounting for KV cache and workspace headroom, a superpod with 64 or more accelerators is a more realistic starting point.

4.2 Interconnect and system integration

A superpod is not simply a pile of accelerators. It requires matching high-speed switching, power delivery, and cooling. This part of the cost is often underestimated and can account for 20%–30% of the total investment.

4.3 Power and operations

Inference on a trillion-parameter model consumes tens of kilowatts. Electricity and cooling costs need to be modeled separately on an annual basis and are often on the same order of magnitude as hardware depreciation.

4.4 Trade-offs between quantization and distillation

If budget is limited, consider:

• More aggressive quantization such as 2-bit weights, which requires re-running quantization-aware training and evaluating accuracy loss;
• Expert pruning or layer reduction to trade accuracy for memory savings;
• Alternative architectures with more extreme sparse activation.

One caution: any compression method must be evaluated end-to-end on the target workload. Offline metrics and online performance often diverge.

5. Deployment checklist

Before placing any purchase order, confirm the following items one by one:

1. Whether the target model’s weight format and quantization scheme are publicly available;
2. Whether the inference framework supports the routing and parallelism strategies required by the MoE architecture;
3. Whether memory estimates include KV cache and runtime overhead;
4. Whether interconnect bandwidth can cover peak all-to-all traffic;
5. Whether power, cooling, and rack weight capacity meet system requirements;
6. Whether there is a fallback plan for small-scale validation, such as running low-concurrency stress tests first.

Summary

Deploying a trillion-parameter MoE model locally is fundamentally a problem governed by three constraints: memory capacity × interconnect bandwidth × power supply. The quantization format determines the memory floor, MoE routing determines communication pressure, and superpod interconnect determines whether the available compute can actually be fully utilized. Once these three factors are properly accounted for, cost estimates no longer remain stuck at the crude “parameters divided by 8” stage. For most teams, the lower-risk path is to first validate communication efficiency and inference framework compatibility on a small-scale cluster before committing to a full superpod expansion.