Memory Requirements and Cost Breakdown for Local Deployment of Very Large MoE Models

1. Turning Parameter Count into Hardware Constraints

Mainstream large language models are increasingly adopting the Mixture of Experts (MoE) architecture. The core idea is straightforward: the total parameter count can be very large, but each token only activates a small subset of experts, which controls compute cost during training and inference.

However, memory logic and compute logic are not the same.

During MoE inference, all expert parameters typically need to reside in storage that can be addressed quickly. In other words, even though each token only activates 16 experts, the remaining experts still need to be on standby. If the router hits an expert that has not been loaded, the inference pipeline stalls.

Take a trillion-parameter model as an example. With a 4-bit weight format, the raw weight data alone approaches 1,400 GB. In practice, you also need to account for scale factors, non-quantized tensors, KV cache, runtime working memory, and communication buffers. In block-wise quantization formats where every 32 4-bit elements share one 8-bit scale factor, the effective bit width rises from 4 bits to roughly 4.25 bits. Under this estimate, weights alone can exceed 1.4 TB.

This means no single accelerator, no matter how large its memory, can realistically hold the full model. The first question local deployment must answer is:

How many cards, how many machines, and what kind of interconnect topology will host the model?

---

2. Tier 1: Just Make It Run, Speed Can Wait

2.1 Defining the Goal

The goal at this tier is not performance but usability. Success means the model loads completely and can produce tokens at a very low rate.

Typical use cases include:

• Local validation of model quality
• Small-batch offline inference
• Preliminary experiments before running evaluation suites
• Technical validation of weight formats and inference pipelines

2.2 Lower Bound on Memory

For a 4-bit trillion-parameter model, a conservative memory estimate is between 1.4 TB and 1.8 TB. Assuming a single accelerator offers 80 GB of usable memory, the ideal case requires at least 20 cards. With redundancy, KV cache, and runtime overhead, the realistic requirement may exceed 30 cards.

Many individuals and small teams do not have GPU clusters at this scale, which naturally leads to CPU memory offloading.

2.3 Feasibility of Heterogeneous Inference

Heterogeneous inference places compute-intensive components on GPUs and storage-heavy but compute-sparse components in CPU memory.

MoE architectures map naturally onto this split:

• Attention layers are involved in every token’s forward pass and have high compute intensity, making them suitable for GPUs.
• Expert layers contain massive parameters, but each token only activates a few experts, making them suitable for memory offloading.

In this mode, memory capacity becomes the primary constraint rather than GPU memory. A dual-socket server with more than 1 TB of DDR memory, combined with one or a few high-performance GPUs, can theoretically load the full model.

2.4 Performance Ceiling

The fundamental bottleneck of heterogeneous setups is the gap between PCIe bandwidth and memory bandwidth.

When expert parameters live in CPU memory, every token generation step requires moving the activated experts’ weights. If a single token needs to read several GB of parameters while practical bidirectional PCIe bandwidth is only tens of GB/s, token generation speed will be capped at a few to a dozen tokens per second.

In practice, heterogeneous setups on 600B-class models can reach around a dozen tokens per second during decoding, but only after applying optimizations such as expert selection policies and caching hot experts on GPU. For trillion-parameter models, the speed will only be lower.

This tier can “run,” but it cannot support interactive conversation.

---

3. Tier 2: Single Node with Multiple GPUs, Targeting Usable Speed

3.1 Hardware Form Factor

When the goal moves from “it runs” to “it is usable,” all weights generally need to fit in GPU memory to avoid frequent PCIe transfers.

Assuming a usable memory requirement of 1.5 TB and 80 GB per card, at least 20 cards are theoretically needed. But simply filling a machine with cards is not enough. Interconnect is the key.

3.2 All-to-All Communication as the Core Bottleneck

During MoE inference, every token goes through a routing step. The routing result is sparse: each token is assigned to a few experts, and those experts are distributed across different cards.

The communication pattern is all-to-all:

1. Each card determines the target card for each token based on the assigned expert.
2. Activations are sent to the target card.
3. The target card computes the expert forward pass.
4. Results are sent back to the original card.

If there are many experts spread across dozens of cards and the cards are connected only by PCIe or ordinary networking, communication latency quickly becomes the bottleneck.

This explains why simply adding more cards does not necessarily improve throughput. Large-scale MoE inference stresses the communication domain first, not raw compute.

3.3 The Value of NVLink and High-Bandwidth Domains

High-speed interconnects matter because they keep all-to-all traffic within the same communication domain, reducing per-transfer latency and improving effective bandwidth.

With common high-speed interconnect architectures, multiple cards within a single node can deliver several times the bandwidth of PCIe. If the 16 activated experts can be kept within the same communication domain, inference efficiency improves significantly.

This is why vendors recommending deployment of trillion-parameter models often require 64 or more accelerators in a supernode rather than a collection of scattered smaller clusters.

---

4. Tier 3: Multi-Node Deployment for Stable Serving

4.1 Why Multiple Machines Are Necessary

When the model exceeds the memory capacity of a single machine, cross-machine deployment becomes unavoidable. At that point, inter-node network bandwidth directly determines the upper bound of inference performance.

A typical configuration approach:

• Eight cards per node forming a high-speed domain
• Multiple nodes connected via RDMA networking
• Expert layers partitioned across machines by layer or by group
• Attention layers placed on machines with fewer compute cards, or co-located with selected experts

4.2 Memory Is Not the Only Cost

In multi-node deployment, cost is no longer determined by GPUs alone. You also need to consider:

• High-speed switches and optical modules
• Bandwidth matching between intra-node NVLink domains and inter-node RDMA networking
• Multi-node weight distribution and version management
• Load balancing and failure recovery for inference services
• Cross-request KV cache reuse policies

In long-context scenarios, KV cache grows rapidly. A text context of a few MB can generate tens or even hundreds of GB of cache usage under dozens of concurrent requests. This memory must be set aside separately in deployment estimates.

4.3 Choosing a Parallelism Strategy

In multi-node inference, common parallelism approaches include:

Tensor parallelism: Splits the parameters of a single layer across multiple cards. Well suited for compute-intensive attention layers, but requires significant communication.

Expert parallelism: Places different experts on different devices. A natural fit for MoE, but requires handling all-to-all communication.

Pipeline parallelism: Splits the model by layer, with different devices handling different layers. Communication volume is low, but pipeline bubbles are likely.

Real deployments usually mix multiple strategies. For example, attention layers may use tensor parallelism, expert layers may use expert parallelism, and pipeline parallelism may be applied between layer groups.

The choice of strategy directly determines hardware utilization, which in turn determines the cost per token.

---

5. Quantization Format Affects More Than Memory

5.1 4-bit Is Not Just Dividing by Two

Going from 8-bit to 4-bit intuitively halves memory usage. Reality is more complicated.

Block-wise quantization requires storing scale factors. If every 32 elements share one 8-bit scale factor, each element carries an additional 0.25 bits. Smaller blocks mean higher scale overhead. Some formats also keep FP16 or BF16 copies of non-quantized layers, which erodes the theoretical savings.

5.2 The Importance of Quantization-Aware Training

A trillion-parameter model quantized only at inference time may show noticeable quality degradation. A more robust approach is to introduce quantization-aware training late in the training process so that model parameters remain well-separated under low-precision representation.

This mainly affects model providers, but deployers should understand that quality differences across formats may matter more than memory differences.

5.3 Matching Weight, Activation, and KV Cache Formats

In an inference pipeline, weights, activations, and KV cache can use different formats. For example, weights in 4-bit, activations kept at 8-bit, and KV cache in FP16. All three must match the input and output requirements of the compute kernels; they cannot be mixed arbitrarily.

If deployment planning focuses only on weight size and ignores activation memory, it is easy to end up in a situation where weights fit but the system runs out of memory as soon as execution starts.

---

6. A Reasonable Framework for Cost Estimation

A local deployment budget should cover the following items:

Cost Item	Key Variables
Accelerators	Quantity, memory, interconnect bandwidth
Servers	Cards per node, RAM capacity, PCIe topology
Networking equipment	Inter-node bandwidth, switch hierarchy
Racks and power	Power density, cooling capacity
Operations and tuning	Parallelism strategy, quantization format, service stability

For a trillion-parameter 4-bit model, if the goal is merely to run, heterogeneous memory offloading can keep hardware costs within a certain range, but generation speed will be extremely slow. Reaching interactive speed requires at least dozens of high-end accelerators with high-speed interconnect. Supporting concurrent external traffic raises costs by another order of magnitude.

There is no single answer to “how much will it cost.” It depends on your tolerance for latency, throughput, and concurrency.

---

7. Conclusion: Define the Service Level First, Then Calculate the Hardware Budget

The memory threshold for local deployment of very large MoE models is determined by the quantized size of the total parameters, while the performance bottleneck is dominated by all-to-all communication. Deployment tiers fall into at least three levels:

1. Barely runs: Heterogeneous offloading, lowest cost, slowest speed.
2. Usable: Single-node multi-GPU, all weights in GPU memory, sufficiently large communication domain.
3. Serviceable: Multi-node supernodes, RDMA networking, significantly higher engineering complexity.

Before purchasing hardware, a more rational sequence is:

1. Define the minimum acceptable tokens per second.
2. Estimate peak concurrency and KV cache memory.
3. Determine the weight format and parallelism strategy.
4. Work backward to the hardware list.

Otherwise, it is easy to fall into traps like “enough memory but poor communication,” “many cards but insufficient bandwidth,” or “it runs but cannot be used.” Local deployment of very large models has never been purely a memory problem. It is a combined trade-off involving communication, engineering, and cost.