Memory Requirements and Cost Breakdown for Deploying Trillion-Scale MoE Models Locally

1. The Relationship Between Parameter Scale and Memory Usage

For a sparse expert model with a total parameter count reaching the trillion scale, its memory requirement cannot be estimated simply by multiplying the parameter count by the number of bytes. The key lies in understanding the storage characteristics of sparse architectures: although each token only activates a few experts, all expert weights must be placed in addressable storage during deployment. In other words, the activation ratio only affects compute consumption, not the storage lower bound.

Consider a model with approximately 2.8 trillion total parameters, comprising hundreds of experts, where only a dozen or so are activated per token. The expert activation ratio is roughly 1.8%. This figure might mislead one into thinking the memory footprint can be compressed significantly. However, during actual inference, the routing mechanism must select target experts from the entire pool at any given moment. If certain experts are not in high-speed memory, they must be fetched from remote storage, which multiplies latency.

Therefore, when estimating memory, calculations must be based on total parameters rather than activated parameters. This is the core reason many local deployment plans encounter miscalculations right from the start.

1. Structural Characteristics of Sparse Expert Models

These models typically consist of two parts. The first is a shared attention module that every token passes through; it is compute-intensive but has a relatively manageable parameter count. The second is a large set of parallel feedforward expert networks, which account for a massive proportion of the parameters but only activate a small fraction at a time.

This structure is naturally suited for a “compute-storage decoupled” deployment approach: placing the frequently activated shared components on high-bandwidth compute devices, while storing the massive expert pool in large-capacity memory and scheduling them on demand.

2. Raw Data Lower Bound After Quantization

When using a 4-bit weight format, the raw weight data size for a 2.8 trillion parameter model is approximately 1400GB. In actual engineering, however, one cannot look solely at raw weights; the overhead of quantization scale factors must also be considered. Calculating with one 8-bit scale for every 32 4-bit elements, the effective bit width per element is roughly 4.25 bits, bringing the total weight and scale size close to 1488GB.

This is merely the lower bound for the weights. Adding non-quantized tensors, metadata, memory alignment, runtime workspaces, and the key-value (KV) cache typically increases actual usage by another 20% to 40%.

3. Comparison with Other Large-Scale Models

To better grasp this scale, consider a 671B parameter dense model as a reference. In an 8-bit floating-point format, the main parameters plus additional prediction module parameters total about 685GB. Even with 4-bit grouped quantization, weights and activations still require about 400GB to run on 8 accelerator cards with 80GB of memory each.

The physical footprint of a trillion-scale sparse model is more than four times this reference model. This means a single 8-GPU machine cannot support a full-capability deployment, necessitating larger-scale nodes or heterogeneous storage solutions.

2. Why Official Guidelines Recommend Super-Node Deployment

Large-scale sparse expert models are far more sensitive to communication bandwidth than ordinary dense models. The reason is that for every token processed, the routing mechanism sends intermediate activations to the device hosting the target expert, and then sends the results back. This process is known as all-to-all communication.

When hundreds of experts are distributed across multiple cards, the frequency of this data transfer is extremely high. If devices are connected via a slow bus, communication latency will directly consume the compute gains. For instance, if an expert computation takes 5 milliseconds but data transfer takes 20 milliseconds, compute utilization drops to just 20%.

1. Communication Domains and Interconnect Topology

The core idea behind the super-node approach is distributing all experts within a single high-bandwidth communication domain. By using high-speed interconnects to form dozens or even hundreds of accelerator cards into a single logical entity, data exchange between experts avoids leaving this communication domain.

This requires high-speed interconnect topologies, such as point-to-point high-speed links or dedicated switch fabrics. Specific interconnect technologies vary by vendor, but from an engineering perspective, the goal is identical: suppress cross-card communication latency to a level far below the time required for a single expert computation.

2. Crossing the Threshold from Single-Node to Multi-Node

A single machine with 8 GPUs is currently the most common deployment unit. For trillion-scale sparse models, however, the total memory and communication bandwidth of 8 cards are insufficient. Crossing this threshold causes a noticeable jump in the cost curve—requiring not only more accelerator cards, but also high-speed switches, host servers with larger memory, and more complex operations and maintenance systems.

3. Tier 1: Functional Validation, Regardless of Speed

If the goal is simply to get the model running, even if it only generates a few tokens per second, the hardware threshold can be significantly lowered. This requires abandoning the insistence on an “all-GPU deployment” in favor of a heterogeneous approach.

1. Basic Concept of CPU-GPU Heterogeneous Inference

The shared attention component of these models is compute-intensive and well-suited to remain on the GPU. Conversely, the massive sparse expert parameters have high storage overhead but low single-activation frequency, meaning they can be entirely offloaded to CPU memory. By using a large-memory server to hold the expert weights, the GPU is left to handle only the high-bandwidth compute portions.

The performance bottleneck of this approach lies in CPU memory bandwidth and PCIe lane speed. However, it allows a deployment that would otherwise require dozens of accelerator cards to be reduced to a single consumer-grade or workstation GPU paired with a large-memory server.

2. Typical Hardware Configuration and Performance Levels

Take a dual-socket server processor paired with a single 24GB GPU as an example. Running a 671B parameter 4-bit quantized dense model, the prefill stage can reach hundreds of tokens per second, while the decoding stage achieves around a dozen tokens per second. While this performance level is unsuitable for production-grade online services, it is sufficient for local validation, model evaluation, and feature debugging.

For a trillion-scale sparse model, the longer communication path resulting from expert offloading means actual decoding speeds will be lower, likely dropping into the single-digit tokens per second range. But for a Tier 1 goal—just getting it to run—this is acceptable.

3. Key Variables Affecting Heterogeneous Performance

The first variable is CPU memory bandwidth. Dual-socket servers typically provide theoretical bandwidth in the hundreds of GB/s, but the random access pattern of expert parameters means actual bandwidth utilization is far lower than the theoretical peak. The second variable is the PCIe lane version and width, which dictates the data exchange rate between the GPU and CPU.

Additionally, expert selection strategy heavily impacts performance. If it is possible to predict and load only the subset of experts most likely to be activated, data transfer volume can be significantly reduced. However, such strategies require additional routing caches or prediction mechanisms, introducing a trade-off between accuracy and speed.

4. Tier 2: Acceptable Inference Speed

Once simply “running” is no longer the issue, the next step is elevating performance to a usable speed. This generally means decoding speeds of at least dozens of tokens per second, and prefill speeds capable of supporting a degree of concurrency.

1. Multi-GPU Parallelism and Expert Sharding

To increase speed, the core objective is keeping more experts resident in high-speed GPU memory. This involves sharding the expert pool across cards, with each card responsible for computing a subset of experts. This requires a full-GPU deployment, where total memory must be at least 1.2 to 1.5 times the size of all weights.

For a trillion-scale 4-bit quantized model, this means 1,500 to 2,000GB of available GPU memory. Based on 80GB per card, at least 20 to 30 accelerator cards are required. Factoring in communication redundancy and batch processing space, a higher configuration is recommended in practice.

2. Batching and Throughput Optimization

In online service scenarios, single-request latency is not the only metric; throughput is equally important. Batching multiple requests can amortize routing communication overhead and expert loading costs. However, this increases memory pressure, as the KV cache grows linearly with batch size.

A sound practice is to create a memory budget table: allocating specific amounts for weights, KV cache, and runtime workspaces, and then working backward to determine the maximum supported batch size and concurrency.

5. Cost Breakdown Framework

The total cost of a local deployment is not limited to hardware procurement. Supporting equipment, power, cooling, and operations and maintenance (O&M) labor must also be factored in.

1. Hardware Costs

The primary cost for a Tier 1 heterogeneous approach is the large-memory server. A dual-socket server configured with 512GB to 1TB of memory paired with a mid-to-high-end GPU can keep total costs in the tens of thousands of dollars. Conversely, a Tier 2 full-GPU solution requires dozens of datacenter-grade accelerator cards and high-speed interconnect switches, driving costs up to the millions of dollars.

2. Power and Cooling

The total power consumption of multi-GPU systems typically ranges from several kilowatts to tens of kilowatts. Based on industrial electricity rates and continuous year-round operation, power costs represent a significant portion of the Total Cost of Ownership (TCO). Furthermore, high-density deployments require datacenter-grade cooling capacity, which standard home or office environments simply cannot provide.

3. O&M and Iteration

Large-scale clusters have a significantly higher failure rate than single machines. Hard drives, memory modules, and accelerator cards can all fail, requiring spare parts and established maintenance procedures. Concurrently, model version iterations will introduce new deployment requirements, meaning early-stage architectures should reserve room for future scaling.

6. Practical Recommendations

For most developers and small-to-medium teams, the Tier 1 heterogeneous approach is a sensible starting point for initial validation. Use a low-cost solution to get the model running, integrate the APIs, and establish the evaluation pipeline first. Once actual business requirements are clear, decide whether to upgrade to a higher-tier solution.

For enterprise-grade production environments, it is highly recommended to complete technical reviews covering memory budgets, communication topologies, and cost models before project initiation. Deploying a trillion-scale sparse model is a systems engineering challenge; bottlenecks can emerge in memory, communication, cooling, or power, requiring holistic planning rather than isolated optimizations.