How Much Hardware Does It Cost to Deploy a Trillion-Parameter MoE Model Locally?

1. Breaking Down the Problem: What Does “Running” Actually Mean?

Before discussing the local deployment cost of a trillion-parameter Mixture-of-Experts (MoE) model, one question must be answered first: what counts as a successful deployment? Different goals lead to hardware budgets that differ by orders of magnitude. In engineering practice, there are three common tiers:

Tier 1: It runs, and speed does not matter. The model loads completely, generates tokens one by one, and produces semantically correct output. A few tokens per second is acceptable. The bottleneck at this tier is storage capacity, not compute.

Tier 2: Single-user inference at 25 tokens per second. 25 tokens per second is roughly double the comfortable human reading speed, making it the minimum bar for a usable experience. The bottleneck here shifts from capacity to the balance between memory bandwidth and compute.

Tier 3: 10 concurrent users, each still getting 25 tokens per second or more, with quantization no lower than a specified level. This tier requires the system to maintain throughput under batched inference. The bottleneck moves to communication bandwidth, batch scheduling, and KV cache capacity.

Each tier is analyzed below. It should be noted that since the full weights of models at this scale are not yet widely public, the speed, memory, and hardware figures discussed below are estimates based on publicly available architecture information. Actual deployments must be verified with benchmarks.

2. The Baseline for Memory Usage: Why MoE Cannot Be Calculated by Activated Parameters

Take a typical trillion-parameter MoE architecture as an example: roughly 2.8 trillion total parameters, 896 experts, 16 experts activated per token, meaning an activation ratio of about 1.79%. There is a common misconception worth highlighting:

MoE deployment memory usage is calculated by total parameters. The activation ratio affects compute consumption only, not storage consumption.

The reason lies in the routing mechanism. During inference, each token only passes through 16 experts, but you cannot predict which 16 experts the next token will route to. Therefore, all 896 experts’ weights must reside in addressable storage, and the relevant experts are fetched based on routing decisions at inference time. This means memory requirements are determined by total parameter count.

Now consider quantization. Assume the model was trained with quantization-aware training from the supervised fine-tuning stage, using a 4-bit weight and 8-bit activation scheme. Converting 2.8 trillion parameters to 4-bit gives a theoretical minimum of:

2.8e12 × 4 bit ÷ 8 = 1400 GB

However, 4-bit floating-point formats typically attach an 8-bit scale for every 32 elements (following the OCP specification). The effective bit width should be calculated as 4.25 bits:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This does not yet account for non-4-bit tensors (such as normalization layers and embedding layers), metadata, memory alignment, runtime workspace, and KV cache. For reference, a well-known 670B-class MoE model in the industry requires about 685GB for main parameters plus auxiliary parameters in FP8 format. After 4-bit quantization, it still needs roughly 335GB for weights plus 37GB for activations — about 400GB total to run on eight 80GB accelerator cards. A trillion-parameter model is more than four times that size, so the lower bound for a full deployment can be inferred: close to 1.5TB for weights alone.

3. Tier 1: Fitting 1.5TB into Addressable Storage

The goal here is simple: fit nearly 1500GB of weights into addressable storage and get the model to emit tokens.

The most straightforward approach is heterogeneous GPU + CPU inference. In MoE, each layer has a shared attention component that is compute-intensive and should stay on GPUs for high-bandwidth processing. The 896 sparse experts, while accounting for the bulk of storage, only activate 16 at a time, so they can be offloaded entirely to CPU memory on a large-memory server.

The hardware profile for this approach is roughly:

• GPU: 1–2 cards, responsible for attention computation and routing. GPU memory requirements are relatively manageable.
• CPU memory: At least 2TB recommended, leaving headroom for weights, KV cache, and runtime workspace.
• Storage: NVMe SSD for weight loading. 4TB or more recommended.
• CPU: Core count does not need to be extreme, but memory bandwidth matters. Multi-channel DDR5 is preferred.

At this tier, the main cost is large-capacity memory. 2TB of DDR5 ECC memory can account for more than half of the total system cost. The advantage is that GPU requirements are minimal, making the overall cost an order of magnitude cheaper than an all-GPU solution. The trade-off is speed — since expert weights need to travel from CPU memory over PCIe to the GPU for computation and then back, per-token latency will be noticeably high. Single-digit tokens per second is the expected outcome.

4. Tier 2: Balancing Compute and Bandwidth for 25 TPS Single-User Inference

To go from single-digit TPS to 25 TPS, the core challenge becomes: how to reduce bottlenecks in the expert weight transfer path.

Several key optimization directions:

1. Increase GPU-resident ratio. Keep frequently accessed experts (hot experts) in GPU memory and leave cold experts in CPU memory. Routing distributions are typically uneven, so this strategy can significantly reduce PCIe traffic.

2. Trade-offs between quantization level and precision. More aggressive quantization (lower bit widths) reduces data transfer volume but sacrifices precision. The requirement that “quantization level must not fall below a specified tier” means compression cannot be unlimited. Optimization must happen within precision constraints.

3. Guaranteeing attention compute capacity. Attention is computed for every token and represents the dense computation path. Its compute capacity directly caps decoding speed. If a single card lacks sufficient compute, attention becomes the bottleneck no matter how fast expert transfer is.

4. KV cache management. In long-context scenarios, the KV cache can grow rapidly. It requires paged management, quantized storage, or even offloading to CPU memory.

The hardware profile for this tier is roughly:

• GPU: 4–8 high-memory accelerator cards, with total GPU memory of at least 320GB recommended.
• CPU memory: 1–2TB.
• Interconnect: High-speed GPU-to-GPU interconnect (NVLink-class) is required to prevent cross-card attention communication from slowing things down.

Cost rises significantly compared to Tier 1, but it remains within the range of a single machine. The actual achievable TPS depends heavily on implementation quality — operator fusion, expert cache hit rate, and batching strategy can all cause several-fold differences.

5. Tier 3: 10 Concurrent Users at 25 TPS — Communication Becomes the Main Battlefield

When concurrency increases from 1 to 10, the nature of the problem changes. Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth: with 896 experts distributed across multiple cards, each token must have its activations shipped to the cards hosting the routed experts after routing, and the results shipped back after computation. The higher the concurrency, the worse the contention and latency of these transfers become. Communication latency directly eats into compute.

The solution is a supernode architecture: use high-speed interconnect to enlarge the communication domain so that expert activation transfers never leave a single communication domain. A common industry recommendation is to build a supernode with 64 or more accelerators. The hardware profile for this tier:

• Accelerators: 64 or more, forming a unified high-speed interconnect domain.
• Interconnect bandwidth: Per-card interconnect bandwidth needs to reach hundreds of GB/s.
• Total memory: With 1.5TB of weights plus KV cache and multi-replica headroom, 4TB or more is recommended.
• Power and cooling: A full rack can draw tens of kilowatts and requires dedicated data center conditions.

At this tier, the cost shifts from “buying a server” to “building a cluster,” involving racks, power supply, cooling, network switching, and full infrastructure. For the vast majority of individual developers and small-to-medium teams, this tier is not economically viable for local deployment. It is better suited for on-demand use through cloud-based elastic compute.

6. Rules of Thumb for Cost Estimation

Based on the analysis above, several practical estimation principles emerge:

Rule 1: Memory is calculated by total parameters, not activated parameters. This is the most easily underestimated factor in MoE deployment and can throw the budget off by an entire order of magnitude.

Rule 2: Quantization bit width must include scale factors and metadata. A nominal 4-bit format is effectively about 4.25 bits. Adding non-quantized tensors and runtime overhead, actual usage is typically 10%–20% higher than the theoretical value.

Rule 3: Communication bandwidth determines the concurrency ceiling; compute determines single-stream speed. A single user can get by with heterogeneous inference. Multi-user concurrency requires solving the interconnect problem.

Rule 4: Choose your tier separately — do not try to solve everything at once. Start with Tier 1 to validate feasibility, optimize for Tier 2 to improve experience, and only then evaluate cluster solutions for Tier 3. The return on investment varies enormously between tiers.

Rule 5: All estimates must be verified with benchmarks. Weight formats, operator implementations, and batching strategies all affect the final numbers. Paper estimates are only useful for budget ranges, not as acceptance criteria.

7. Summary

Deploying a trillion-parameter MoE model locally is fundamentally a multi-constraint optimization problem involving storage capacity, memory bandwidth, interconnect bandwidth, and compute balance. The three tiers do not simply represent low, medium, and high configurations — they represent three different system architecture approaches: heterogeneous offloading solves the capacity problem, expert caching and quantization solve the single-stream speed problem, and supernodes solve the concurrent communication problem. Knowing which tier you are in is the key to spending your budget on the real bottleneck.