Running Massive MoE Models Locally: Hardware Cost Estimates and Tiered Deployment Options

1. A Core Premise First: The Challenge of Local Deployment Isn’t “Running” — It’s “Running Well”

The new generation of ultra-large-scale Mixture-of-Experts (MoE) models now widely discussed across the industry share several traits: trillion-scale total parameters, sparse activation, low activation ratios, and quantization-aware training. Take the recently much-discussed 2.8-trillion-parameter model as an example. Its design goal is to reduce per-token inference compute through a sparse expert structure while maintaining high generation quality. That does not mean the bar for local deployment is low — quite the opposite: it raises the requirements for total VRAM, storage bandwidth, intra-node interconnects, and inference frameworks.

Before calculating costs, one commonly misunderstood point must be clarified: a MoE model’s VRAM footprint is determined by its total parameter count, not by the number of “activated parameters.” Even if a 2.8-trillion-parameter model activates only about 1.79% of its experts for each token, the inference system still has to keep all expert weights in storage that can be addressed quickly. In other words, sparse activation reduces compute density, not weight residency requirements.

When evaluating local deployment costs, therefore, you cannot look only at compute metrics such as “how many tokens per second.” Three conditions must be satisfied simultaneously:

1. All weights can be loaded in full;
2. The medium holding the weights offers sufficiently high read/write bandwidth;
3. Communication latency between GPUs does not become a serious bottleneck during expert routing.

Together, these three points set the lower bound for hardware selection, quantization level, and deployment design.

---

2. Starting from Weight Size: How Much VRAM or Memory Does 2.8T Parameters Actually Need

2.1 The Direct Relationship Between Total Parameters and VRAM

Assume the model has 2.8T total parameters — roughly 2.8 × 10¹². Stored at standard FP8 precision, the weights alone would theoretically occupy:

2.8T × 1 Byte = 2800 GB

Since real inference also requires activations, optimizer states, communication buffers, and runtime metadata, a full FP8 deployment needs at least 3 TB of effective storage. That figure far exceeds the combined VRAM of a common 8-GPU — or even 16-GPU — server.

2.2 Weight Footprint After Low-Bit Quantization

With a 4-bit weight format, the raw weights compress to roughly:

2.8T × 0.5 Byte = 1400 GB

In practice, however, industrial-grade 4-bit formats are rarely pure 4-bit storage. Some microscaling formats, for instance, attach one 8-bit scale factor to every 32 4-bit elements, pushing the effective width from 4 bits to about 4.25 bits. Recalculating on that basis:

2.8T × 4.25 bit ÷ 8 = 1487.5 GB

And that still excludes:

• Non-4-bit tensors such as the routing network, embedding layers, and normalization layers;
• Alignment padding and duplicated data produced by model sharding;
• The KV cache at inference time;
• Multi-GPU communication buffers and workspaces.

So even with an aggressive 4-bit weight format, merely “fitting” the model requires at least 1.5 TB of available storage. If you also want acceptable inference speed, the actual VRAM or high-speed memory requirement will be higher still.

2.3 A Straightforward Comparison with the Previous-Generation 671B Model

To grasp the scale, it helps to look at the engineering experience around the previous-generation 671B MoE model. At FP8, its main weights occupy about 685 GB; after 4-bit quantization, roughly 335 GB of weights plus 37 GB of activations — about 400 GB in total — is needed just to run, and only barely, on eight 80 GB accelerator cards.

A 2.8T-parameter model is more than four times that size. Even assuming more advanced compression, its minimum deployment threshold will far exceed the VRAM capacity of an ordinary single GPU server. This is exactly why official deployment guidance points straight to supernodes with 64 or more cards, rather than the usual single-node 8- or 16-card setups.

---

3. The Hidden Costs of MoE Architecture: Expert Routing and All-to-All Communication

3.1 Why Large-Scale MoE Is Extremely Sensitive to Communication Bandwidth

In a dense model, each layer’s computation can be completed within one GPU or a small group of GPUs, and communication is concentrated at tensor-parallel or pipeline-parallel boundaries.

Large-scale MoE inference follows a different logic:

1. Each token entering a MoE layer first goes through routing computation;
2. The router picks a few experts out of hundreds;
3. The token’s hidden state is sent to the physical devices hosting those experts;
4. The experts run the forward computation;
5. The results are sent back to the original device.

If experts are spread across dozens or even hundreds of GPUs, with each card holding only a subset of expert weights, then every token’s processing triggers multiple cross-device data transfers. With a structure of 896 experts and 16 activated per token, this communication pattern is not occasional — it happens for every token, at every layer.

3.2 Why a Supernode, Not an Ordinary Cluster

Ordinary Ethernet or medium-bandwidth interconnects have markedly higher latency for cross-node communication than intra-node interconnects. Once expert routing paths involve multiple cross-node hops, end-to-end latency accumulates quickly, and per-token generation time ends up being dominated by communication rather than computation.

High-performance deployment of large-scale MoE therefore typically requires:

• Keeping expert weights within a single high-speed interconnect domain wherever possible;
• Using NVLink, ultra-high-speed switches, or equivalent mechanisms to build a large-bandwidth, low-latency communication domain;
• Avoiding all-to-all traffic that crosses too many network tiers.

This is why supernodes with 64+ cards are the recommended starting point for this class of model. It is not that “you need 64 cards to fit the weights” — it is that “you need 64 cards to get acceptable communication and parallel efficiency.”

---

4. Tier 1: Just Make It Run, Speed Be Damned

For individual researchers, teaching labs, or early-stage evaluations, the first goal is simply to get the model to emit tokens, even slowly.

4.1 When VRAM Falls Short, Can CPU Memory Fill the Gap?

The most direct approach is heterogeneous inference: keep the compute-intensive, parameter-light components on the GPU, and offload the expert weights — which dominate storage but are only partially activated at any moment — to CPU memory.

In this scheme:

• Compute-intensive parts such as attention modules, shared layers, and input/output embeddings stay on the accelerator;
• Most of the sparse expert layers live in host memory;
• At inference time, only the expert weights selected by the router are loaded into the GPU for computation.

This approach has already been validated by community practice with the previous-generation 671B model. A dual-socket high-end server CPU paired with a consumer or workstation GPU can run the 4-bit quantized MoE model — slowly, but with complete inference.

4.2 Where a Pure Heterogeneous Setup Struggles at 2.8T

Transplanting the same approach to a 2.8T-parameter model magnifies the problems considerably:

• Total weights exceed 1.4 TB. Even parked in CPU memory, that calls for at least 1.5 TB of usable memory, which typically means fully populating a dual-socket server with high-capacity DIMMs;
• The granularity of each expert load, the caching strategy, and memory bandwidth all become bottlenecks;
• Even though only a few experts activate per token, each individual expert’s weights are still sizable, and CPU-to-GPU transfer time slows generation dramatically;
• The routing module itself must exchange data between GPU and CPU frequently.

As a result, the definition of “it runs” gets compressed to extremely low speeds — single-digit tokens per second, or lower. For interactive use cases this is nearly useless, but it still has value for weight verification, one-off inference, academic observation, and early research.

4.3 Reference Minimum Hardware for Tier 1

From an engineering-feasibility standpoint, a minimum Tier 1 setup would likely need:

• At least 1.5 TB of usable DRAM;
• At least one mid-to-high-end GPU for attention and routing;
• A motherboard with the highest available PCIe bandwidth, to ease the CPU-GPU transfer bottleneck;
• An inference framework that supports MoE state management and on-demand weight loading.

Total hardware cost would likely land in the range of tens of thousands of US dollars, driven mainly by the large-memory server and the high-end accelerator rather than by GPU count itself.

---

5. Tier 2: Usable by a Single User at 25 tokens/s

For developers, researchers, or heavy individual users, 25 tokens/s is a practical threshold for interactive use. At that speed the model can produce a several-hundred-word answer within seconds — suitable for debugging, writing assistance, and automation tasks.

5.1 Speed Targets Impose Harder Constraints on the Memory Medium

Once the goal shifts from “it runs” to “it runs smoothly,” CPU-memory-based schemes are essentially out. The reason:

generation speed = compute time + weight transfer time + communication time

In a heterogeneous setup, weight transfer dominates absolutely. Even if a single forward pass is fast, every generated token must wait for expert weights to be pulled from system memory onto the accelerator, and overall throughput collapses to single-digit tokens/s.

To run at 25 tokens/s, cross-media transfers must be slashed, with weights resident in high-bandwidth VRAM as much as possible.

5.2 Must All Weights Reside in GPU VRAM?

Ideally, at minimum, all frequently activated weights and the associated routing structures should live in high-speed VRAM. For an ultra-large MoE, even though each token activates only a few experts, factoring in batching, time slicing, and cache hit rates, you still need enough GPU VRAM to hold most of the active weights. Otherwise, even with a low average activation ratio, constant weight swapping will cause stutter.

5.3 Is Multi-GPU Pooling Enough?

Within a single node, multiple accelerators can form one large VRAM pool over a high-speed interconnect. For small or mid-sized MoE models, that expands usable VRAM significantly. But once total weights exceed 1.4 TB, even a single node with 8 to 16 cards falls short of full residency.

Assume 80 GB per card: eight cards provide only 640 GB against roughly 1.5 TB of quantized weights — clearly not enough. Sixteen 80 GB cards reach just 1.28 TB — still insufficient. Tier 2’s minimum system may therefore be forced up to multi-node deployment.

5.4 Quantization Level vs. Speed

At a 25 tokens/s target, quantization should not go too low, for two main reasons:

1. Excessively low weight precision degrades generation quality, most visibly in long-form text, reasoning, and code generation;
2. Some low-bit formats add extra computation and memory-access overhead during dequantization, hurting sustained throughput.

Tier 2 is therefore more likely to use a mid-level quantization — for example, an industrial-grade scheme between 4-bit and 6-bit — rather than forcing the model into an extremely narrow bit width.

5.5 Budget and Hardware Form Estimate

Weighing weight footprint, communication needs, and output speed, Tier 2 would likely require:

• Multiple high-speed interconnected servers with combined VRAM approaching or exceeding 1.5–2 TB;
• High-speed interconnect inside each machine;
• Sufficiently high bandwidth and low latency between machines;
• An inference framework supporting stable expert parallelism and weight sharding.

The overall budget could reach the equivalent of several hundred thousand US dollars, depending on accelerator pricing, server configuration, and the interconnect approach. Note that the cost is not linear: once you move from a single node to multiple nodes, hardware costs rise — and so do deployment complexity and operating costs.

---

6. Tier 3: 10 Concurrent Users, Each at No Less Than 25 tokens/s

Tier 3 means serving multiple concurrent requests while preserving output quality and speed. The challenge is no longer just single-request latency, but total system throughput, scheduling strategy, and communication efficiency.

6.1 Concurrency Amplifies Every Resource Cost

If a single request generates at 25 tokens/s, ten concurrent requests imply 250 tokens/s of aggregate throughput. If all ten are decoding at the same time, the system must juggle simultaneously:

• 10 KV caches;
• 10 independent routing decisions;
• 10 sets of expert computation requests;
• 10 sets of results to return.

With a dense model, concurrency can amortize compute cost through larger batch sizes. With a large-scale MoE, however, growing batch sizes make expert load balancing harder and increase communication volume, so system efficiency can actually decline.

6.2 Why Concurrent Deployment Is Harder

In a single-request scenario, the system can pick the optimal path and minimize unnecessary communication. A multi-request scenario resembles a real service environment, where the system must cope with:

• Varying token lengths across requests;
• Requests at different generation stages;
• Expert load that fluctuates sharply over time;
• High-speed interconnect links contended by many requests at once.

Tier 3 therefore generally calls for a dedicated serving-grade inference framework with dynamic batching, request queuing, expert caching, and load balancing. Ordinary single-node inference scripts rarely meet the bar.

6.3 Quantization Must Not Fall Below the Engineering Floor

Tier 3 is not just about “generating output” — it demands stable quality under long-term service. Quantization therefore cannot be compressed without limit. The usual recommendation is to stay at an engineering level that does not materially impair core capabilities — mainstream schemes between 4-bit and 8-bit — and to calibrate and validate against the target workload.

---

7. A Complete Breakdown of Where the Costs Come From

7.1 Hardware Costs

Hardware spending concentrates in:

• Accelerators: count and model directly determine total VRAM;
• Servers: dual-socket or multi-socket CPU servers to host multiple accelerators and high-speed interconnects;
• Memory: if any part of the weight set lives in system memory, large quantities of high-capacity DIMMs are needed;
• Storage: model weights, datasets, KV cache persistence, and snapshots call for fast solid-state drives;
• Networking: multi-node deployments need 400G, 800G, or faster fabrics such as RoCE or InfiniBand.

For Tier 2 and Tier 3, the real cost is rarely one or a few accelerator cards — it is the cost of the entire compute cluster.

7.2 Interconnect Costs

Large-scale MoE cannot reach high performance over traditional Ethernet. Both inside a supernode and across nodes, you need high-density, low-latency interconnect solutions. This portion of the budget is typically on par with — sometimes higher than — the accelerator cost itself.

7.3 Engineering and Operations Costs

Local deployment is more than buying hardware. You also need:

• An engineering team fluent in large-scale parallel inference;
• Quantization and tuning targeted at the specific hardware;
• Inference service setup, load testing, and capacity scaling;
• Failure recovery and version upgrade mechanisms.

For most small and mid-sized organizations, this operational overhead may deserve even more attention than the hardware bill.

---

8. Making Sound Decisions Before the Weights Are Released

At this stage, the full weights and technical reports for the relevant models have not been officially released, so purchasing hardware now carries significant uncertainty. The more prudent path:

1. Validate model quality first through the official API or cloud-hosted access;
2. Wait for the actual weights to become available and run benchmarks;
3. Focus on VRAM footprint, per-layer tensor shapes, communication patterns, and post-quantization quality changes;
4. Settle the final hardware plan based on measured data.

Individual users should watch for community-provided low-spec deployment options down the road rather than committing a large budget up front. Enterprise users are better served by an elastically scalable route that avoids locking in hardware configurations too early.

---

9. Conclusion

Local deployment of trillion-parameter MoE models has become a “systems engineering” problem, no longer a simple single-node GPU question. The core difficulties are oversized total weights, extreme interconnect requirements, and scheduling complexity under concurrency.

Broken down by tier:

• “Just make it run”: barely achievable with a CPU-GPU heterogeneous setup;
• Single user at 25 tokens/s: requires multi-GPU, possibly multi-node, VRAM residency;
• 10 concurrent users at 25 tokens/s each: requires a mature serving-oriented inference architecture and a complete high-speed interconnect environment.

Until the actual weights are released, the most sensible strategy is to keep tracking the technology, wait for real benchmark results, and plan the budget in phases. Model capability matters — but whether the model can be deployed stably at a controllable cost is what ultimately determines whether it reaches production.