Deploying a Trillion-Parameter MoE Model Locally: Hardware Costs from Barely Running to High Performance

Trillion-parameter mixture-of-experts (MoE) models are becoming the new normal in the open-source ecosystem. When a model reaches 2.8 trillion parameters with nearly 900 experts, the hardware bill behind the phrase “local deployment” is far more complex than deploying a dense model. This article breaks the problem down into calculable components: how to estimate the minimum memory footprint, what each of three deployment tiers actually requires, and where the money really goes.

One caveat before we start: aside from publicly available architecture descriptions and quantization format specifications, all speed, memory, and hardware figures below are engineering estimates based on public information. They will need to be validated with benchmarks once the actual weights are released. The value of these estimates is in establishing orders of magnitude, not exact quotes.

1. Getting the Math Right: Minimum Memory for a Trillion-Parameter MoE

1.1 Total Parameters Dictate Storage; Active Ratio Dictates Compute

The most common misconception about MoE architectures is that “fewer active parameters means less memory.” That is not true. Take a 2.8-trillion-parameter model with 896 experts and 16 experts activated per token as an example. The expert activation ratio is roughly 1.79%. But this ratio only affects compute consumption—it does not affect storage requirements. During inference, all 896 experts must reside in addressable storage. When routing selects 16 of them, the corresponding weights are fetched for computation. Memory planning must therefore be based on the total parameter count.

At the architecture level, the backbone of this class of models typically consists of linear-attention-style components (such as KDA, or Kimi Delta Attention) combined with Attention Residuals, with Gated MLA appearing as one of the internal components. These structural details affect the shape of the per-layer compute graph and the size of the KV cache, but the bulk of the weight volume still sits in those 896 experts.

1.2 Quantization Levels and Real Memory Footprint

Assume the model was trained with quantization-aware training from the SFT stage, using MXFP4 for weights and MXFP8 for activations. Converting 2.8 trillion parameters to 4-bit gives a raw data floor of approximately:

2.8T × 4 bit ÷ 8 ≈ 1400 GB

But MXFP4 is not pure 4-bit. Under the OCP microscaling specification, every 32 4-bit elements carry one 8-bit scale factor, yielding an effective bit width of:

4 + 8/32 = 4.25 bit
2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

That 1487.5 GB is still just the net data for weight elements plus scale factors. It does not account for non-4-bit tensors (some normalization layers and embedding layers may retain higher precision), metadata, memory alignment padding, runtime workspace, and KV cache. In practice, you should reserve an additional 10%–20% headroom on top of this figure.

1.3 Using a Known Model as a Yardstick

A useful calibration point comes from a public reference case: a 671B-parameter MoE model in FP8 format takes about 685 GB for the main parameters plus MTP parameters. With 4-bit AWQ quantization, it drops to roughly 335 GB of weights plus 37 GB of activations—about 400 GB total—which is what allows it to run on eight 80 GB accelerators (640 GB combined).

By comparison, 2.8 trillion parameters is more than four times the size of 671B. Even at just over 4-bit quantization, the weights are in the 1.5 TB range. That sets the floor for full deployment—this is not a problem a single card or a single workstation can solve. It necessarily enters the realm of multi-node, multi-accelerator, or even supernode deployments.

1.4 The Underappreciated Cost of Communication

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple cards, each token’s activations must be sent to the cards hosting the selected experts, computed, and sent back after routing. If this round trip crosses nodes over standard Ethernet, communication latency will directly eat into compute utilization—you end up with cards waiting for data rather than computing.

The common industry approach is therefore to use high-speed interconnects (NVLink or equivalent) to make the communication domain as large as possible, keeping expert transfers within a single domain. This is why official-grade recommendations land at “a supernode of 64 or more accelerators”—not because compute is insufficient, but because compute utilization collapses when the communication domain is too small.

2. Tier 1: Just Make It Run, Speed Not a Concern

2.1 Goal Definition

The only goal at this tier is to fit roughly 1,500 GB of weights into addressable storage and have the model produce tokens reliably, even at single-digit tokens per second. This tier is suitable for validating weight integrity, running functional smoke tests, and getting the inference pipeline working end to end.

2.2 Heterogeneous Inference: Splitting Work Between GPU and CPU

The most straightforward approach is heterogeneous CPU-GPU inference. The core insight is that an MoE layer has a shared attention component (in this model class, KDA plus Gated MLA) that is compute-intensive but accounts for a small fraction of total parameters, so it stays on the GPU for high-bandwidth computation. The 896 sparse experts, which dominate storage but only activate 16 at a time, are offloaded entirely to CPU memory on a large-memory server.

This approach is represented by the familiar “expert offload” class of inference frameworks. A typical configuration is a dual-socket high-core-count server CPU, large-capacity DDR5 memory (512 GB minimum recommended, 1 TB or more preferred), and one or several consumer or professional GPUs.

2.3 Expected Performance Range

Public results for a comparable approach on a 671B model at 4-bit quantization (about 377 GB of weights) show roughly 200+ tokens per second in prefill and about 14 tokens per second in decode, under expert selection constraints. Extrapolating to 2.8 trillion parameters and 1.5 TB of weights, decode speeds in the single digits are a reasonable expectation.

The cost center at this tier is memory, not GPU. A dual-socket server with 1 TB of memory plus a mid-to-high-end GPU typically lands in the range of tens of thousands of US dollars—by far the cheapest option among the three tiers.

2.4 Engineering Considerations

• Memory bandwidth is the critical bottleneck. DDR5 bandwidth is far lower than HBM. During decode, every token requires loading the activated expert weights from memory to the compute unit, so bandwidth directly caps the speed.
• Expert swapping policies need tuning. Which experts stay GPU-resident and which stay on the CPU should be decided based on the routing distribution of actual requests.
• Do not expect this tier to support interactive applications. Its role is validation and experimentation.

3. Tier 2: 25 Tokens per Second for a Single User

3.1 Why 25 tps Is a Watershed

Twenty-five tokens per second is roughly the lower bound for “reading fluency”—the average human reads at a speed that makes 25 tps feel like generation is slightly ahead of reading, with no perceptible waiting. Achieving this speed essentially rules out pure CPU offload. More of the computation must stay in high-bandwidth accelerator memory.

3.2 Configuration Approach: Keep Hot Experts in Accelerator Memory

A viable path is “partial expert residency on GPU”: by analyzing the request distribution, identify the subset of high-frequency experts and keep them accelerator-resident, while cold experts remain in CPU memory. Combined with quantization at no less than 4-bit (and keeping some layers at 8-bit where necessary), this reduces the per-token weight transfer volume.

On the hardware side, this tier typically requires a single node with eight high-bandwidth accelerators, with combined memory ranging from 640 GB to 1.5 TB depending on the card type, plus high-speed inter-card interconnect. If accelerator memory is still insufficient for all weights, CPU memory serves as secondary storage, but the CPU-to-GPU transfer path must be wide enough (PCIe 5.0 or above, or multi-card NVLink bridging).

3.3 Cost Range

A single-node eight-accelerator high-bandwidth system typically costs well into the six figures in US dollars, depending on the card type and system vendor. With supporting power, cooling, and data center conditions, the total investment for this tier is roughly in the $200,000–$400,000 range. Compared to Tier 1, the money is primarily spent on keeping computation in accelerator memory.

3.4 Key Tuning Points

• Expert residency policy: tier experts by routing frequency—hot experts accelerator-resident, warm experts swapped on demand, cold experts CPU-only.
• KV cache management: in long-context scenarios, the KV cache expands rapidly and requires paged management and quantized compression.
• Batching strategy: with a single user, batch size is 1, meaning GPU utilization is naturally low. Techniques like speculative decoding are needed to improve effective throughput.

4. Tier 3: 10 Concurrent Users at 25+ tps Each

4.1 Translating Requirements into Engineering Targets

Ten concurrent users at 25 tps each means total throughput of at least 250 tps, with quantization no lower than 4-bit. This volume exceeds the comfort zone of a single 8-accelerator node and enters multi-node or supernode territory.

4.2 Why a Supernode Is Necessary

Returning to the communication analysis in Section 1: when experts are distributed across multiple cards, all-to-all communication volume grows linearly with concurrency. At 10 concurrent streams, if the communication domain spans nodes over a standard network, latency quickly becomes the bottleneck. The recommended architecture for this tier is therefore a supernode of 64 or more accelerators, using NVLink-class interconnect to keep the communication domain within a single enclosure.

A rough capacity calculation: with 80 GB per card and 64 cards, the combined memory is about 5 TB. After accounting for KV cache (which can consume hundreds of GB to 1 TB under 10 concurrent long-context streams) and runtime overhead, the remaining space is just enough to hold 1.5 TB of weights with headroom. This is where the “64 cards” figure comes from—it is the balance point among weights, KV cache, and workspace.

4.3 Cost Range

A 64-accelerator supernode typically costs in the millions of US dollars, and the cards are not the only line item: high-speed interconnect switches, facility power (power draw at this tier easily exceeds tens of kilowatts), liquid cooling, and an operations team are all part of the bill. For most teams, this tier means building a private data center or renting high-density compute capacity on a long-term basis.

4.4 Critical Software Stack Capabilities

• Expert parallelism (EP) strategy: how the 896 experts are partitioned across 64 cards directly affects load balancing.
• Communication-computation overlap: all-to-all transfers must be pipelined with expert computation, otherwise bandwidth utilization cannot reach acceptable levels.
• Request scheduling: continuous batching and priority scheduling under 10 concurrent streams.
• Quantization consistency: the mixed-precision path of MXFP4 weights with MXFP8 activations requires full support from the inference engine.

5. Beyond the Hardware: Three Easily Overlooked Hidden Costs

First, power and cooling. Tier 3 power draw is measured in tens of kilowatts. Depending on industrial electricity rates and PUE, annual electricity costs can reach tens of thousands of US dollars—and that does not include the CAPEX of the cooling infrastructure itself.

Second, storage and loading. Loading 1.5 TB of weight files, version management, and multi-replica distribution all require fast local storage (NVMe arrays). A cold start can take tens of minutes.

Third, tuning labor. Going from “it runs” to “it runs well” involves extensive benchmarking, routing statistics, and parallel strategy tuning. This labor cost is often absent from budget spreadsheets, but it represents a meaningful share of the total.

6. Deployment Recommendations: Match the Tier to Your Needs

Tier	Goal	Typical Configuration	Cost Range
1	Runs, speed not a concern	Dual-socket large-memory server + 1–2 GPUs, expert CPU offload	Tens of thousands of USD
2	25 tps for a single user	Single node, 8 high-bandwidth accelerators, hot experts GPU-resident	$200K–$400K
3	10 concurrent × 25 tps	Supernode with 64+ accelerators, high-speed interconnect	Millions of USD

One final note: the memory, speed, and cost figures in the table above are estimates based on publicly available architecture information. The real numbers will need to be validated against standard benchmarks once the weights are released—especially decode speed, whose sensitivity to implementation details far exceeds most people’s expectations. Before any budget approval, run a small-scale validation. It is more reliable than any estimate on paper.