Local Deployment of Trillion-Parameter MoE Models: A Cost and Hardware Selection Guide

    Trillion-parameter mixture-of-experts (MoE) models cannot be served from a single GPU, because memory consumption is driven by the *total* parameter count rather than the fraction of experts activated per token. This guide explains how to size a real memory floor for a ~2.8T-parameter MoE model stored in 4-bit micro-scaling format (roughly 1500 GB of weights on their own), uses a ~671B model as a point of comparison, and maps three practical deployment targets — bare minimum throughput, ~25 tokens/s for a single user, and 10 concurrent users at 25+ tokens/s — to the hardware each one demands. It also covers GPU-CPU heterogeneous inference, the four main cost drivers (accelerator count, system memory, high-speed interconnect, power and cooling), and a verification checklist for re-checking every estimate against real benchmarks once official weights are released.

    1. Why Deploying Trillion-Parameter MoE Models Locally Is So Hard

    As model sizes cross into the trillions of parameters, local deployment is no longer a matter of “buy one GPU and run it.” Take a representative mixture-of-experts (MoE) architecture with roughly 2.8 trillion total parameters: it contains hundreds of expert subnetworks, and each token is routed to only about a dozen of them during inference.

    Many people draw an intuitive but incorrect conclusion from this: if only a small slice of the model is activated, surely you only need a small slice of memory?

    The answer is no. MoE memory consumption is determined by the total parameter count, not by the activation ratio. You can load all 896 experts into storage and then pull just 16 of them into computation per token, based on routing. The activation ratio affects compute demand — how many matrix multiply-accumulate operations you perform per second — but it does nothing to reduce the weight volume that must stay resident. This distinction is the first cornerstone of any deployment cost estimate.

    2. How Much Memory You Actually Need: Budget for Total Size, Not the Active Fraction

    To estimate memory, start with the weight storage format. The current mainstream approach compresses weights to 4-bit using a micro-scaling floating-point format: every 32 4-bit elements share one 8-bit scale factor. So the effective bit width per weight element is not 4 bits, but 4 + 8/32 = 4.25 bits.

    From there you can compute a lower bound:

    1. # Memory estimate for 4-bit micro-scaling weights
    2. params = 2.8e12 # total parameter count
    3. bits_per_elem = 4 + 8 / 32 # 4-bit data + one 8-bit scale per 32 elements = 4.25
    4. weights_bytes = params * bits_per_elem / 8
    5. print(weights_bytes / 1e9) # ≈ 1487.5 GB (raw weights only, theoretical floor)

    In other words, the weights alone approach 1500 GB. That figure does not yet include non-4-bit tensors, metadata, memory alignment, the runtime workspace, or the KV cache that keeps growing during inference. To actually run the model, addressable storage must be clearly larger than this number.

    For comparison, a MoE model with roughly 671B total parameters needs about 335 GB of weights after 4-bit quantization, plus roughly 37 GB for activations and other overhead — about 400 GB in total. It typically runs comfortably on 8 high-end accelerators with 80 GB each (640 GB combined). A trillion-class model is more than four times that footprint, which firmly locks in the deployment floor. This is exactly why models of this class generally need a supernode of dozens of accelerators linked by high-speed interconnect.

    3. Three Deployment Tiers and the Hardware Strategy Behind Each

    Requirements vary enormously between users. It helps to split them into three tiers, from “can it run at all” to “does it run well,” with hardware strategy escalating accordingly.

    Tier 1: It just needs to run, speed is irrelevant.
    The only goal is fitting roughly 1500 GB of weights into addressable storage and getting tokens out of the model, even at single-digit tokens per second. The most practical approach at this tier is GPU-CPU heterogeneous inference (covered in the next section): a large-memory server holds the vast majority of the weights, while a small amount of VRAM accelerates the critical operators. This is the lowest cost of entry and is well suited to validation and learning.

    Tier 2: Around 25 tokens/s for a single user.
    At this point, grinding through CPU memory is no longer enough — you need enough memory bandwidth to feed the data. The rule of thumb in the field is that 25 tokens/s for one user corresponds to reading roughly 25 times the model’s size in data every second, which places a real, measurable demand on memory bandwidth. In practice you want as much of the weight as possible resident in high-bandwidth memory, minimizing traffic across media, and the hardware spend rises noticeably compared with Tier 1.

    Tier 3: 10 concurrent users, 25+ tokens/s per user, and no aggressive quantization.
    This is the hardest tier. Concurrency multiplies both compute and bandwidth requirements, while the quantization-precision constraint limits how far you can shrink storage. Stacked together, these demands almost inevitably require a high-bandwidth interconnected cluster built from professional-grade accelerators. This is an enterprise- or datacenter-level investment; personal local deployment is essentially impractical, and most people spread the cost by renting on demand in the cloud instead.

    4. GPU-CPU Heterogeneous Inference: Getting 1500 GB of Weights into Memory

    Heterogeneous inference is the core weapon of Tier 1. Each MoE layer has two parts: an attention backbone shared by all experts (compute-heavy and sensitive to bandwidth), and a large population of sparse experts of which only a few are activated per token (the bulk of the storage, with comparatively modest compute needs).

    A mature general pattern looks like this:

    • Keep the attention backbone on the GPU, using high-bandwidth memory for the latency-sensitive computation;
    • Offload the sparse experts to CPU memory, where a large capacity pool holds them and each one is pulled in on demand when routing selects it.

    This division of labor — GPU computes the backbone, CPU stores the experts — is a well-established, repeatedly validated pattern in the field, and it shifts most of the single-GPU capacity pressure onto inexpensive system memory. The cost is the data-transfer overhead between CPU and GPU, which limits speed. That is the physical reason Tier 1 can only promise that the model runs.

    5. Where the Money Actually Goes: The Main Cost Drivers

    Rather than quote specific prices (they fluctuate with the market, and inventing them would be misleading), it is more useful to break down the cost structure. It is driven by four lines:

    1. Number of accelerators. Determined directly by the memory floor. Whether the model runs at all depends on total storage capacity; how fast it runs depends on the accelerators and their bandwidth. This is the single largest expense.
    2. System memory. On the heterogeneous inference route, more than 1500 GB of weights must stay resident in memory, so a large-capacity memory server is mandatory.
    3. High-speed interconnect. Trillion-class MoE models are extremely sensitive to multi-device communication: after routing, each token’s activations must be moved to the device holding the target expert and moved back once the computation finishes. If interconnect bandwidth is insufficient, communication latency directly eats into compute throughput — which makes high-bandwidth inter-chip links (high-bandwidth inter-chip bus approaches and similar) a hidden but critical investment.
    4. Power and cooling. The power draw and heat dissipation of dozens of accelerators are not negligible. For a self-built cluster, this is an ongoing operating cost.

    For individuals and small teams, the pragmatic path is to minimize cost with a heterogeneous setup during validation, then use on-demand cloud rental during steady-state use to amortize hardware depreciation — rather than committing to a one-time heavy capital outlay.

    6. Recommendations and a Verification Checklist

    Before you obtain the model weights and start real measurements, put together a checklist:

    • Confirm the total parameter count and quantization format, then recompute the weight floor from them (remember: total size, not the activated portion);
    • Reserve headroom for the KV cache and runtime overhead — real storage requirements exceed the raw weights;
    • Decide which tier you are targeting, “just runs” or “high concurrency at high speed”; the two differ by an order of magnitude in budget;
    • Assess interconnect requirements — for any multi-device deployment, include communication bandwidth in your bottleneck analysis;
    • Run small-sample benchmarks first, and validate each estimate item by item against real tokens/s figures and memory consumption.

    One important caveat: until the official full weights and technical details are public, all of the speed, memory, and hardware numbers above are engineering estimates based on publicly available information. Their value lies in providing a measurement methodology and a sense of scale. Once the weights are officially released, calibrate every one of these figures against measured benchmarks before making a purchase or rental decision.