Local Deployment of Trillion-Parameter MoE LLMs: Three Hardware Tiers and Cost Estimates

    This article explains why total parameters, not active parameters, set the memory floor for local MoE deployment, then breaks down three practical hardware tiers: a minimal CPU-GPU heterogeneous setup, a single-user 25 token/s configuration, and a 10-concurrent-user enterprise supernode. It also provides order-of-magnitude cost estimates and three simple formulas for sizing future ultra-large models.

    1. What You Are Deploying: Scale Sets the Lower Bound

    Before discussing cost, you need to understand three basic facts about this class of model. They directly determine the floor price of local deployment.

    Fact 1: Total parameter count determines the VRAM/memory lower bound, not the number of active parameters. Take a currently public ultra-large MoE model as an example: its total parameter scale has reached 2.8 trillion, with hundreds of experts per layer (estimated at 896 in this example). Each token routes to only about 16 of them, for an expert activation ratio of about 1.79%. Many people new to MoE assume that “only 1.79% is activated, so VRAM demand is also 1.79% of the total.” This is the most common misconception. During inference, all expert weights must reside fully in addressable storage. Routing decides which experts are used, not which experts are stored. The activation ratio affects compute consumption per forward pass, not storage occupancy.

    Fact 2: After 4-bit quantization, weight size has a simple lower-bound formula. The model was trained with quantization-aware training from the pretraining stage, using 4-bit microscaling floating-point weights (MXFP4 in the OCP specification) with 8-bit activations. A rough estimate:

    • Raw weights: 2.8T × 4 bit ÷ 8 = about 1400 GB
    • Including one 8-bit scale per 32 elements: about 2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB
    • Not yet included: non-4-bit tensors (attention-related components usually use higher precision), metadata and alignment overhead, runtime workspace, and KV cache

    In other words, weights alone approach 1.5 TB, with no redundancy margin at all.

    Fact 3: A reference point calibrates intuition. Consider another publicly discussed 671B-class MoE model: in 8-bit floating-point format, main parameters plus MTP parameters total about 685 GB. Even with aggressive 4-bit compression, it still needs about 335 GB of weights plus about 37 GB of activations, roughly 400 GB in total, to run on eight high-end accelerators (about 640 GB of VRAM combined). The target model is more than four times larger, so the hardware floor for a “full-capability deployment” is naturally more than four times higher. This is the logical starting point for all cost estimates.

    2. Tier 1: It Just Needs to Run, Speed Does Not Matter

    This tier has only one goal: fit nearly 1500 GB of weights into a machine and make the model emit tokens, even if only single-digit tokens per second.

    Recommended approach: CPU-GPU heterogeneous inference. The core idea is to split MoE computation into two categories:

    Component Characteristics Placement
    Shared backbone (attention / linear attention modules) Compute-intensive, required for every token GPU memory
    Sparse expert layers (16 of 896 active) Dominant storage, sparse per computation CPU memory

    The inference flow is roughly as follows: the shared part runs quickly on the GPU, the activated values from expert layers are sent over a high-speed bus to the CPU side with large memory, and the CPU computes results and sends them back. Open-source projects have already validated this route. Public measurements show that a dual-socket server CPU paired with a single consumer-grade GPU can run a 671B-class 4-bit model at about 14 tokens/s decode speed (prefill at about 200+ tokens/s), and that figure usually depends on the expert-selection strategy.

    Cost estimate (order-of-magnitude reference):

    • Dual-socket server CPU (multiple memory channels) + 512 GB–768 GB DDR5 memory + 1–2 consumer/professional GPUs: total system budget roughly in the range of tens of thousands to low hundreds of thousands of RMB.
    • If the expert count far exceeds the reference model and expert-layer routing is more complex, CPU-side memory may need 1.5–2 TB to comfortably hold all weights and runtime overhead, and the budget rises accordingly.
    • Expected experience: single-user conversation is usable; long-context prefill is slow; real-time scenarios such as code completion are unfriendly.

    The essence of this tier: trade time for money. It proves that “running an ultra-large model locally in a private environment” is feasible, but it is only suitable for latency-insensitive scenarios such as low-frequency Q&A, offline batch processing, and internal knowledge-base summarization.

    3. Tier 2: Single User at 25 Tokens/s

    Going from “single-digit tokens/s” to 25 tokens/s is a qualitative change, not a quantitative one, because the bottleneck shifts from “can it fit” to “can it be fed fast enough.”

    What determines speed? During decoding, every generated token requires moving the activated weights. Roughly speaking, ideal decode speed ≈ effective memory bandwidth ÷ bytes that must be read per forward pass. To reach 25 tokens/s under 4-bit quantization, you need hundreds of GB/s or more of “weight-read bandwidth,” and when experts are distributed across multiple cards, inter-card communication overhead is added on top.

    Architecture-level impact: the model’s backbone uses a linear attention variant plus attention residual design, and a gated form of MLA is one of its components. Compared with conventional full attention, linear attention significantly reduces KV cache growth over long sequences and the attention compute during decoding, which is favorable for actual tokens/s. However, all-to-all communication in expert layers remains a hard constraint.

    Hardware and cost estimate (order-of-magnitude reference):

    • Need 4–8 high-VRAM accelerators (such as 80 GB class) to form a small cluster, with weights sharded and resident in VRAM to avoid moving expert weights over PCIe.
    • Intra-node NVLink or equivalent high-bandwidth interconnect is preferred, so that moving activated values for expert routing does not become the bottleneck.
    • Budget is roughly in the range of hundreds of thousands to over one million RMB, depending on accelerator generation and quantity.
    • To preserve speed under long context, KV cache also consumes several GB to tens of GB of VRAM per card, so procurement should reserve headroom.

    Conclusion: single-user 25 tokens/s is the dividing line between “serious personal use” and “toy.” The cost is a hardware budget an order of magnitude higher than Tier 1.

    4. Tier 3: 10 Concurrent Users, Still 25+ Tokens/s per User

    This is the enterprise-grade goal and the hardest tier. Concurrency affects hardware in three ways:

    1. Compute must scale linearly while communication does not collapse. Batch concurrency improves compute utilization, but with 896 experts distributed across dozens of cards, every token must move activated values to target cards after routing, compute, and move them back. The officially recommended deployment form is a supernode of 64+ accelerators, precisely to address this: use ultra-high-speed interconnect (NVLink/NVSwitch domain or equivalent technology) to enlarge the communication domain so all-to-all traffic stays within a single domain. If it crosses nodes over a normal network, latency will directly eat into compute.
    2. KV cache VRAM scales with concurrency. With 10 concurrent users × long context, KV cache can occupy hundreds of GB, and the low-growth advantage of linear attention is diluted.
    3. Quantization level cannot be compromised. To satisfy both VRAM and bandwidth constraints, weights must remain at the 4-bit microscaling floating-point level, while quantization-aware training preserves accuracy. This is exactly the design completed from the SFT stage for this model, with less loss than “post-training hard quantization.”

    Cost estimate (order-of-magnitude reference):

    • Hardware form: 8 nodes × 8 cards = 64-card-class supernode, with high-speed switching network and IB/RoCE networking. Total system investment is usually in the tens of millions of RMB.
    • Operating cost: total system power is in the tens of kilowatts. Facility power, cooling, and operations should also be included.
    • If full capability is not required, evaluate distilled smaller models or layer-wise quantization strategies to cut the budget by an order of magnitude, but the capability ceiling drops accordingly.

    5. Overview and Decision Guidance

    Tier Hardware Form Budget Magnitude (Reference) Suitable Scenarios
    Just needs to run Large-memory server + a few GPUs (heterogeneous) Tens of thousands to low hundreds of thousands of RMB Low-frequency Q&A, offline batch processing
    Single user at 25 tokens/s 4–8 high-VRAM accelerator node Hundreds of thousands to over one million RMB Frequent personal/small-team use
    10 concurrent at 25 tokens/s 64-card-class supernode Tens of millions of RMB Enterprise internal production

    Finally, it must be emphasized that the speed, VRAM, and cost figures above are engineering estimates based on public information. After an equal-weight model is actually released, each item needs to be verified with standardized benchmarks. Providing the estimation method is more valuable than providing the numbers. If you master three formulas — “total parameters × bytes per parameter = weight size,” “effective bandwidth ÷ bytes read per token ≈ decode speed,” and “concurrency × per-request overhead ≈ cluster size” — you can make a first-pass, reliable judgment about the local deployment cost of any next-generation ultra-large model.