Local Deployment for Trillion-Parameter MoE Models: Three Tiers and Real Cost Estimates

1. Three Assumptions Before We Start

Before discussing how much money is required, the analytical framework must be established. Otherwise, all the numbers will lack a solid foundation.

Assumption 1: All figures below are engineering estimates. For some trillion-scale models, the complete weights and official technical reports may not yet be fully public. The speed, memory, and hardware configuration numbers in this article are derived from publicly available architectural information. Any serious purchasing decision must wait until the weights are available and then be verified item by item with benchmarks, especially for the three metrics most likely to cause problems: time to first token, long-context degradation, and concurrent throughput.

Assumption 2: A clear reference model profile. This article uses a typical next-generation trillion-parameter MoE model as the analysis target: roughly 2.8 trillion total parameters, expert-parallel architecture, 896 experts, 16 active experts per token, and an expert activation ratio of about 1.79%. Note that this is the expert activation ratio, not the strict parameter activation ratio. The sequence modeling backbone uses a hybrid design of a linear attention variant plus gated latent attention, combined with an attention residual structure, aiming to balance quality and efficiency under long contexts.

Assumption 3: MoE deployment cost is determined by total parameters, not active parameters. This is the most commonly misunderstood point. The activation ratio only affects compute consumption and energy per token. At the storage level, however, all 896 experts must be placed in addressable storage, and only 16 are loaded at inference time based on routing. Sparsity saves on how much compute is used per step, not on how much must be stored in total.

2. Memory Accounting: 1.5 TB Is Only the Starting Point

2.1 A Closer Look at Quantization Formats

Models of this class typically undergo quantization-aware training (QAT) starting from the supervised fine-tuning stage, with weights using a 4-bit microscaling floating-point format such as MXFP4 and activations using an 8-bit microscaling format such as MXFP8. This brings two direct benefits: the accuracy loss of 4-bit weights is controlled, and inference engines do not need complex post-training quantization calibration.

However, the calculation cannot simply be based on “4-bit”:

• Lower bound for raw data: 2.8 trillion parameters × 4 bit ÷ 8 ≈ 1400 GB.
• Microscaling formats are not pure 4-bit. According to the open microscaling format specification, every 32 4-bit elements carry one 8-bit scale factor. The actual footprint is 2.8 trillion × 4.25 bit ÷ 8 ≈ 1487.5 GB.
• On top of that, the following must be added: non-4-bit tensors such as embedding and output layers, router and normalization layer parameters, quantization metadata and alignment padding, runtime activation workspace, and KV cache that grows linearly with concurrency and context length.

Conclusion: For a full-quality 4-bit deployment, the weight lower bound is above 1.5 TB, with KV cache counted separately.

2.2 A Reference Point for Scale

Consider an industry MoE model at the 671B parameter level. Stored in FP8, the main parameters plus the multi-token prediction module total about 685 GB. Even when compressed to about 335 GB of weights plus 37 GB of activations using a common 4-bit post-training quantization method, yielding about 400 GB in total, it still requires 8 accelerator cards with 640 GB of combined memory to barely run. A model with 2.8 trillion total parameters is more than four times larger. That is why full-scale local deployment has never been a problem solvable by a single high-end card.

2.3 How to Estimate KV Cache

A common formula for KV cache memory is: KV memory ≈ 2 × layers × KV heads × head dimension × sequence length × bytes per precision × concurrency. Once concurrency and context length grow, dozens of gigabytes can easily exceed the budget, so capacity planning must reserve for it in advance.

# Bandwidth model for decode speed (estimates expert weight reads only,
# ignoring attention and KV reads)
def decode_tps(total_params, num_experts, active_per_token,
               bits_with_scale, bandwidth, efficiency=0.55):
    """bandwidth: sustained effective bandwidth of the storage medium (B/s)"""
    bytes_per_token = total_params * (active_per_token / num_experts) \
                      * bits_with_scale / 8
    return bandwidth * efficiency / bytes_per_token
# Expert weights to read per token: 2.8e12 * 16/896 * 4.25/8 ≈ 26.6 GB
# Dual-socket DDR5 memory (theoretical ~500 GB/s per socket):
# theoretical upper bound ~17 tps, but 5-10 tps is realistic in practice
# A single large-memory HBM card has ample bandwidth but cannot hold 1.5 TB
# of weights, so multiple cards must share the load

3. Tier 1: It Runs, but Speed Is Not the Goal

3.1 Technical Approach: GPU-CPU Heterogeneous Inference

MoE has an inherent compute-storage mismatch, and this creates room for low-cost solutions:

• The attention portion of each layer, consisting of linear attention plus gated latent attention, has few parameters but is compute-intensive, so it stays on the GPU to take advantage of high bandwidth.
• The 896 sparse experts account for the vast majority of parameters, but only 16 are active per token. They can all be offloaded to the large CPU memory pool.
• At inference time, activations are sent to the corresponding experts based on routing. Sparse activation fits naturally into the heterogeneous split between large-but-slow memory and small-but-fast GPU memory.

Several general-purpose heterogeneous inference engines in the open-source community already implement this approach: expert weights are laid out in large memory according to NUMA affinity, combined with highly optimized CPU matrix kernels, so that PCIe transfers are hidden behind computation.

3.2 Hardware List and Investment Range

• A dual-socket server with 24 or more memory slots and at least 1.6 TB of total DDR5 memory: roughly 150,000 to 200,000 RMB.
• One or two 24-48 GB GPU accelerator cards for attention computation and scheduling: roughly 20,000 to 80,000 RMB.
• Total: roughly 200,000 to 300,000 RMB, with output speed in single-digit tokens per second.

3.3 Suitable Use Cases

The value of this configuration is validation: verifying the inference pipeline, checking quantization quality, running offline batch processing, and handling extremely low-frequency internal Q&A. Interactive use is out of the question. Do not attempt to use it for production traffic.

4. Tier 2: 25 Tokens/s for a Single User

4.1 Why 25 tps Is a Key Threshold

25 tokens/s exceeds the comfortable reading speed of most people and is generally accepted as the minimum for a single-user interactive experience. But the decode phase is a classic memory-bound problem: for every generated token, the weights of the 16 routed experts must be moved from storage to the compute unit. The active parameters are roughly 2.8 trillion × 1.79% ≈ 50 billion. At an effective 4.25-bit footprint, each token requires reading about 26.6 GB of expert weights. Together with attention reads and KV access, the total approaches 30 GB per token.

Reaching 25 tps therefore requires at least 750 GB/s of sustained effective read bandwidth. A dual-socket DDR5 server typically delivers only 500-700 GB/s of real-world effective bandwidth, and after kernel and scheduling overhead, little headroom remains. This is the physical reason CPU offloading cannot reach higher speeds. The conclusion is direct: the weights must be placed in HBM-class high-bandwidth memory.

4.2 Deriving the Minimum Configuration from Capacity

• With 1.5 TB of weights plus runtime workspace and KV cache, total HBM capacity must start at about 1.8 TB.
• With 80 GB-class accelerator cards, this means 24 or more cards. With 96-141 GB large-memory cards, 12-16 cards is the capacity floor.
• In practice, the engineering recommendation is to plan for either two 8-card nodes with RDMA high-speed networking or one 16-card supernode. Inter-card interconnect bandwidth directly determines expert-parallel efficiency.
• Total system investment, including cards, servers, networking, and storage: roughly 3 to 6 million RMB.

4.3 Three Easily Overlooked Pitfalls

1. Cross-node all-to-all communication per layer must go through RDMA. Under small batch sizes, communication latency becomes a disproportionately large share of total time.
2. Mixed prefill-decode scheduling and KV paging management directly determine the real-world experience under long contexts.
3. With batch size 1, GPU utilization is very low, and the cost per token is actually higher than in concurrent scenarios. Exclusive use by a single user is not necessarily cheaper.

5. Tier 3: 10 Concurrent Users with at Least 25 Tokens/s Each

5.1 Concurrency Changes the Problem

10 concurrent streams × 25 tps = 250 tps aggregate decode throughput. The good news is that batching can amortize sparse reads. With 10 tokens each routed to 16 experts, the expected number of distinct experts hit is 896 × (1 − (880/896)^10) ≈ 148. Each step reads about 148 × 1.66 GB ≈ 245 GB, which averages to about 25 GB per token. The aggregate bandwidth requirement is about 6.1 TB/s. The bad news: all-to-all communication happens at every layer and every step, and under small batches the communication latency eats into compute efficiency. KV cache grows linearly with concurrency, and the compute pressure of the attention portion also increases significantly.

5.2 Why a Supernode Is Needed

Public deployment guidance for models of this class commonly recommends a supernode with 64 or more accelerator cards. The ultra-high-bandwidth interconnects bring all 896 experts into a single large communication domain, so routed activation transfers never leave the domain. With 64 cards, total HBM capacity exceeds 5 TB, leaving ample room for weights, KV cache, and workspace, while data parallelism can further expand the concurrency ceiling.

5.3 Investment and Total Cost of Ownership

• System purchase: on the order of tens of millions of RMB. Accelerator cards dominate the cost, while full racks, switch fabrics, CPUs, storage, and networking add a scaling factor on top.
• Power and cooling: total system power ranges from tens to a hundred kilowatts, costing hundreds of thousands to over a million RMB annually.
• Colocation, dedicated operations staff, and a 3-5 year accelerator depreciation cycle must all be included in the total cost of ownership (TCO).

6. A Fourth Option: Skip the Hardware and Rent Cloud Compute

For most teams, self-hosting is not the only answer. Three models each have their own economic logic.

Model	Data Boundary	Cost Structure	Suitable Scenarios
Direct model API calls	Data leaves your boundary	Pay-as-you-go operating expense	Rapid validation, non-sensitive workloads
Cloud GPU instances or containers	Data stays within your boundary	Operating expense, elastic	Bursty or validation workloads
Private self-hosted deployment	Fully controlled	Large capital expenditure plus operations	Sustained full load, strict compliance

For example, Baidu AI Cloud provides GPU cloud servers, container platforms, high-speed networking, object storage, and monitoring and alerting capabilities, which can be used to stand up multi-card inference environments. It also offers a model service platform for on-demand access to large model services. For teams in the validation stage or with pronounced peak-to-trough load patterns, converting a one-time capital expenditure of tens of millions of RMB into predictable operating expenses is often the more rational choice. Private self-hosting only makes sense when the workload is consistently near full load and zero data-boundary tolerance is required.

7. Final Thoughts

MoE sparsity saves compute, not storage. The lower bound of deployment cost is pinned down by total parameters multiplied by quantization bit width. The dividing lines among the three tiers are set by bandwidth and communication. “It runs” only requires a large-memory server plus one or two cards, at roughly 200,000 to 300,000 RMB. 25 tps for a single user requires moving weights into HBM, starting in the millions of RMB. 10 concurrent users × 25 tps requires a supernode-class cluster, on the order of tens of millions of RMB, plus long-term operational costs. Work out the capacity and bandwidth budgets first, then decide whether to buy, rent, or call an API directly. Otherwise, the money may end up in the wrong place.