Local Deployment of a Trillion-Parameter MoE Model: Three-Tier Performance and Cost Estimates
This article provides a practical framework for estimating the local deployment cost of a roughly 2.8-trillion-parameter MoE model whose full weights are not yet public. It divides deployment requirements into three tiers, explains the memory and communication constraints, estimates hardware and operating costs, and provides a benchmark checklist for validating real-world performance after the weights are released. All speed, memory, card-count, and cost figures are planning estimates based on public information and should be verified with measurements once the model is available.
Local Deployment of a Trillion-Parameter MoE Model: Three-Tier Performance and Cost Estimates
Note: As of the writing of this article, the complete weights and technical report for this type of model, which has roughly 2.8 trillion parameters, have not yet been publicly released. The official statement says they will be published soon. Except for publicly disclosed architecture information and general deployment experience, the speeds, memory usage, card counts, and costs mentioned below are estimates derived from public information. They are intended only to help establish an order-of-magnitude judgment and a selection framework. After the weights are released, each item should be verified with measured benchmarks.
For local deployment of a trillion-parameter model, there are usually only two common questions: how much GPU memory is required, and how much will it cost. Neither question can be answered in a general way, because the answer depends on the required speed. It is better to split the requirements into three tiers:
- Tier 1: It only needs to run; speed does not matter.
- Tier 2: Single-user output speed reaches 25 tokens/s.
- Tier 3: 10 concurrent users, with each user still receiving at least 25 tokens/s, and quantization precision no lower than FP8.
The hardware scale between these tiers can differ by more than a factor of two. Below, we start from the weight size and then provide configuration and cost estimates for each tier.
1. First, Look at the Size: What 2.8 Trillion Parameters Mean for GPU Memory
1. MoE Memory Is Calculated by Total Parameters; Activation Ratio Mainly Determines Compute
This model uses a sparsely activated Mixture-of-Experts architecture. It has 896 experts, and each token activates 16 experts, so 16 / 896 is approximately 1.79%. This is the expert activation ratio, not a strict parameter activation ratio. The backbone uses KDA (Kimi Delta Attention) and Attention Residuals, with Gated MLA as one of its components. The above architecture information comes from public disclosure.
To understand deployment cost, we first need to clarify two concepts that are often confused:
- Resident GPU memory is determined by total parameters. At runtime, only a small portion of the parameters may be used for computation, but all 896 experts must still be loaded into memory. During inference, the relevant experts are selected according to the routing result. Attempting to load only the experts that might be used is either impossible because routing cannot be predicted in advance, or it degrades into a swap-in/swap-out mode where weights are temporarily fetched from external storage for every token. In that case, throughput would become unacceptable.
- Compute consumption is determined by activated parameters. The activation ratio affects FLOPs and bandwidth requirements per token, and it also indirectly affects how many weights need to be read per token. However, it does not reduce the total number of weights that must be loaded.
This also explains why offline batching cannot solve the memory problem for this type of model. Even if full routing statistics are collected in advance, the computation path is still determined by the input, and the weights must remain fully resident. Offline batching can save repeated loading and repeated preprocessing, but not GPU memory.
2. From Quantization to Resident Memory: A Byte-Level Calculation
This model adopts quantization-aware training (QAT) from the SFT stage, combined with an MXFP4 weight + MXFP8 activation format. This is a block-level micro-scaling format that the industry is currently promoting. There are two important points to keep in mind when interpreting these numbers:
- The 4 and 8 in FP4/FP8 refer to the storage bit width per element, not decimal precision. A 4-bit value means 4 bits per parameter. The memory calculation should be compared with FP16 and INT8 using the same standard.
- Block formats include scaling factors, so the real usage is higher than the raw bit width. According to the OCP micro-scaling specification, each group of 32 4-bit elements is accompanied by one 8-bit shared exponent. The average is therefore 4.25 bits per parameter (4 + 8/32):
2.8 × 10¹² × 4.25 ÷ 8 ≈ 1.4875 × 10¹² bytes ≈ 1487.5 GB
This 1487.5 GB is only the lower bound for the weights. It does not yet include:
- Unquantized tensors: Some word embedding parameters, normalization parameters, routers, some projections in attention, and possibly attention scaling or gating parameters are usually stored in BF16/FP16 or higher precision. Even if they account for only 0.1% to 0.5%, that corresponds to 2.8 to 14 GB. Although distributed across dozens or hundreds of tensors, they should not be ignored.
- Framework overhead and alignment: Every GPU needs its own copy of the framework, CUDA context, communication libraries, and descriptors. Tensor alignment by page or block can cause fragmentation. NCCL buffers and CUDA Graph capture memory also consume space.
- KV cache: If MTP (multi-token prediction) heads are enabled, additional workspace must be allocated for them.
- Inference runtime: Activation tensors, temporary workspaces, sampling buffers, and similar structures.
A practical approach is to reserve 15% to 30% of the weight size as a runtime budget. Adding this layer to 1487.5 GB gives a plannable capacity range of about 1710 to 1930 GB. If we assume 80 GB of usable memory per card, which is about 97% of the nominal value because the rest is used by ECC and reserved regions, this corresponds to 22 to 26 high-end accelerator cards. Empirically, the usable weight space of a single card usually does not exceed 75% to 80% of its nominal memory. For example, a 16 GB nominal card is often advised to hold only 11 to 12 GB of weights. Using this estimate, the actual requirement would be 32 to 40 cards in the 80 GB class, which corresponds to 4 to 5 standard 8-card servers. Higher-spec 141 GB-class cards would correspond to roughly half the number of cards.
In other words, 1487.5 GB is the “raw weight lower bound,” not an answer such as “a few cards are enough.”
Example code: estimating weight memory and card count (all parameters can be replaced with measured values)
import mathdef estimate(tot_params=2.8e12, bits=4.25, card_gb=80, frac=0.75, overhead=0.20):"""bits=4.25: FP4 weights + one 8-bit scale per 32 elements; frac: fraction of card memory usable for weights"""raw_gb = tot_params * bits / 8 / 1e9 # Raw weight sizeusable = card_gb * frac # Memory available for weights per cardneed_gb = raw_gb * (1 + overhead) # Planned capacity after reserving runtime overheadcards = math.ceil(need_gb / usable) # Required number of accelerator cardsnodes = math.ceil(cards / 8) # Estimated nodes based on 8 cards per serverreturn raw_gb, need_gb, cards, nodesfor card in (80, 141):print(f"{card} GB card:", estimate(card_gb=card))# 80 GB card: (1487.5, 1785.0, 30, 4)# 141 GB card: (1487.5, 1785.0, 17, 3)The formula itself is not complicated. The difficult part is calibrating
fracandoverheadaccurately. It is recommended to first run a minimal deployment, such as a single 8-card server, measure the actual usage, and then linearly extrapolate to the target scale.
3. Capacity Is Only the First Threshold: Why a Super Node Is Needed
Loading the weights into GPU memory only makes the model “startable.” Whether it can run efficient inference depends on communication.
Routing communication: all-to-all is a major overhead. Under standard tensor parallelism, each card holds part of the experts. After the router selects experts for a token, the token must be dispatched to the card where the target expert resides, computed there, and then aggregated back. This dispatch-compute-aggregate process is all-to-all communication. Large-scale MoE inference is extremely sensitive to all-to-all bandwidth: communication latency can directly offset the compute benefits brought by parallelism. The deeper the model and the more nodes involved, the more obvious the accumulated latency becomes.
Cross-node bandwidth cliff. Within a node, card-to-card interconnects such as NVLink typically provide per-card bandwidth in the hundreds of GB/s to 1 TB/s range. Cross-node networks such as IB or RoCE often provide only 25 to 50 GB/s per card, which is roughly an order of magnitude lower. Expert distribution within a standard 8-card node is tolerable, but once experts are distributed across nodes, the cross-node segment can become the main bottleneck.
Super nodes act as a communication domain. The official recommendation is to use a super node composed of more than 64 accelerators. Such systems connect dozens or hundreds of cards through dedicated switching chips into a large card pool, making inter-card communication comparable to traditional intra-node interconnects and allowing most token routing communication to occur within a high-bandwidth domain.
Trinity/MoE-specific P2P and disaggregated modes. Framework-level optimizations are also important:
- All-to-all for batched requests may degrade into point-to-point transfers, in which case NCCL P2P can be used instead of collective communication primitives.
- Prefill and Decode have very different workloads, so prefill/decode disaggregated deployment can be used.
The direct cost implication is that if the goal is smooth inference, hardware cannot be added linearly as “8-card server multiplied by number of servers.” The premium for super nodes or equivalent interconnect capability must be included. This premium can be a major cost item and may be as significant as the GPU cost itself.
2. Three-Tier Configuration and Cost Estimates
Costs are estimated using the following assumptions: the purchase price of a complete high-end accelerator server, including GPU plus CPU, memory, and network interfaces, is about RMB 300,000 per unit for the 80 GB class and about RMB 700,000 per unit for the 141 GB class. Power consumption per server is about 10 kW and 16 kW respectively. A combined PUE of 1.3 is used, with an industrial electricity price of 0.7 RMB/kWh. Depreciation is calculated over 3 years, about 26,280 hours. Unit prices are assumptions and should be replaced with actual quotes. All speed figures are estimates derived from public information and have not been measured.
| Tier | Goal | Weight Precision | Recommended Configuration (80 GB class) | Equivalent Card Count | Purchase Budget | Electricity Cost per Hour | Estimated Speed |
|---|---|---|---|---|---|---|---|
| 1 | Just needs to run; speed does not matter | FP4, lower precision if necessary | 4 to 5 eight-card nodes | 32 to 40 | About RMB 1.2 million to 1.5 million | About RMB 18 to 22 | Single-digit to low-teens tokens/s |
| 2 | Single user ≥25 tokens/s | FP4, some critical layers kept at higher precision | 4 to 5 eight-card nodes plus equivalent interconnect capability | 40+ | More than RMB 1.5 million | More than RMB 22 | About 25 tokens/s |
| 3 | 10 concurrent users, each user ≥25 tokens/s, precision no lower than FP8 | Weights in FP8 | 6 to 9 eight-card nodes, or a comparable scale of 141 GB-class cards | 50 to 70+ | From about RMB 1.8 million to 2.7 million | From about RMB 28 to 38 | 10-way × ≥25 tokens/s |
A few additional notes:
- Downward room for Tier 1: If weight precision lower than FP4 is allowed, for example 2-bit or even lower in extreme scenarios, the configuration can be reduced to 2 to 3 eight-card nodes. However, the quality loss cannot be ignored.
- Advantages of 141 GB-class cards: Memory is nearly doubled, so the required number of cards and rack count can decrease significantly, and electricity costs are also lower. However, the unit price is higher. The actual choice depends on the power and cooling conditions of your data center.
- Hidden costs of Tier 3: With 10 concurrent users, compute consumption during the prefill phase increases significantly, and prefill and decode can affect each other. For example, prefill may interrupt decode, causing TPOT jitter. If prefill/decode disaggregation is used, an additional resource pool is required, and the overall cost will rise further.
3. Limits of the Estimate: What Is Not Included
- Speed is uncertain. The 25 tokens/s target is an assumption in this article, not a measured result for this model. Actual speed depends on model size, quantization method, inference framework, and hardware bandwidth. Once the weights are released, measured results should be used.
- Data center costs. The electricity cost above only includes the power consumption of the GPU servers. It does not include networking equipment, storage, or cooling. Roughly estimated with PUE 1.3, the electricity cost for 40 to 50 cards is about RMB 16,000 to 22,000 per month, and about RMB 600,000 to 800,000 over three years. The absolute amount is not small.
- Software maturity. Components such as KDA, Gated MLA, Attention Residuals, and MTP require support from inference frameworks and low-level operator libraries. When a new model is first released, quantization kernels, communication optimizations, and other components may not yet be fully ready. This means:
- The speed achieved in the first week may be 30% to 50% lower than the steady-state speed.
- Framework version upgrades and driver compatibility issues will continue to consume operations effort.
- The official recommendation of a 64-card super node may not be deployable in ordinary data centers because of power, cooling, and network requirements.
- Long-context KV cache. Inference frameworks usually allocate KV cache automatically based on available memory, but the KV cache of ultra-large-scale MoE models can grow quickly, depending on layer count, sequence length, and attention structure. In 128K-context and multi-turn dialogue scenarios, KV cache usage may reach 10% to 20% of the weight size. This should be reserved separately when planning capacity.
4. Decision Recommendations
| Requirement | Recommended Option |
|---|---|
| Want to experience or validate quality, without requiring speed | Wait for the weights to be released, then use Tier 1 to get it running; or directly use the official API or cloud service |
| Single user, 25 tokens/s is sufficient | Tier 2: 40+ 80 GB-class cards, or 17 141 GB-class cards. Pay attention to measured TPOT and tokens/s |
| Small team of up to 10 users sharing, requiring 25+ tokens/s | Tier 3: 50 to 70+ cards. Pay attention to prefill/decode disaggregation and scheduling policies |
| Production-grade SLA | Add redundant nodes, monitoring, alerts, and failover on top of the base configuration. Cost may increase by another 30% to 50% |
5. Benchmark Checklist After the Weights Are Released
Below is a checklist that can be executed directly. It is shown as pseudocode:
□ Weight loading and memory usage- Record loading time, peak memory, and resident memory- Check whether there is an optimization that loads only part of the experts (usually not available)□ Single-request latency- Measure tokens/s, TPOT (Time Per Output Token),and TTFT (Time To First Token)□ Concurrency capability- Gradually increase concurrency (1 → 4 → 10 → 20),and record tokens/s and TPOT P99 at each level□ Communication overhead- Use profiling tools to inspect the proportion of time spent in NCCL;if all-to-all takes too much time,it indicates insufficient interconnect capability or an unoptimized topology□ Precision regression- Compare the precision gap between FP4/FP8 and BF16 on public evaluation sets (if provided officially)
A special reminder: All numbers should be based on the official release. Architecture parameters such as 896 experts, 16 activated experts, KDA backbone, Gated MLA, and MTP come from public disclosure, but quantization methods, measured speed, and memory usage must be tested on your own hardware after the weights are released. The estimates in this article are only intended to help you make decisions about how many cards to buy and how much budget to reserve. They cannot replace real measurements.
If you already have specific GPU models, card counts, or a budget, you can plug the hardware parameters into the estimation function above to obtain a rough configuration list. Then, after the weights are released, verify each item step by step.
