Local Deployment of Trillion-Parameter MoE Models: Memory Math and Three Cost Tiers

Local Deployment of Trillion-Parameter MoE Models: From Memory Estimation to Three Cost Tiers

Local deployment of large Mixture-of-Experts (MoE) models is, at its core, an exercise in storage and bandwidth arithmetic. Before discussing how much it costs, you need to understand how much memory the model actually consumes, what determines compute consumption, and where communication bottlenecks arise. This article uses a MoE model with 2.8 trillion total parameters as its reference point. Starting from its architectural characteristics, it breaks down hardware configurations and cost ranges for three deployment targets. One important caveat: until the official weights and technical report are released, all speed, memory, and hardware figures below are estimates based on publicly available architecture information. They must be validated with benchmarks once the weights are available.

1. Getting the math right: how much memory do 2.8 trillion parameters actually need

This model uses a MoE architecture with 896 experts, activating 16 experts per token, giving an expert activation ratio of roughly 1.79%. A commonly confused concept needs clarification: expert activation ratio is not the same as parameter activation ratio. The routing mechanism only determines how compute is distributed. Memory usage, by contrast, must account for the total parameter count—all 896 expert weights have to reside in addressable storage, and at inference time the activations are routed to the devices holding the relevant experts.

The architecture backbone consists of KDA (Kimi Delta Attention) and Attention Residuals, with Gated MLA as a component. The model has incorporated quantization-aware training from the SFT stage, using MXFP4 for weights and MXFP8 for activations. At 4-bit precision, the raw lower bound for 2.8 trillion parameters is approximately 1,400 GB. However, MXFP4 is not pure 4-bit: according to the OCP specification, every 32 4-bit elements carry an accompanying 8-bit scale, bringing weights plus scales to roughly 4.25 bits per element:

2.8T × 4.25 bit ÷ 8 ≈ 1,487.5 GB

This does not yet include non-4-bit tensors, metadata, alignment overhead, runtime workspace, or KV cache. As a cross-reference, a 671B-parameter MoE model in FP8 requires about 685 GB for main parameters plus MTP parameters. After AWQ 4-bit quantization, it still needs approximately 335 GB for weights plus 37 GB for activations—about 400 GB total—to run on a single high-end server with eight 80 GB GPUs. This model is more than four times that size, which sets the lower bound for full deployment: roughly 1.5 TB is only the weight threshold, and a practical configuration should leave at least 20% headroom.

2. Communication is the hidden ceiling

The official recommendation for models of this class is deployment on a supernode composed of 64 or more accelerators. The reason is that large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth: 896 experts are distributed across multiple cards, and each token activation must be moved to the card holding the target expert, then moved back after computation. Communication latency directly eats into compute efficiency, and the higher the expert parallelism, the more pronounced this overhead becomes. The value of a supernode lies in using high-speed interconnect (such as NVLink or an equivalent) to enlarge the communication domain so that activation transfers stay within a single domain.

For local deployment, this has two implications. First, simply stacking ordinary PCIe servers is unlikely to yield good throughput. Second, if the goal is merely “it runs,” you can deliberately forgo high-speed interconnect and use a heterogeneous setup to sidestep the communication bottleneck.

3. Three deployment tiers and cost estimates

Tier 1: Just make it run, speed is irrelevant

The only goal is to fit roughly 1.5 TB of weights into addressable storage and have the model emit tokens, even at single-digit tokens per second.

The most straightforward approach is GPU + CPU heterogeneous inference. Each MoE layer has a shared attention component (KDA plus Gated MLA in this model) that is compute-intensive and should stay on GPUs for high-bandwidth processing. The 896 sparse experts dominate storage but only 16 are activated per token, so they can all be offloaded to CPU memory on a large-memory server. Mature open-source heterogeneous inference frameworks already follow this approach, separating compute-intensive components from storage-intensive components.

Example hardware configuration:

• GPU: 1–2 accelerators with 80 GB of memory each, handling attention and routing computation;
• CPU memory: at least 2 TB of DDR5 to hold all expert weights (about 1.5 TB at 4-bit) plus runtime overhead;
• Storage: 2 TB or more of NVMe SSD for weight loading and paging;
• CPU: a high-core-count server platform with multi-channel memory support.

Cost range: a single high-memory server (2 TB RAM + 1–2 high-end accelerators) typically falls in the ¥200,000–¥400,000 range. Expected speed is single-digit tps, suitable only for validation, debugging, and offline batch processing. This tier has no interactive value.

Tier 2: Single-user output at 25 tps

25 tps is the baseline for an acceptable interactive experience. At this tier, CPU memory bandwidth becomes the bottleneck—expert weights must be read from memory on every forward pass, and effective DDR5 bandwidth is far lower than GPU memory bandwidth. To sustain 25 tps for a single user, you generally need to place hot experts or all experts in GPU memory, or adopt a tiered “GPU memory + high-bandwidth memory” strategy.

A realistic configuration is 8–16 accelerators with 80 GB each (640 GB–1.28 TB total GPU memory), paired with a large-memory host for expert tiering and caching. If all weights must reside in GPU memory, you would need about 20 cards with 80 GB each, which drives up cost sharply. A more economical approach is to keep hot experts in GPU memory, leave cold experts in system memory, and perform dynamic scheduling based on routing statistics.

Cost range: an 8-card server runs roughly ¥1.5–¥2.5 million; a 16-card system runs roughly ¥3–¥5 million. The key at this tier is not raw card count, but whether the scheduling strategy between GPU memory and system memory is sufficiently granular.

Tier 3: 10 concurrent users at 25 tps per user, with quantization no lower than the specified format

Ten concurrent users means KV cache and activation memory multiply, while each stream must still sustain 25 tps—putting dual pressure on both compute and bandwidth. At this tier, a supernode approach is mandatory: 64 or more accelerators connected via high-speed interconnect to form a single communication domain, with expert parallelism distributed within it so that all-to-all traffic stays inside the domain.

Example configuration:

• Accelerators: 64–128, each with 80 GB or more of memory;
• Interconnect: high-speed card-to-card interconnect (NVLink-class), fully interconnected within a node, with high-bandwidth RDMA across nodes;
• Memory: 1 TB or more per node for KV cache and workspace;
• Storage: distributed high-speed storage for weight loading and checkpoint management.

Cost range: at an estimated ¥80,000–¥150,000 per accelerator, a 64-card system costs roughly ¥5–¥10 million, and a 128-card system can reach ¥10–¥20 million, excluding data center space, power, and cooling upgrades. This tier falls into the small-scale compute cluster category and is appropriate for teams with clear service volume projections.

4. Quantization format is the cost lever

The model natively supports MXFP4 weights with MXFP8 activations, which is already a fairly aggressive quantization scheme. If Tier 3 requires “quantization no lower than” a given format, that means you cannot compress further to save memory, and the memory floor is locked at roughly 1.5 TB. Conversely, the choice of quantization format directly determines the hardware tier: allowing lower bit widths lets you use fewer cards; locking in higher bit widths forces you to accept a higher hardware threshold.

In practice, the accuracy preservation claimed by quantization-aware training must be validated with benchmarks after deployment. Degradation from low-bit quantization often surfaces only in long-context and complex reasoning scenarios.

5. Deployment recommendations

1. Build a budget framework before the weights are released: work backward from 1.5 TB of weights, 20% headroom, and target concurrency to determine memory requirements. Avoid discovering after hardware delivery that the model does not fit.
2. Validate the heterogeneous approach first for Tier 1: use a large-memory server with a small number of GPUs to run the full pipeline end-to-end, verify routing correctness and output quality, then decide whether to upgrade.
3. Focus on expert scheduling for Tier 2: profile expert activation distribution under real workloads and pin high-frequency experts to GPU memory. This can significantly reduce the number of cards required.
4. Plan the communication domain early for Tier 3: a supernode is not simply stacking cards. Interconnect topology and expert distribution strategy require joint tuning, otherwise all-to-all traffic will consume the compute advantage.
5. Reserve a verification phase: all estimates must be checked item by item with benchmarks once the weights are released, especially tps and actual memory footprint.

Local deployment of a trillion-parameter MoE model is ultimately about trade-offs among storage capacity, communication bandwidth, and compute density. The three tiers correspond to three different trade-off logics: Tier 1 sacrifices speed for cost, Tier 2 trades scheduling sophistication for GPU memory, and Tier 3 uses interconnect to buy compute efficiency. Being clear about which tier you need is what makes the spending defensible.