A Tiered Guide to Local Deployment Costs and Performance for Trillion-Parameter LLMs

1. Background: When Model Parameters Enter the Trillion-Tier

As of this writing, a MoE (Mixture-of-Experts) architecture large model with total parameters reaching 2.8 trillion has not yet fully open-sourced its weights and technical report, with the official release window expected within a week. Apart from the officially announced architecture specifications and inference API pricing, the speeds, VRAM footprints, and hardware configurations discussed below are engineering estimates based on publicly available information. These figures will need to be verified item by item using benchmark tests once the weights are officially released.

The reason for making these estimates in advance is that local deployment of trillion-level MoE models fundamentally differs from traditional dense models: VRAM footprint is calculated based on total parameters, while compute consumption is calculated based on activated parameters. This distinction dictates the magnitude of costs and the design approach for deployment strategies.

1.1 Architecture Essentials and Memory Calculation Logic

The model adopts a MoE architecture with 2.8 trillion total parameters, comprising 896 experts. Each token activates 16 of them, resulting in an expert activation ratio of approximately 1.79%. It must be emphasized that this is the expert activation ratio, which is not equivalent to the strict parameter activation ratio—because non-expert components such as the attention backbone, embedding layers, and routing network are fully engaged.

The architectural backbone is KDA (Kimi Delta Attention) with Attention Residuals, featuring Gated MLA as a component. For deployment, the key takeaway is a single conclusion:

All 896 experts must fit into addressable storage, and during inference, 16 experts are recalled based on routing results.

This means the lower limit of memory (or VRAM) capacity is determined by the total parameters, not the activated parameters.

1.2 Capacity Lower Limit Imposed by Quantization Formats

This model underwent quantization-aware training from the SFT (Supervised Fine-Tuning) stage, utilizing MXFP4 for weights and MXFP8 for activations. Converting 2.8 trillion parameters to 4-bit results in a raw data lower limit of approximately 1400GB.

However, MXFP4 is not a pure 4-bit format—according to the OCP specification, every 32 4-bit elements are accompanied by an 8-bit scale. Therefore, the actual overhead of weight elements plus scales is:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This does not account for non-4-bit tensors (such as normalization layers and embedding layers), metadata, memory alignment, runtime workspace, and KV cache. In other words, 1500GB is only the theoretical lower limit, and actual deployment requires a substantial margin on top of this figure.

1.3 Horizontal Reference: Scaling from 100B-Level MoE

Taking a widely deployed 671B-level MoE model as a reference: in FP8 format, 671B main parameters plus 14B MTP parameters total about 685GB. Even with 4-bit quantization, it requires about 335GB for weights plus 37GB for activations, totaling around 400GB to run on 8 80GB accelerator cards (640GB VRAM).

The volume of a 2.8-trillion-parameter model is over 4 times that of 671B. Even after quantization, the capacity lower limit for a full deployment is dictated by this scale—a single machine with 8 cards can no longer accommodate it; it necessitates multi-node multi-GPU setups or heterogeneous inference paradigms.

1.4 Communication: An Underestimated Bottleneck

Official recommendations suggest deploying such models using supernodes composed of 64 or more accelerators. The reason is that large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth:

• 896 experts are distributed across multiple cards;
• After routing, each token’s activations must be transferred to the card hosting the target expert for computation, and then transferred back;
• Communication latency directly consumes compute power; the more cards involved and the further the cross-domain distance, the more significant the degradation.

The value of a supernode lies in using high-speed interconnects (such as NVLink or equivalent solutions) to enlarge the communication domain, ensuring these transfers do not leave a single communication domain. This is the watershed for whether trillion-level MoE can be practically deployed.

2. Tier 1: Run It, Regardless of Speed

2.1 Target Definition

The sole objective: fit the nearly 1500GB of weights into addressable storage so the model can generate tokens, even if it’s only single-digit tokens per second. This tier is suitable for verifying model behavior, conducting precision alignment, and running through the pipeline, without pursuing an interactive experience.

2.2 Core Concept: GPU + CPU Heterogeneous Inference

The most straightforward approach is heterogeneous inference. The representative concept is: leave compute-intensive parts on the GPU, and offload storage-intensive parts to CPU memory.

Specifically for the MoE architecture:

• Each layer has a shared attention component (KDA plus Gated MLA here), which is compute-intensive and placed on the GPU for high-bandwidth computation;
• The 896 sparse experts occupy the bulk of storage but only activate 16 at a time, so they are entirely offloaded to CPU memory and fed by a large-memory server.

2.3 Hardware Configuration Estimate

Based on the 1500GB weight lower limit, it is recommended that the CPU side has no less than 2TB of memory, leaving room for the KV cache, workspace, and operating system. The GPU side only needs to handle the attention backbone and activated expert computation; one or two high-VRAM cards are sufficient to start.

Indicative configuration:

CPU:   Dual-socket server-grade CPUs with sufficient memory channels
Memory: 2TB+ DDR5
GPU:   1~2 80GB-class accelerator cards
Storage: NVMe SSD 2TB+ (to store weights, resident in memory after initial load)

Expected inference speed: single-digit tps, with time-to-first-token potentially reaching tens of seconds. It runs, but it is not user-friendly.

2.4 Applicable Scenarios and Limitations

• Suitable for: precision verification, running evaluation benchmarks, offline batch processing;
• Not suitable for: any interactive applications;
• Primary bottleneck: CPU memory bandwidth and PCIe transfer between GPU and CPU.

3. Tier 2: Single-User Usage, Output Speed Reaching 25 tps

3.1 Target Definition

Usable for single-user interaction: output speed ≥ 25 tps, with quantization level no lower than 4-bit (such as the MXFP4/MXFP8 combination). This tier requires pulling expert computation back to the GPU, or at least keeping hot experts resident on the GPU.

3.2 Dual Constraints of Capacity and Compute

To achieve 25 tps, two problems must be solved:

1. Capacity: 1500GB of weights cannot entirely fit into a single machine’s GPU VRAM, requiring tiered storage + on-demand loading;
2. Bandwidth: The transfer speed of expert weights from storage to GPU must keep up with the token generation pace.

3.3 Feasible Paths

Path A: Multi-node GPU Pool + Expert Parallelism

Distribute the 896 experts across multiple GPUs, with each card holding a subset of experts. After token routing, activations are transferred via high-speed interconnects. This is the closest to a “native” deployment but requires extremely high interconnect bandwidth, typically necessitating supernode-level hardware.

Path B: GPU VRAM + Large-Capacity NVMe Tiering

Hot experts remain resident in GPU VRAM, while cold experts are stored on NVMe, dynamically swapped in and out based on routing results. This requires optimization in routing prediction and caching policies; otherwise, swap latency will break the 25 tps target.

Path C: High Memory Bandwidth CPU + Small-Scale GPU

If CPU memory bandwidth is sufficiently high (such as multi-channel DDR5) and the expert activation ratio is extremely low (1.79%), computing on the CPU side might approach usable speeds. However, this depends on the degree of specific implementation optimization and represents the most uncertain part of these estimates.

3.4 Configuration Estimate

Taking Path A as an example, a rough estimate:

Accelerators: 8~16 80GB-class cards, forming a single communication domain via high-speed interconnect
Total VRAM: 640GB ~ 1280GB (insufficient to hold all weights, requires tiering)
CPU Memory: 1TB+ as weight backup
Interconnect: Card-to-card bandwidth ≥ 400GB/s level

Note: Whether this configuration can stably achieve 25 tps depends on actual benchmark tests after the weights are released. Before the weights are published, any specific tps number can only be regarded as a magnitude estimate.

4. Tier 3: 10 Concurrency, Single-User Output Speed ≥ 25 tps

4.1 Target Definition

10 concurrent requests, with each request outputting at no less than 25 tps, and quantization level no lower than 4-bit. This is the entry threshold for production-level services.

4.2 Why Concurrency Changes Everything

The challenges introduced by concurrency are not a simple linear stack:

• KV cache bloat: 10 concurrent streams mean the KV cache footprint is about 10 times that of a single stream, requiring a large amount of additional VRAM or memory to be reserved;
• Expert activation conflicts: Different requests route to different experts, requiring the GPU to handle multiple sets of expert computations simultaneously, increasing the frequency of VRAM swapping;
• Communication amplification: all-to-all communication volume scales with concurrency, making interconnect bandwidth a hard constraint.

4.3 Where the Official 64-Card Supernode Recommendation Comes From

The official recommendation of using 64 or more accelerators to form a supernode is precisely designed for this tier. Calculated at 80GB per card, 64 cards total 5120GB of VRAM—enough to hold all weights (approximately 1500GB) plus KV cache, activation workspace, and redundancy. Concurrently, the high-speed interconnect within the supernode can restrict all-to-all communication to a single communication domain.

4.4 Configuration Estimation Framework

Accelerators: 64+ cards, supernode-level interconnect
Total VRAM: 5120GB+
CPU Memory: 2TB+ (for routing, scheduling, preprocessing)
Storage: High-performance NVMe cluster for weight loading and checkpointing
Network: Intra-node high-speed interconnect + inter-node high-bandwidth RDMA

4.5 Cost Magnitude Indication

The hardware investment for this tier is at the data-center level and is not suitable to be measured by a “single machine cost” mindset. A more realistic path is:

• Rent GPU bare-metal or supernode instances from cloud providers on demand;
• Or adopt a general cloud-native architecture of “object storage for weights + container platform for scheduling + monitoring/alerting for O&M” to convert fixed costs into elastic costs.

5. Cost Accounting Methods and Considerations

5.1 Reusable Capacity Calculation Formula

For any MoE model, the local deployment capacity lower limit can be estimated using the following formula:

Weight Capacity ≈ Total Parameters × (Quantization Bits + Scale Overhead) ÷ 8
Deployment Lower Limit ≈ Weight Capacity × 1.2 (Metadata/Alignment/Workspace) + KV cache

The KV cache needs to be estimated separately based on concurrency, context length, number of layers, and hidden size.

5.2 Tiered Selection Comparison

Tier	Target	Hardware Magnitude	Applicable Scenarios
1	Runs successfully	Single machine + large memory	Verification, evaluation
2	Single user 25 tps	8~16 cards with high-speed interconnect	Personal research, small-scale trials
3	10 concurrency 25 tps	64 cards + supernode	Production services

5.3 Items to Verify

After the weights are released, the following figures must be verified item by item using benchmark tests:

• Actual weight file size and quantization format;
• Time-to-first-token and output tps under single-card and multi-card setups;
• Loss of all-to-all communication under different interconnect solutions;
• Actual KV cache footprint at the target context length.

6. Summary

Local deployment of trillion-level MoE is essentially a tri-constraint problem of storage capacity + communication bandwidth + compute power. Quantization can compress weights to the 1500GB level, but it cannot eliminate the communication demands of expert parallelism; heterogeneous inference can make the model “run,” but it is difficult to reach an interactively usable speed; true production-level concurrency still requires supernode-level high-speed interconnects.

For most teams, the rational path is: first use heterogeneous solutions to verify precision, then use elastic cloud computing to verify performance, and finally decide whether to build in-house or continue renting based on actual business volume. Before the weights are officially released, preparing the capacity accounting methods and selection frameworks is more valuable than chasing any specific numbers.