Deploying a Trillion-Parameter MoE Model Locally: Hardware Requirements and Cost Breakdown

1. Clarifying a Core Issue: All Model Weights Must Reside in Memory

Before discussing local deployment options, it is important to understand a key fact: although a sparse expert model activates only part of its parameters during a single inference pass, this does not mean you can load only part of the weights.

Consider a sparse expert model with a total parameter count in the trillions. Its design divides the model into multiple expert networks, and each forward pass routes to only a subset of them. This significantly reduces the computation per inference, but all expert weights must still reside in VRAM or system memory. The reason is that different input samples activate different expert paths, and the inference engine cannot determine in advance which weights will be used. Therefore, any expert must be accessible at any time.

This characteristic leads directly to a seemingly counterintuitive conclusion: when deploying a sparse expert model, the lower bound on VRAM capacity is determined not by the “activated parameters” but by the “total parameters.”

2. Estimating the Actual VRAM Footprint of the Weights

When assessing hardware requirements, the first step is to estimate the size of the model weights themselves. A widely used estimation formula in the industry is:

Weight size (bytes) = Total parameters × Bits per parameter / 8

For a model with approximately 2.8 trillion total parameters, if 4-bit precision is used for weight storage, the theoretical calculation is as follows:

2.8 × 10^12 × 4 bit / 8 = 1.4 × 10^12 bytes = 1.4 TB

There is an important technical detail worth noting here: a model natively using a 4-bit weight format is fundamentally different from “forcibly quantizing 16-bit weights down to 4 bits.” Native low-precision training or native low-precision storage means the model was designed and trained to adapt to that precision format, with weight distribution, numerical range, and quantization error all within expectations. Post-hoc quantization, by contrast, may introduce additional precision loss, especially in layers where activation value dynamic ranges are large.

Therefore, roughly 1.4 TB of weight data is the hard lower bound for deploying this model. On top of that, additional VRAM must be reserved for the following:

• Activation cache: intermediate computation results during inference;
• KV cache: stored historical key-value pairs in the attention mechanism, which grow linearly with context length;
• Runtime overhead: memory occupied by the inference framework itself;
• State space or routing cache: temporary data generated during expert routing.

Taking all these factors together, the total VRAM actually required for a single user is approximately 1.5 TB or more. This figure is the core constraint of the entire deployment plan.

3. Deployment Tier One: “Just Able to Run”

If the goal is merely to load the model in a local environment and complete inference, without concern for generation speed, the hardware threshold is still far from low.

From a pure capacity perspective, traditional data center GPUs, despite having large per-card VRAM, may not natively execute 4-bit weight formats. If the hardware does not support native low-precision computation, dequantization must be performed at runtime to restore weights from 4-bit to a higher precision before participating in matrix operations. This means:

1. Dequantization itself consumes additional computational resources and time;
2. The actual VRAM occupancy of weights may increase due to runtime formats;
3. Inference efficiency will be noticeably lower than on hardware with native low-precision support.

Therefore, a more realistic recommendation in the industry is to use a newer generation of accelerator cards with native low-precision support. Even with this approach, a single node requires an 8-GPU configuration to hold all weights and runtime data. This is already server-grade deployment, not something an ordinary workstation can handle.

Actual inference speed under this configuration is usually slow, with generation speed potentially only a few to a dozen tokens per second. However, for scenarios such as offline batch processing, model evaluation, or functional validation, it remains feasible.

4. Deployment Tier Two: 25 Tokens/s for a Single User

If the goal is to provide a reading experience close to fluent for a single user, generation speed typically needs to be raised to around 25 tokens per second.

The key bottleneck to achieving this is still not compute power but VRAM capacity. The fact that a single card cannot hold all weights means a multi-GPU parallel scheme is necessary. Common parallel strategies include:

Tensor Parallelism: Split the weights of a single layer across multiple cards
Pipeline Parallelism: Distribute different layers across different cards
Expert Parallelism: Route different experts to different devices

For trillion-parameter sparse expert models, a combination of multiple parallel strategies is usually required. An 8-GPU node is the minimum viable deployment unit, transferring intermediate results between cards over high-bandwidth interconnects.

It is worth noting that whether an 8-GPU configuration can stably reach 25 tokens/s is affected by multiple variables:

• Context length: long contexts significantly increase KV cache and attention computation overhead;
• Sequence length distribution within a batch: padding strategies for variable-length inputs affect actual throughput;
• Hardware interconnect bandwidth: inter-card communication may become a bottleneck;
• Scheduling efficiency of the inference framework: different frameworks vary markedly in their optimization for sparse experts.

Therefore, 25 tokens/s is a reasonable target value, but before going live it must be validated through stress testing under real context lengths and real request distributions. It cannot be extrapolated from theoretical compute alone.

5. Deployment Tier Three: Supporting 10 Concurrent Requests

When the service needs to respond to multiple user requests simultaneously, the evaluation metric shifts from single-request latency to aggregate throughput. Taking 10 concurrent requests at 25 tokens/s each as an example, the system needs to stably deliver approximately 250 tokens/s of total output capacity.

From a VRAM capacity perspective, 10 concurrent requests do not necessarily require additional hardware. A single-node 8-GPU configuration can, in terms of capacity, hold the KV caches and runtime states corresponding to dozens of concurrent requests. If the context length of individual requests is within a normal range, the theoretically supportable concurrency would be even higher.

The real challenge lies in computational throughput and scheduling. When 10 requests are simultaneously in the generation phase, each forward pass must process 10 different sequences. These sequences may be at different decoding steps, with varying lengths and different expert routing combinations. The inference engine needs to efficiently organize these requests into batches, controlling latency while maintaining computational density.

Key optimization techniques include:

• Continuous batching: dynamically inserting newly arrived requests into the batch currently being executed;
• Paged KV cache management: reducing VRAM fragmentation and improving cache utilization;
• Preemptive scheduling: segmenting long sequences to prevent a single large request from blocking others;
• Expert load balancing: preventing certain popular experts from becoming computational bottlenecks.

The degree to which these capabilities are implemented depends on how deeply the inference engine supports sparse expert models, not merely on hardware specifications. Therefore, the performance of 10-way concurrency must also be determined through actual testing and cannot be simply extrapolated from VRAM capacity or theoretical compute.

6. Cost Structure: Reference Pricing for Cloud Deployment

Since the required hardware is data-center grade, individuals or small-to-medium teams typically do not have the means to purchase it directly. A more realistic path is renting GPU instances through cloud platforms.

Taking an 8-GPU accelerated instance from a certain cloud platform as an example, the per-card hourly price is in the range of several US dollars, and the hourly cost of an 8-GPU cluster is approximately several tens of US dollars. If long-term reserved instances are chosen, a certain discount is usually available, but even at the discounted price, the monthly cost of an 8-GPU cluster remains considerable:

Assuming the discounted unit price for an 8-GPU cluster is about $38/hour
Monthly cost ≈ 38 × 24 × 30 ≈ $27,360

At current exchange rates, this is roughly equivalent to nearly 200,000 RMB per month. This covers only the compute rental cost and does not include additional overhead such as bandwidth, storage, and operations.

For short-term validation scenarios, on-demand rental may be more flexible; for long-term production environments, reserved instances or annual/monthly packages offer cost advantages. Whichever approach is chosen, the following should be clarified before starting the project:

1. Target concurrency scale and single-request latency requirements;
2. Typical and maximum values of context length;
3. Expected daily average and peak call volumes;
4. Acceptable cost ceiling.

7. Summary and Decision Guidance

Returning to the original question: how much does it cost to deploy a trillion-parameter sparse expert model locally? The answer can be summarized at three levels:

Deployment Goal	Hardware Starting Point	Cost Magnitude
Just able to run	8-GPU new-generation accelerator node	From tens of US dollars/hour
Smooth for a single user	8-GPU new-generation accelerator node	Same as above, requires empirical validation
10-way concurrency	8-GPU node may suffice	Same as above, requires attention to scheduling optimization

The core conclusion is: VRAM capacity is the first threshold; weight data of 1.4 TB or more means any physical deployment must be at the multi-GPU server level. But once past the capacity threshold, performance depends more on the optimization capabilities of the inference engine and actual stress test results.

For most developers and small-to-medium enterprises, directly purchasing or long-term renting a full hardware setup is not economical before completing proof of concept. A more pragmatic approach is: first validate model effectiveness through short-term rentals, while evaluating whether smaller-scale alternatives can meet business needs, and then decide whether to expand to production-grade deployment. After all, for the vast majority of application scenarios, directly calling a cloud-hosted model service often offers clear advantages in cost, operational complexity, and iteration speed.