Cost and Hardware Strategies for Locally Deploying Trillion-Scale MoE Models

1. The Core Challenge: Deploying a Distributed System, Not Just a Model

When a model’s parameter count reaches the trillion scale, the difficulty of local deployment shifts from selecting a single GPU or server to designing a coordinated system for storage, compute, and communication. Take a MoE architecture with approximately 2.8 trillion total parameters and 16 activated experts per token as an example. While it features sparse activation, the inference system must still place all expert weights into addressable storage before dynamically reading the activated portions based on routing results. Therefore, “sparsity” only reduces the compute required for a single forward pass; it does not lower the total storage capacity threshold.

This means a deployment plan must answer four questions simultaneously:

1. How much total storage is required for all weights, quantization scales, runtime workspaces, and the KV cache;
2. Whether this storage should reside in GPU VRAM, CPU memory, or slower external media;
3. Whether the communication cost of routing tokens to experts on different devices will negate the computational benefits;
4. Given a specific concurrency level, output speed, and quantization precision, what the magnitude of the hardware cost will be.

This article explores these questions across three tiers—“just getting it to run,” “single-user usability,” and “multi-concurrency service”—providing estimation methods rather than tying recommendations to specific hardware models or vendors.

---

2. Calculating Storage First: Why “Activated Parameters” ≠ “Memory Footprint”

Many developers mistakenly estimate an MoE model’s memory footprint based solely on its “activated parameters.” For instance, if only 16 experts are activated per token, it might seem that only a small fraction of the weights needs to be loaded. However, in an actual inference system, routing decisions occur at runtime, and different tokens will hit different experts. If the system had to fetch expert weights from disk or remote storage on the fly for every token, the latency would be unacceptable. Therefore, the standard engineering practice is to keep all expert weights resident in high-speed addressable storage, while the GPU or accelerator only reads the activated portions for computation.

Using the 2.8 trillion parameter model as an example, if we apply basic 4-bit integer quantization, the raw weight lower bound is approximately:

2.8T × 4 bit ÷ 8 = 1400 GB

However, when using block quantization formats, every 32 4-bit weight elements are accompanied by an 8-bit scale. Based on standard specifications, the effective bits per weight element roughly calculate to:

4 bit + 8 bit ÷ 32 ≈ 4.25 bit

Therefore, the combined weights and scales total approximately:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This figure does not yet include the following additional allocations:

• Non-4-bit tensors, such as embeddings, output layers, and certain normalization parameters;
• Metadata, routing tables, and expert index structures;
• Tensor alignment and fragmentation losses;
• Runtime workspaces, such as intermediate activations and temporary buffers required for inference (though gradient-related buffers are not needed);
• KV cache, which grows rapidly in long-context and multi-concurrency scenarios.

Therefore, during actual deployment, the theoretical weight volume should be viewed as a “lower bound” rather than the “final configuration.” When planning hardware, it is recommended to reserve at least 15% to 25% additional space for non-quantized weights, runtime buffers, and the KV cache. For long-text services, the KV cache alone might occupy tens to hundreds of gigabytes of VRAM and must be modeled separately.

---

3. Compute and Communication: The Real Bottlenecks of MoE Inference

Storage is just the entry barrier; communication is often the critical factor determining system viability. MoE forward computation typically consists of two paths:

1. Shared Path: Includes attention mechanisms, shared feed-forward layers, and routing computations. This part is compute-intensive and well-suited for high-compute devices.
2. Expert Path: All experts are distributed across different accelerators. Each token is sent to a small number of expert devices based on routing results, computed, and then the results are returned.

The biggest issue introduced by expert distribution is all-to-all communication. Assuming experts are distributed across dozens of accelerator cards, every token must undergo the process of “routing decides target expert → activation transferred to target card → expert computes → result transferred back.” If the communication domain is small or interconnect bandwidth is insufficient, inference throughput will degrade rapidly as the number of devices increases.

Therefore, ultra-large-scale MoE inference typically requires a super-node hardware organization rather than simply stacking independent servers. The core objective is to confine expert communication to a high-bandwidth, low-latency communication domain as much as possible. Common practices include:

• Using high-bandwidth interconnects to group multiple accelerators into a single communication domain;
• Placing frequently co-activated experts within the same communication domain to reduce cross-domain transfers;
• Reducing worst-case communication hops through expert placement strategies;
• Applying different parallelism strategies to different layers to balance compute and communication.

The communication pressure can be estimated as follows: Assume a single token activates 16 experts. Although the activation tensor transferred for each expert is not large, under high concurrency, the frequency of communication increases linearly or even super-linearly. If the expert computation time is extremely short while communication latency is high, the effective device utilization will drop sharply. At this point, even doubling the compute capacity might not yield a noticeable improvement in throughput.

---

4. Tier 1: Just Getting It to Run (Speed Irrelevant)

The deployment goal for the first tier is to fit all ~1500 GB of weights into addressable storage so the model can generate tokens, even if it only outputs a single-digit number per second. The core contradiction at this tier is balancing “total storage” against “cost per unit capacity.”

Relying entirely on high-end accelerator VRAM is feasible, but it would require at least dozens of cards to accommodate all weights and runtime space. For most developers or small teams, this upfront investment is prohibitively high. A more realistic approach is heterogeneous CPU-GPU inference.

The engineering logic behind this approach is:

1. Keep the shared attention component on the accelerator. This part has a small parameter count but is compute-intensive, making it suitable for high-bandwidth, high-compute devices.
2. Offload the expert layers to CPU memory. The expert layers account for the vast majority of parameters, but each token only activates a tiny fraction. Thus, they can reside in large-capacity memory and be read on demand.
3. Let memory bandwidth dictate expert computation speed. CPU-side inference speed is typically bottlenecked by memory read bandwidth rather than floating-point compute power.
4. Exchange intermediate results between the accelerator and CPU via a high-speed bus. Although the single-transfer latency is higher than intra-device communication, it is acceptable for a “just run” scenario.

Assuming a single server can be configured with several terabytes of memory, a combination of a few accelerators and a massive-memory server can successfully load the full weights. The output speed might only be a few tokens per second, but this is sufficient for verifying model behavior, running offline batch processing, or conducting functional testing.

The cost for this tier is primarily driven by the large-memory server, a few accelerators, and high-speed storage. Compared to an all-accelerator solution, the overall cost can be reduced by several times or even an order of magnitude, at the expense of extremely low speeds, making it unsuitable for interactive use.

---

5. Tier 2: Single-User Interaction at 25 tps

When the goal shifts to “achieving an output speed of about 25 tokens per second for single-user interaction,” the system must balance both latency and throughput. At this tier, not only must all weights reside in faster media, but each decoding step must also be completed within a strict time limit.

An output of 25 tps means generating one token every 40 milliseconds. Note that decoding is an autoregressive process; the output of one token depends on the result of the previous one. Therefore, the optimization target is not to increase total throughput, but to reduce single-step latency. For MoE models, the single-step latency is composed of the following serialized parts:

Total Latency ≈ Input processing latency
               + Routing decision latency
               + Activation communication latency
               + Shared layer compute latency
               + Expert layer compute latency
               + Output processing latency

Communication latency and expert layer compute latency usually account for the majority of this time. To achieve 25 tps, the following conditions must be met:

• Expert weights must be kept as close to the compute unit as possible to reduce cross-device reads;
• Expert communication for a single token should be completed within the same high-speed communication domain;
• No single device should be overloaded, or its compute capacity will become a bottleneck;
• Memory and VRAM bandwidth must match the target speed.

A rough estimate: If the expert weights read during a single decoding step are on the order of tens of gigabytes, achieving the 40-millisecond single-step target requires an effective read bandwidth of at least hundreds of GB/s or even TB/s. This is where the high-bandwidth memory (HBM) of high-end accelerators proves its value. Although CPU memory offers large capacity, its bandwidth is typically more than an order of magnitude lower than that of high-end accelerator VRAM. Thus, relying solely on the CPU makes it nearly impossible to push speeds to 25 tps.

Therefore, Tier 2 usually requires at least a portion—and often the majority—of expert weights to be moved back into accelerator VRAM. The exact ratio depends on the expert activation patterns and communication topology. In practice, tiered storage can be employed: high-frequency experts reside in VRAM, while low-frequency experts are placed in CPU memory or slower media, using prefetching and caching strategies to reduce average latency. The cost of this solution will be significantly higher than Tier 1, but lower than a full-VRAM deployment.

---

6. Tier 3: 10 Concurrent Requests at 25 tps Per User

The third tier introduces the variable of “10 concurrent requests,” fundamentally changing the nature of the problem. Concurrency is not simply multiplying the single-user cost by 10; it amplifies communication, VRAM bandwidth pressure, and scheduling complexity.

Under 10-way concurrency, the system must simultaneously process multiple tokens at different decoding stages. These tokens may route to entirely different sets of experts. This means:

1. VRAM bandwidth requirements surge. The total volume of expert weights read per step grows roughly linearly with the concurrency level.
2. All-to-all communication conflicts intensify. Tokens from different requests may simultaneously send activations to the same device, creating communication hotspots.
3. KV cache expands rapidly. Each concurrent request has its own independent context. Under long-context scenarios, the KV cache can easily become the new VRAM bottleneck.
4. The scheduler must be highly efficient. It needs to dynamically balance the load across devices, preventing some expert devices from idling while others face queuing backlogs.

Consequently, Tier 3 typically implies organizing more accelerators into a super-node, and maintaining a quantization precision floor to ensure output quality. For system design, the question to re-evaluate is no longer “can we fit the weights,” but rather “can we handle the concurrent workload within latency constraints.”

A simplified formula can express the throughput upper bound:

Effective Throughput ≈ min(
    Compute upper bound,
    VRAM/Memory bandwidth upper bound,
    Communication upper bound
) × Scheduling efficiency

In concurrent scenarios, the communication upper bound is often reached before the compute upper bound. Expanding the communication domain scale, optimizing expert placement, and adopting more efficient quantization formats can all mitigate this issue, but at the cost of increased hardware, leading to a substantial rise in total cost.

The cost at this tier is no longer just a few accelerators and a large-memory server; it usually requires a small cluster or super-node. For individual developers, Tier 3 is generally not suitable for self-hosting and is better served by renting on-demand super-node instances in the cloud. Enterprise users need to evaluate the long-term cost difference between building in-house and renting.

---

7. Trade-offs Between Quantization Precision and Speed

Quantization is the core method for reducing the deployment costs of trillion-scale models. Typically, weights use 4-bit block quantization, while activations can use 8-bit or higher precision to maintain inference quality. Key considerations for quantization include:

• Whether Quantization-Aware Training (QAT) was introduced during the training phase, which significantly affects model performance at low bitrates;
• How block size is configured—smaller blocks increase scale overhead but may reduce quantization error;
• Which layers are sensitive to quantization and require higher precision;
• Whether a mixed-precision strategy is used, applying different bit widths to different layers.

In system design, quantization impacts not only storage volume but also bandwidth and compute. Low-bit weights reduce the data volume that needs to be read per token, effectively increasing available bandwidth. However, if quantization introduces noticeable quality degradation, you must increase activation precision or adjust the quantization strategy, which inversely increases compute and storage overhead.

Therefore, the three deployment tiers often correspond to different quantization configurations:

• Tier 1 can use the most aggressive quantization to lower the storage threshold;
• Tier 2 requires finding a balance between speed and quality;
• Tier 3, due to high concurrency and multi-user service orientation, should generally maintain a minimum quantization precision floor to avoid perceptible quality fluctuations.

---

8. How to Estimate Deployment Costs

When estimating costs, it is important to distinguish between “upfront investment” and “pay-as-you-go.” For local deployment, the primary cost components include:

1. Accelerator costs: Tied to VRAM capacity, VRAM bandwidth, compute power, and interconnect capabilities;
2. CPU and memory costs: Account for a high proportion in heterogeneous deployments or expert offloading strategies;
3. Intra-node interconnect costs: E.g., high-speed buses, switch chips, and cables;
4. Inter-node network costs: High-performance network equipment is required if cross-node networking is necessary;
5. Storage costs: For storing weights, runtime images, and logs;
6. Power and cooling costs: The hidden expenses of running high-power consumption equipment over the long term cannot be ignored;
7. Maintenance and operations costs: Including monitoring, alerting, scheduling, and fault recovery.

A practical estimation approach is:

Total storage requirement → Divide by per-device available VRAM/Memory
→ Obtain minimum device count
→ Estimate if bandwidth and compute meet target latency and concurrency
→ If not, increase device count or upgrade device specs
→ Aggregate hardware, network, power, and operations costs

Using the ~1500 GB weight lower bound as an example: if a single device has 80 GB of available VRAM, a pure-VRAM solution requires at least 19 to 20 devices; if a device has 128 GB, it requires about 12 devices. If CPU memory offloading is used, Tier 1 deployment can be achieved with a few accelerators plus several terabytes of memory, though speed will drop significantly.

Once concurrency and speed requirements increase, simply looking at “device count” becomes meaningless. The evaluation must factor in per-device VRAM bandwidth, inter-device interconnect bandwidth, and communication topology. In actual projects, it is recommended to first conduct small-scale experiments to measure single-token latency, per-step communication volume, and bandwidth utilization, and then extrapolate performance when scaling to the target size.

---

9. Practical Recommendations

For developers or teams, before a model’s weights and official deployment guidelines are fully released, blindly purchasing hardware is not advisable. The following preparatory work can be done in advance:

1. Conduct deployment validation with smaller-scale MoE models. Measure communication proportions, VRAM usage, and scheduling efficiency under the same architectural paradigm to form a reusable estimation model.
2. Build a parameterized cost model. Use parameter count, quantization bit width, concurrency, target tps, device VRAM, and bandwidth as variables to output the required device count and recommended topology.
3. Evaluate the feasibility of heterogeneous deployment. If Tier 1 meets your needs, CPU memory offloading can drastically reduce costs.
4. Focus on super-nodes and high-speed interconnect technologies. Once entering Tier 2 and Tier 3, communication capacity becomes just as critical as VRAM capacity.
5. Allocate sufficient time for benchmarking. Once weights are released, use real workloads as the standard and calibrate previous estimates with actual data.
6. Consider the complementary value of cloud elasticity. For sporadic high-concurrency demands or validation phases, cloud super-node instances can avoid upfront hardware investments while providing a repeatable experimental environment.

Locally deploying a trillion-scale MoE model is not a simple “configuration choice” problem, but a system design challenge involving storage, compute, communication, quantization, and scheduling. Only by clearly breaking down the constraints and validating step-by-step along the path of “can run — usable — serviceable” can you arrive at a truly practical solution, rather than stopping at back-of-the-napkin estimates.