Local Deployment of Trillion-Parameter MoE Models: Cost Breakdown and Three Implementation Tiers

1. The Core Problem: Fitting 1.5TB of Weights into Addressable Storage

When we talk about locally deploying a Mixture-of-Experts model with a total parameter count in the trillions, the real question is not “can the model run at all” but rather how to fit nearly 1.5TB of weights into an addressable, schedulable storage hierarchy while continuously producing tokens.

Take a 2.8-trillion-parameter MoE architecture as an example. Its key characteristics are:

• Number of experts: 896 experts, with 16 activated per token, giving an expert activation ratio of roughly 1.79%.
• Backbone structure: A combination of linear attention and attention residuals, with gated MLA as one component.
• Quantization scheme: Quantization-aware training introduced from the supervised fine-tuning stage, using 4-bit floating-point weights and 8-bit floating-point activations.

A common misconception here is that MoE memory usage is determined by total parameters, not activated parameters. The activation ratio only affects compute consumption, not storage. At inference time, all 896 experts must reside in the storage hierarchy, and the router decides which 16 to invoke for each token.

The Lower Bound for Raw Weight Data

Converting 2.8 trillion parameters to 4-bit yields a raw data lower bound of about 1400GB. However, 4-bit floating-point formats are not “pure 4-bit.” Under the widely adopted OCP microscaling specification, every 32 4-bit elements carry an 8-bit scale factor, resulting in an effective bit width of:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB

This does not yet account for non-4-bit tensors, metadata, memory alignment, runtime workspace, or KV cache. In other words, 1.5TB is only an optimistic starting point.

For reference, a 671B-class MoE model in 8-bit format requires about 685GB for main parameters plus multi-token prediction parameters. Even with 4-bit weight quantization, roughly 335GB of weights and 37GB of activations—about 400GB total—are needed to run on eight 80GB accelerators (640GB combined). A 2.8-trillion-parameter model is more than four times larger, which sets the storage lower bound for a full deployment.

2. Communication: The Underestimated Bottleneck

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple accelerators, every token must have its activations moved to the cards hosting the target experts after routing, then moved back after computation. Communication latency along this path directly eats into compute performance.

A common industry response is to use supernode architectures—expanding the communication domain with high-speed interconnects such as NVLink or protocols with equivalent bandwidth, so that expert transfers never leave a single communication domain. The official deployment recommendation for models of this class is typically a supernode of 64 or more accelerators.

However, this approach is extremely expensive and unrealistic for most teams. We therefore need to work backward from the target use case and design deployment solutions across three tiers.

3. Three Deployment Tiers and Cost Estimates

Tier 1: Just Make It Run, Speed Not a Priority

Goal: Fit nearly 1500GB of weights into addressable storage and have the model produce tokens, even at single-digit throughput.

Approach: GPU + CPU heterogeneous inference. The key insight is that each MoE layer contains a shared attention component (compute-intensive), while the 896 sparse experts dominate storage but only 16 are activated at a time.

The compute-intensive attention component can therefore stay on GPUs for high-bandwidth computation, while all sparse experts are offloaded to CPU memory on a large-memory server.

Example hardware configuration:

Component	Configuration	Notes
GPU	1–2 consumer cards with 24GB each	Handles attention and routing computation
Memory	1.5–2TB DDR5	Holds all expert weights
Storage	2TB NVMe	Model loading and swapping
CPU	Server platform with multi-channel memory	Memory bandwidth determines expert read speed

Cost range: Approximately $11,000–$21,000 (varies with memory capacity and GPU tier).

Expected performance: Single-user output of roughly 1–5 tps; time-to-first-token may reach tens of seconds. Suitable for validation, research, and demo scenarios.

Key constraint: CPU memory bandwidth is the hard bottleneck. All 896 experts must traverse PCIe and memory channels, so actual throughput depends on the number and frequency of memory channels. Server platforms with 12 or more memory channels are strongly preferred.

Tier 2: Single User, 25 tps Output

Goal: Smooth single-user interaction with stable output at 25 tps or higher.

Approach: 25 tps means each token must be computed and moved within 40ms. Feeding experts purely from CPU memory is no longer sufficient. Frequently activated experts must be cached in GPU memory or high-speed caches, forming a tiered expert cache.

Example hardware configuration:

Component	Configuration	Notes
GPU	4–8 accelerators with 80GB each	Attention + hot expert cache
Total GPU memory	320–640GB	Can cache approximately 20%–40% of experts
Memory	1.5TB or more	Cold experts reside here
Interconnect	High-speed inter-card interconnect	Reduces expert transfer latency

Cost range: Approximately $110,000–$280,000.

Expected performance: Single-user 25–40 tps; time-to-first-token of 1–3 seconds.

Key constraint: Expert hit rate determines actual speed. A hot-expert prefetching mechanism based on routing statistics is required—before inference, the experts most likely to be activated are preloaded into GPU memory based on historical routing distributions.

# Hot expert cache illustration (pseudocode)
class ExpertCache:
    def __init__(self, gpu_capacity, cpu_experts):
        self.hot = {}          # Experts in GPU memory
        self.cold = cpu_experts # Experts in CPU memory
        self.hit_stats = {}    # Routing hit statistics
    def route(self, token):
        expert_ids = router(token)
        for eid in expert_ids:
            if eid not in self.hot:
                self._promote(eid)   # Swap from CPU memory to GPU memory
            self.hit_stats[eid] += 1
        return [self.hot[e] for e in expert_ids]
    def _promote(self, eid):
        # Evict based on LRU + hit frequency
        victim = self._select_victim()
        self.cold[victim] = self.hot.pop(victim)
        self.hot[eid] = self.cold.pop(eid)

Tier 3: 10 Concurrent Users, 25+ tps per User

Goal: 10 concurrent requests, each with output no lower than 25 tps, at no less than 4-bit quantization.

Approach: 10 concurrent users means aggregate throughput must reach 250 tps. At this point, the communication domain must be large enough; otherwise all-to-all transfers become an unacceptable overhead.

Example hardware configuration:

Component	Configuration	Notes
Accelerators	32–64 cards	Supernode scale
Total accelerator memory	2.5–5TB	All experts reside in accelerator memory
Interconnect	High-speed interconnect domain	Expert transfers stay within the domain
Memory	2TB or more	KV cache and workspace

Cost range: Approximately $1.1 million–$2.8 million.

Expected performance: 10 concurrent users × 25+ tps; time-to-first-token of 500ms–1s.

Key constraint: The bottleneck shifts from storage to communication and scheduling. Required capabilities include:

1. Expert parallelism strategy: Distribute 896 experts evenly across all cards, with each card hosting roughly 14–28 experts.
2. Load balancing: The router must be aware of per-card load to prevent hot experts from causing congestion on a single card.
3. KV cache management: KV cache usage across 10 concurrent requests is significant and requires paged attention and cache reuse mechanisms.

4. Cost Structure Breakdown

Breaking down the costs across the three tiers reveals the following distribution:

Cost item	Tier 1	Tier 2	Tier 3
Accelerators	5%	60%	75%
Memory	60%	15%	8%
Interconnect and networking	5%	10%	12%
Storage	10%	5%	2%
Power and cooling	20%	10%	3%

As shown, Tier 1 costs are dominated by memory, while Tier 2 and Tier 3 costs are dominated by accelerators and interconnect. This explains why the cost gap between “it can run” and “it runs fast” spans orders of magnitude.

5. Recommended Deployment Path

For most teams, the pragmatic path is:

1. Validate feasibility at Tier 1 first: Use heterogeneous inference to run the full pipeline end to end, confirming model behavior, routing distributions, and expert activation statistics.
2. Collect routing data, then optimize: Design the hot-expert caching strategy based on real routing distributions, and evaluate whether Tier 2 is worth the investment.
3. Decide on Tier 3 based on business concurrency needs: If the use case is internal only, Tier 2 is usually sufficient. If external service is required, consider a supernode deployment.

One important caveat: all speed, memory, and hardware figures above are estimates based on publicly available information. After the weights and technical report are officially released, each item must be verified through benchmarking. In particular, the following variables can significantly affect actual results:

• Whether the actual expert activation distribution is uniform.
• Precision loss from 4-bit quantization in real inference.
• Actual KV cache usage.
• Effective interconnect bandwidth.

6. Summary

Locally deploying a trillion-parameter MoE model is fundamentally a storage hierarchy design problem: where to place 1.5TB of weights, how to move them, and how to cache them. GPU + CPU heterogeneous inference solves the “can it run” problem, tiered expert caching solves the “how fast can it run” problem, and supernodes solve the “can it serve multiple users simultaneously” problem.

The three tiers correspond to three orders of magnitude in cost: tens of thousands, hundreds of thousands, and millions of dollars. Which tier to choose depends on whether your goal is validation, internal use, or production service.