Local Deployment of Trillion-Parameter MoE Models: Cost Breakdown and Three-Tier Hardware Strategy

1. Defining the Problem: What Actually Determines Deployment Cost

The most common mistake when discussing the cost of locally deploying a trillion-parameter MoE model is treating parameter count as equivalent to memory footprint, or confusing activation ratio with compute ratio. Before diving into specific configurations, four foundational concepts need to be clarified.

Total Parameters Set the Storage Floor; Activated Parameters Set the Compute Ceiling

Consider a typical sparse MoE architecture with total parameters in the trillion range, composed of hundreds of experts. Each token is routed to only a small subset of experts—for example, 16 out of 896 experts, giving an activation ratio of roughly 1.79%. A key nuance: expert activation ratio is not the same as parameter activation ratio. The router only decides which experts participate in computation, while the shared attention backbone, embedding layers, and output layers remain fully active at all times.

This leads to a critical conclusion:

• Memory (storage) requirements scale with total parameters—all expert weights must reside in addressable storage, with only the activated experts dynamically loaded during inference.
• Compute requirements scale with activated parameters—the parameters actually involved in matrix multiplication are far fewer than the total, which is precisely why MoE architectures can support large capacity at relatively low compute cost.

Quantization Level Directly Determines Weight Volume

The mainstream approach for large-scale MoE inference today is mixed low precision with 4-bit weights and 8-bit activations. Some models also incorporate quantization-aware training from the supervised fine-tuning stage, keeping quantization loss manageable. However, 4-bit does not simply mean 4 bits per parameter. Per common block-wise quantization conventions, every 32 4-bit elements require an accompanying 8-bit scale factor, translating to roughly 4.25 bits per parameter.

A quick calculation for a 2.8-trillion-parameter model:

Raw 4-bit lower bound: 2.8e12 × 4 bit ÷ 8 ≈ 1400 GB
Including scale overhead: 2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This 1487.5 GB represents only the raw weight elements. It does not yet include non-4-bit tensors (such as layers kept at higher precision), metadata, memory alignment padding, runtime workspace, and KV cache. Engineering practice typically calls for an additional 15%–30% headroom on top of this figure.

Calibrating Intuition with a Reference Point

A useful reference: a 671B-class MoE model in FP8 format, including main parameters and multi-token prediction (MTP) parameters, totals approximately 685GB. Even with 4-bit weight quantization, the weights still require about 335GB. Adding roughly 37GB of activation overhead brings the total to around 400GB—just enough to run on 8 accelerator cards with 80GB of memory each (640GB total). A trillion-parameter model is at least 4 times larger, placing the storage floor for full deployment firmly in the 1.5TB range. This is a physical constraint that no optimization trick can bypass.

Communication Bandwidth Is an Underestimated Hidden Cost

Large-scale MoE inference is extremely sensitive to all-to-all communication. With hundreds of experts distributed across multiple cards, every token must have its activations shipped to the device hosting the target expert, and the results shipped back after computation. Communication latency directly eats into compute gains. This is why official recommendations often call for supernodes of 64 or more accelerators connected via high-bandwidth interconnects (such as NVLink or equivalent high-speed buses), keeping expert transfers within a single communication domain. The hardware cost of this step often exceeds the compute cost itself.

2. Three Deployment Tiers and Corresponding Hardware Configurations

The following progression moves from “it runs” to “usable by one person” to “usable by many concurrently.” Note: apart from architecture and publicly available pricing, all speed, memory, and configuration figures are estimates based on public information and must be validated through benchmarking during actual implementation.

Tier 1: Just Make It Run

The only goal is to fit roughly 1.5TB of weights into addressable storage and have the model produce tokens reliably—even at single-digit tokens per second.

The most straightforward approach is GPU + CPU heterogeneous inference. Each MoE layer has a shared attention component that is compute-intensive and should stay on GPU using high-bandwidth compute. Meanwhile, the hundreds of sparse experts—which account for the vast majority of storage but only a tiny fraction of per-token computation—can be offloaded entirely to CPU memory on a large-memory server.

The typical technical pattern is expert offloading with sparse activation scheduling:

# Conceptual illustration: expert weights reside in CPU memory, swapped in on demand based on routing
expert_store = CPUExpertPool(total_experts=896, dtype="int4")
for token in batch:
    experts = router(token)                  # select 16 experts per token
    weights = expert_store.gather(experts)   # move from CPU memory to GPU memory
    out = compute_attention(token) + moe_forward(token, weights)

The cost breakdown is roughly:

Component	Suggested configuration	Estimated cost range
GPU	1–2 accelerators with 80GB memory for attention and scheduling	Tens of thousands of USD
Memory	1.5TB+ DDR5, 2TB recommended for headroom	Tens of thousands of USD
Storage	2–4TB NVMe SSD for weights and cache	Thousands of USD
Platform	Server motherboard and power supply supporting large memory	Tens of thousands of USD

Total cost falls approximately in the USD 20,000–45,000 range. The tradeoff is extremely low throughput and high first-token latency, making this suitable only for validation, debugging, and functional demos.

Tier 2: Single User at 25 Tokens per Second

25 tokens/s is an empirical threshold for comfortable reading. Achieving this requires moving beyond the data transfer bottleneck of CPU offloading and toward GPU high-bandwidth memory. The approach shifts to high-end single-node multi-GPU with unified high-bandwidth memory.

Key design points:

• Layered weight placement: the attention backbone and high-frequency experts stay in GPU memory; low-frequency experts remain in CPU memory or are fetched on demand over high-speed interconnect.
• Memory capacity over raw compute: the bottleneck at this tier is capacity and bandwidth, not FLOPS. Prioritize total node memory that covers most hot weights.
• KV cache needs a separate budget: in long-context scenarios, KV cache can consume tens of gigabytes and must be accounted for separately from weight memory.

A viable reference configuration is 8 accelerators with 80GB each (640GB total GPU memory) paired with 1–2TB of system memory, connected via a hybrid PCIe/NVLink topology. Including card prices and system integration, total cost is typically in the hundreds of thousands of USD. This configuration supports smooth single-user interaction but lacks multi-user concurrency.

Tier 3: 10 Concurrent Users at 25+ Tokens per Second, 4-bit or Higher Quantization

This is the threshold for true production viability: 10 concurrent users × 25 tps = 250 tps of aggregate throughput, with weight precision no lower than 4-bit mixed quantization. A single node can no longer handle this; a supernode cluster becomes necessary.

The core challenge is not compute but communication:

• Roughly 1.5TB of weights must be distributed across dozens of cards, making expert parallelism (EP) mandatory.
• Per-token all-to-all transfer volume grows linearly with concurrency, so the communication domain must be large and fast enough.
• Supporting infrastructure is required: request scheduling, continuous batching, prefix caching, and other inference serving optimizations.

A typical configuration is a supernode of 64+ accelerators with a high-speed interconnect fabric. Single-cluster cost is typically in the millions of USD. With tighter budgets, a 32-card configuration with more aggressive expert placement and request scheduling can be used, at the cost of reduced concurrency or shorter context lengths, bringing cost down to the several hundred thousand to low millions of USD range.

A comparison of the three tiers:

Tier	Target	Key bottleneck	Typical scale	Cost range
L1	Runs at all	Storage capacity	1–2 GPUs + 2TB RAM	USD 20,000–45,000
L2	Single user, 25 tps	GPU memory bandwidth and capacity	8-GPU single node + large RAM	Hundreds of thousands of USD
L3	10 users × 25 tps	All-to-all communication	64-card supernode	Millions of USD

3. Four Categories of Hidden Costs That Are Easy to Overlook

1. Quantization is not a free lunch. Mixed 4-bit quantization compresses weights below 1.5TB, but activations, scale factors, and non-quantized layers constitute additional overhead. Capacity planning should reserve 1.2×–1.3× of the raw data size.

2. KV cache scales linearly with concurrency and context length. Under 10-user concurrency with long contexts, KV cache can reach hundreds of gigabytes and must be included in the memory budget. Consider paged attention and cache reuse mechanisms where needed.

3. The silent cost of communication fabric. Switches, cables, topology design, and tuning for a supernode deployment often cost as much as the accelerators themselves, yet are the most frequently overlooked line item.

4. Operations and power. A 64-card cluster draws tens of kilowatts. Data center modifications, cooling, and long-term electricity costs must be modeled separately in TCO calculations.

4. Implementation Recommendations

• Do capacity planning before hardware selection: estimate the weight floor using “total parameters × bits per parameter ÷ 8 × safety factor,” then add KV cache and workspace to derive the real memory requirement.
• Work backward from your target: validation stages can use heterogeneous offloading without overinvesting. Only commit to a cluster once concurrency and latency SLAs are clearly defined.
• Benchmark everything after weights are released: all speed, memory, and concurrency figures are estimates until official weights and technical reports are available. Use measured throughput, first-token latency, and long-context performance as the final basis for hardware decisions.
• Evaluate managed and dedicated instances first: when self-hosting TCO exceeds expectations, compare unit token costs of dedicated cloud instances or managed inference services before deciding whether to build or rent.

Local deployment of trillion-parameter MoE models is fundamentally an engineering tradeoff among storage capacity, communication bandwidth, and concurrent throughput. Spending on the actual bottleneck is more important than stacking more accelerators.