Deploying a Trillion-Parameter MoE Model Locally: Three-Tier Plans and Costs
A practical guide to running a trillion-parameter MoE model locally, covering three performance tiers—from minimal loading to 10 concurrent users at 25 tokens/s each—along with storage math, memory bandwidth limits, communication bottlenecks, and cost estimation formulas.
Deploying a Trillion-Parameter MoE Model Locally: Three-Tier Plans and Costs
1. Define Your Target Tier First
Before deploying a large model, define your goal. Different goals imply very different hardware requirements. Three common tiers:
- Just make it run: The model loads completely and generates tokens—no speed requirement, even single-digit tokens/s is acceptable.
- 25 tokens/s for a single user: Output speed reaches 25 tokens/s during interactive use, close to smooth reading speed.
- 10 concurrent users, 25 tokens/s each: Ten users at once, each with at least 25 tokens/s—total system throughput of roughly 250 tokens/s.
All speed, memory, and hardware figures in this article are estimates based on publicly available architecture information. Once the weights and technical report are officially released, each item needs to be verified with real benchmarks.
2. Model Size Sets the Storage Floor
The model uses a Mixture-of-Experts (MoE) architecture with 896 experts, activating 16 experts per token—an expert activation ratio of about 1.79%. Note that this ratio only describes the sparsity of the expert layers; it is not the strict parameter activation ratio, because shared components such as attention layers participate in every computation.
MoE memory usage is calculated from total parameters, not activated parameters. During inference, the router only invokes 16 experts per token, but all experts must reside in addressable storage; otherwise they cannot be called on demand. The activation ratio affects compute consumption, not storage requirements.
2.1 Calculating Weight Storage
The model uses quantization-aware training: weights in 4-bit format, activations in 8-bit. Weight storage can be estimated in three steps:
- Raw data floor: 2.8 trillion parameters × 4 bits ÷ 8 ≈ 1,400 GB
- Block scaling overhead: Every 32 4-bit elements share one 8-bit scaling factor, adding 0.25 bits per element on average—effectively 4.25 bits. By this calculation, the weights alone come to about 1,487.5 GB
- Other overhead: Non-4-bit tensors, metadata, memory alignment, runtime workspaces, and the KV cache all require additional space
So plan for at least 1.5 TB of weight storage, and expect total memory needs to be even higher.
def weight_gb(params_billion: float, bits: float, block: int = 32, scale_bits: int = 8) -> float:"""Estimate storage for quantized weights (GB), excluding KV cache and runtime overhead."""effective_bits = bits + (scale_bits / block if block else 0)total_bytes = params_billion * 1e9 * effective_bits / 8return total_bytes / 1e9print(weight_gb(2800, 4)) # 1400.0, raw data floorprint(weight_gb(2800, 4)) # ~1487.5 with block scaling (block=32, scale_bits=8)
For reference, a mainstream open-source MoE model with roughly 671B parameters runs on 8 accelerators with 80 GB of memory each, requiring about 400 GB for weights and activations. The model discussed here has roughly 4× the parameter count, so the deployment scale must be scaled up accordingly.
3. Three Deployment Tiers
3.1 Tier 1: Just Make It Run
The core idea here is heterogeneous inference—GPUs and CPUs working together:
- GPU: Hosts the attention layers. These are compute-intensive and need high-bandwidth memory for efficiency.
- System memory (CPU side): Stores all sparse experts. Each token only invokes 16 of them, which are fetched into the compute device based on routing results.
Hardware requirements are driven mainly by memory capacity: the server needs at least 1.5 TB of system memory, plus one or two accelerators for the attention layers.
There are two main speed bottlenecks: the cost of transferring routed experts from memory to the compute device, and CPU-side compute and memory bandwidth. If the expert cache hit rate is low, the same experts get transferred repeatedly, further degrading speed. Single-digit tokens/s is a reasonable expectation for this tier.
In terms of cost, the main expense is a large-memory server; the accelerator count is small, so total investment is relatively manageable.
3.2 Tier 2: 25 Tokens/s for a Single User
At this tier, the speed requirement dictates the architecture: weights must primarily reside in the accelerator’s high-bandwidth memory—no relying on frequent expert transfers from system memory.
Speed is capped by memory bandwidth. During decoding, every generated token requires reading the weights activated for that step. A rough formula:
Per-stream decode speed (tokens/s) ≈ effective memory bandwidth (GB/s) ÷ activated weights read per token (GB)
For example: suppose each token requires reading about 60 GB of activated weights (a hypothetical value for illustration—verify with real measurements). To hit 25 tokens/s, effective bandwidth must be at least about 1,500 GB/s. Multi-accelerator setups can usually meet this number, so the real bottleneck often isn’t bandwidth—it’s communication.
Communication determines latency. With 896 experts spread across many cards, every token must go through two all-to-all communications per layer: first sending activations to the cards hosting the target experts, then retrieving the results. Across the model’s many layers, this latency accumulates. Achieving this tier’s speed typically requires enlarging the communication domain with high-speed interconnects so that most transfers stay within a single fast communication domain rather than crossing slower network links.
Capacity floor calculation:
- Weight floor: 1.5 TB ÷ 80 GB per card ≈ 19 cards
- Accounting for the KV cache, runtime workspaces, and communication buffers, practical planning usually requires 32+ cards
- The official recommended deployment scale is a supernode of 64+ accelerators—a safer starting point
3.3 Tier 3: 10 Concurrent Users at 25 Tokens/s Each
This is the most demanding tier. Total system throughput must reach roughly 250 tokens/s while keeping per-stream latency stable.
Batching amortizes weight reads. Multiple requests share expert weights within the same decode step, reducing weight read counts, and total throughput can scale close to linearly—provided the batch is large enough and the scheduler organizes requests efficiently.
KV cache grows linearly with concurrency. Each stream must store its own history of keys and values, estimated as:
KV cache memory ≈ concurrency × context length × KV bytes per token
With 10 concurrent streams at 32K context each, the KV cache consumes substantial memory. This must be reserved separately from the weights, or you will hit out-of-memory errors as soon as concurrency ramps up.
Communication volume scales up too. The larger the batch, the more activations moved per step, raising interconnect bandwidth requirements accordingly. Hardware for this tier is usually larger than Tier 2, and requires more refined parallelism strategies—for example, combining expert parallelism with pipeline parallelism.
4. How to Calculate Costs
Deployment costs follow two paths: buying hardware to build your own, or renting compute by the hour. Either way, you can estimate with these formulas.
Self-built costs:
- One-time investment = server and accelerator purchase price + networking equipment + data center modifications
- Ongoing costs = power + cooling + operations staff
- Amortize over the expected service life to get an hourly depreciation cost
Rental costs:
- Hourly cost = number of cards × per-card hourly rate + storage and network fees
- Daily cost = hourly cost × hours run per day
Cost per token:
Cost per million tokens ≈ hourly cost ÷ (actual throughput in tokens/s × 3600) × 1,000,000
“Actual throughput” here should be measured under real workloads, not peak figures. Uneven load, request queuing, and warm-up time all reduce utilization.
