Trillion-Parameter MoE LLM Local Deployment: Cost, Memory, and Throughput Sizing Guide
This guide uses an open-source Mixture-of-Experts (MoE) model with roughly 2.8 trillion total parameters, 896 experts, and 16 activated experts per token as a case study to explain how to size local deployment. It defines three practical targets: merely running the model, delivering 25 tokens/s for a single user, and supporting 10 concurrent users with at least 25 tokens/s per user. It covers VRAM estimates, scaling-factor overhead, interconnect requirements, super-node design, hardware cost estimates, self-hosting versus cloud economics, optimization practices, and benchmarking steps.
Introduction: Ask About Speed and Cost First, Not Just Whether It Can Run
In open-source communities, questions like “Can this very large model be deployed locally?” often start with a conclusion: “Eight 80GB GPUs are enough,” or “A well-configured server can run it.” Once “it can run” is confused with “it can deliver production-level throughput,” the result can differ by an order of magnitude.
A notable recent public example is an open-source MoE model with roughly 2.8 trillion total parameters. Its scale is more than four times that of the previous 671B-class MoE generation, and the VRAM, speed, and cost required for local deployment scale up accordingly. This article uses that model as a reference and breaks down cost planning into three target profiles:
- Profile 1: It only needs to run; speed is not critical.
- Profile 2: A single user achieves at least 25 tokens/s output speed.
- Profile 3: 10 concurrent users are supported, with at least 25 tokens/s output speed per user.
One caveat up front: except for the official recommendation to use a super node with 64 or more accelerators, the speed, VRAM, and hardware figures in this article are estimates based on public information. Once complete model weights are released, they should be verified with benchmark tests. The compute and pricing parameters below are modeling assumptions only; actual procurement and quotes should be based on current market conditions.
1. Understand the Model: MoE Sparsity Does Not Mean Light Weight
Key Parameters
The model has about 2.8 trillion total parameters and uses an MoE architecture with 896 experts. Each token activates 16 experts, giving an expert activation ratio of approximately:
16 ÷ 896 ≈ 1.79%
A common point of confusion is that 1.79% is the expert activation ratio, not a strict parameter activation ratio. The model also contains shared attention modules at each layer—KDA and Gated MLA in this case—as well as routing, normalization, and embedding layers. These components participate in computation for every token. Therefore, the share of parameters actually involved in the forward pass is higher than 1.79%. The exact number will only be clear after the weights are released.
VRAM Is Based on Total Parameters; Activation Ratio Only Affects Compute
MoE is often described as “computing with only a small portion of the parameters,” which leads some people to assume that VRAM requirements shrink by the same ratio. That is a misunderstanding. MoE VRAM usage is driven by total parameters. The expert activation ratio mainly affects compute cost per step.
Think of a library. Even if only a few books are borrowed at a time, the shelves must still hold every book. During inference, the router can decide which books to retrieve for each step, but it cannot decide whether the shelves should exist. All 896 experts must be stored in addressable memory, and then 16 experts are selected per token for computation.
This leads to the first basic rule: any estimate that uses “only tens of billions of activated parameters” to calculate VRAM requirements underestimates deployment difficulty.
2. VRAM Math: 1.4 TB Is Only the Starting Point
FP16 Is 5.6 TB; Even at 4-bit, It Is Still About 1.4 TB
At 16-bit precision, 2.8 trillion parameters require about 5.6 TB just for weights.
This model introduces quantization-aware training (QAT) from the SFT stage and uses a combination of 4-bit weights and 8-bit activations during inference. Even at 4-bit, the bare minimum is approximately:
2.8×10¹² × 4 bit ÷ 8 = 1.4×10¹² bytes ≈ 1.4 TB
Do Not Forget Scaling Factors
“4-bit” does not mean every weight occupies exactly 4 bits. In microscaling formats such as MXFP4, weights are grouped into blocks of 32 elements, and each group has an 8-bit shared scaling factor to extend dynamic range. The approximate overhead, including weights and scaling factors, is:
2.8×10¹² × (4 + 8/32) bit ÷ 8 ≈ 1.49×10¹² bytes ≈ 1487.5 GB
This adds only about 87.5 GB over the bare 1.4 TB. What is easier to overlook is the remaining overhead: unquantized tensors such as embeddings, routing, and normalization layers; framework metadata; memory alignment; runtime workspaces; and growing context state. Adding these together, the practical VRAM requirement is likely to be in the 1.6–1.8 TB range. Engineering plans should leave at least 10%–20% headroom.
Comparison: What Does 4x Scale Mean?
The previous 671B-class MoE model, including about 14B auxiliary prediction parameters, requires roughly 685 GB in FP8 format. Eight 80GB accelerators provide 640 GB total, which is not enough, so quantization is required. With mainstream 4-bit weight quantization, weights take about 335 GB. Adding activations and workspace brings the total to roughly 400 GB, allowing it to run on eight 80GB accelerators.
The new model has more than four times the parameter count. Even at 4-bit, its weights alone are about 2.3 times the total VRAM of an eight-accelerator node. This is why the official recommendation points to a super node with 64 or more accelerators.
Why a Super Node Is Needed
Large-scale MoE inference is highly sensitive to all-to-all communication bandwidth and latency. The 896 experts are distributed across different accelerators. After routing, each token’s hidden state must be sent to the accelerator hosting the target expert, computed there, and then returned to the main path. Larger batches and more layers increase this data movement.
If the communication domain is split by a standard data-center network with typical bandwidth of 100–400 Gb/s, cross-node round-trip latency can eat into compute gains. The result is poor GPU utilization and delayed token generation.
Super nodes use high-speed inter-accelerator interconnects, such as NVLink or equivalent private interconnects, or RDMA-based high-bandwidth, low-latency networks, to place dozens of accelerators in one large communication domain. This keeps expert movement inside the high-speed interconnect boundary as much as possible. At its core, the super node solves the communication-domain problem, not merely the VRAM capacity problem.
3. Three Target Profiles: One Model, Three Deployment Strategies
Profile 1: It Only Needs to Run
Goal: The model can load and generate complete outputs. A single-digit number of tokens/s is acceptable.
Approach: Use heterogeneous GPU/CPU inference. This is the most practical path for running an ultra-large MoE model on a single node. The general idea is to let different components do what they do best:
- Compute-intensive shared components: The attention modules present in every layer, KDA and Gated MLA in this model, are not the largest part of the parameters but are computed for every token. They should stay on GPUs and benefit from high memory bandwidth.
- Storage-intensive sparse experts: The 896 experts hold most of the parameters, but only 16 are used per token. They can be offloaded to large-capacity CPU memory and fetched as needed.
Reference case: One heterogeneous inference framework has published a demo using a dual-socket server-class CPU with one mainstream consumer GPU with 24GB of VRAM, running a 4-bit version of a 671B-class MoE model with about 377 GB of weights. Under constrained expert-selection policies, prefill speed reached several hundred tokens/s, and decode speed was about 14 tokens/s.
Extrapolation: The new model is roughly four times larger, and expert weights alone approach 1.5 TB. To maintain comparable behavior, such a setup typically needs more than 2 TB of CPU memory and as many memory channels as possible to improve fetch bandwidth. The result is more likely to be in the single-digit tokens/s range.
Important limitation: “Can load” is not the same as “can be used.” Context length, batch size, and expert-selection policy all affect experience. This profile is suitable for validating outputs and debugging prompts, not for providing a service.
Profile 2: 25 tokens/s for a Single User
Compute demand is modest. The hard parts are VRAM and communication.
During decoding, every generated token requires reading the expert weights activated for that token. A rough estimate is:
1487.5 GB × (16 ÷ 896) ≈ 26.6 GB
Adding shared attention and other components, it is reasonable to estimate about 30 GB per token. At 25 tokens/s, aggregate memory-read bandwidth must be at least about 750 GB/s. Accounting for memory-access efficiency, planning should assume more than 1 TB/s. This is within the range of high-end GPU memory bandwidth, suggesting that bandwidth alone is not the hardest constraint; spreading the weights across devices and making routing communication efficient are more important.
On the compute side, using “2.8T total parameters × 1.79% ≈ 50B activated parameters” as a rough estimate, each token requires about 0.1 TFLOPs of forward computation. At 25 tokens/s, this is only about 2.5 TFLOPS, far below the dense compute peak of modern GPUs. The main determinants of speed are memory access and communication, not raw compute.
Configuration recommendation: A single eight-accelerator node with 640 GB cannot hold about 1.5 TB of weights. At least two to three eight-accelerator servers are needed, forming a cluster of 24–32 accelerators. Accelerators within a node should use high-speed private interconnect, and nodes should be connected with a high-speed RDMA network.
Optimization: Default routing may make some expert-hosting nodes hotspots. Topology-aware scheduling can help, for example by placing frequently conflicting experts in the same communication domain or on the same node. This follows the same principle as expert-selection policies in Profile 1: instead of changing the model, shorten the communication path.
Profile 3: 10 Concurrent Users, Each at ≥25 tokens/s
Moving from “one user at 25 tokens/s” to “10 users each at 25 tokens/s” is not a linear increase in difficulty:
- Throughput: Aggregate output must reach 250 tokens/s. At 30 GB per token, expert-weight reads alone require about 7.5 TB/s.
- State cache: Context state grows with concurrency and context length. Longer contexts make this more significant, so space must be reserved from the weight budget.
- Prefill pressure: Ten concurrent streams mean more and longer inputs must be processed in parallel, multiplying all-to-all communication frequency.
- Per-accelerator compute: Using the rough activated-parameter estimate, each stream at 25 tokens/s needs about 2.5 TFLOPS. Ten streams total about 25 TFLOPS. With two-way tensor parallelism, each accelerator would need to sustain about 12.5 TFLOPS, or roughly 25% dense compute utilization. That is not a low bar for MoE, because routing imbalance, underutilized expert compute units, and communication stalls all reduce effective compute. Therefore, this profile usually requires a super node with 64 or more accelerators and may use higher-precision weights, such as FP8 or above, to trade some efficiency for stability.
Profile 3 is essentially a capacity-planning problem. If a measured single-stream full-speed concurrency capability is N streams, stably supporting 10 streams with at least 25 tokens/s per stream usually requires 20%–30% headroom, meaning the cluster should be planned for about N ≥ 13–15 streams.
Comparison of the Three Profiles
| Dimension | Profile 1: It only needs to run | Profile 2: 25 tokens/s for one user | Profile 3: 10 concurrent users, each ≥25 tokens/s |
|---|---|---|---|
| Hardware form | One large-memory server plus 1–2 GPUs | Multi-node cluster with 24–32 accelerators | Super node with 64+ accelerators |
| Interconnect requirement | PCIe is enough | High-speed inter-accelerator interconnect within a node; high-speed RDMA between nodes | Large communication domain, usually within one super node |
| Output speed | Single-digit tokens/s | About 25 tokens/s | ≥25 tokens/s per stream; aggregate ≥250 tokens/s |
| Suitable scenario | Output validation, prompt debugging | Individual or small-team daily use | Stable API service for external users |
4. Cost: Self-Host, Use Cloud, or Go Hybrid
Conclusion first. Based on rough public-market estimates:
- Profile 1: one large-memory heterogeneous inference server, about RMB 150,000–300,000.
- Profile 2: about RMB 3 million–7 million.
- Profile 3: about RMB 10 million–20 million in hardware, plus ongoing electricity, bandwidth, and operations costs.
If there is no GPU procurement budget but the desired service quality is similar to Profile 3, using a cloud-hosted large-model API billed per token is usually the more cost-effective choice. The self-hosting cost analysis below does not expand on that path.
Cost Structure: Hardware Is Only One Part
- One-time costs: GPU servers, including CPUs, memory, and local storage; high-speed networking; data-center modifications; deployment and tuning labor.
- Ongoing costs: electricity, bandwidth, maintenance contracts, and operations labor.
- Hidden costs: iteration risk. Large-model hardware evolves quickly and depreciates quickly. Total cost of ownership should be evaluated over a 2–3 year horizon.
Hardware Cost Estimate
Because channel prices and power assumptions vary widely by region, GPU prices below are modeled assumptions: about RMB 100,000 per 80GB-class accelerator, RMB 50,000 per 48GB-class accelerator, and RMB 15,000 per 24GB-class accelerator. Assuming roughly 700W power per GPU-class configuration, PUE of 1.3, industrial electricity at RMB 0.8/kWh, and 7×24 operation, the approximate costs are:
| Profile | GPU count and specification | GPU procurement cost | Server and network, etc. | One-time total | Annual electricity at full load |
|---|---|---|---|---|---|
| Profile 1 | 1×48GB + 1×24GB | RMB 65,000 | RMB 35,000 | About RMB 100,000 | About RMB 4,000 |
| Profile 2 | 32×80GB | RMB 3,200,000 | RMB 1,300,000 | About RMB 4,500,000 | About RMB 175,000 |
| Profile 3 | 64×80GB | RMB 6,400,000 | RMB 3,800,000 | About RMB 10,200,000 | About RMB 350,000 |
A few notes:
- For Profile 2, whether 24 or 32 accelerators are chosen, GPU cost is roughly RMB 2.4 million–3.2 million. Larger clusters have higher marginal costs for networking, racks, and operations. This is why the table uses 32 accelerators as the planning value.
- Profile 3 has a higher share of server and network costs because super nodes demand higher-speed interconnect, power delivery, and data-center conditions.
- Electricity figures already include cooling and power losses via PUE = 1.3. Actual results should be based on data-center bills.
A notable conclusion is that the cost gap across profiles can be several times or even tens of times larger, but the per-token cost gap is not as large. Profile 1 often fails to utilize hardware fully, making unit output cost high. Profiles 2 and 3 improve hardware utilization through parallelism and batching, substantially lowering per-token cost. If budget permits, moderate overprovisioning is often more economical than barely meeting the minimum runtime requirement.
Cloud or Self-Host: Use Break-Even Months
Let total self-hosting investment be C, monthly electricity, bandwidth, and maintenance cost be M, and the equivalent monthly cloud price be P. The following script estimates the payback period:
# Unit: RMB; parameters are illustrative and should be replaced with actual quotesdef break_even_months(capex, monthly_opex, monthly_cloud):"""Calculate the months at which self-hosting becomes cheaper than a monthly cloud instance."""monthly_saving = monthly_cloud - monthly_opexif monthly_saving <= 0:return float("inf") # Cloud price is lower than self-hosted operating cost; self-hosting never breaks evenreturn capex / monthly_savingscenarios = {"Profile 1": dict(capex=100_000, monthly_opex=350, monthly_cloud=4_000),"Profile 2": dict(capex=4_500_000, monthly_opex=14_600, monthly_cloud=200_000),"Profile 3": dict(capex=10_200_000, monthly_opex=29_200, monthly_cloud=400_000),}for name, p in scenarios.items():print(f"{name}: approx. {break_even_months(**p):.1f} months to break even")# Profile 1: approx. 27.4 months# Profile 2: approx. 24.2 months# Profile 3: approx. 27.5 months
The results fall in the 24–28 month range, which aligns with common hardware depreciation and iteration cycles. This means:
- Long-term stable workload: If demand is predictable, utilization is high, and the usage period exceeds 1.5 times the break-even point, self-hosting is more economical.
- Variable workload: If the system is idle at night and relies on elastic scaling, pay-as-you-go cloud resources are more flexible.
- Rapid validation: If the project is in a POC stage, renting GPU resources on demand is often the best way to validate feasibility before committing to long-term investment.
- Hybrid mode: A self-hosted baseline plus cloud-based elastic expansion can balance cost and peak demand. Public cloud platforms may offer GPU cloud servers, elastic containers, and model-serving services as candidates for the elastic layer. Specific specifications and pricing should be confirmed in official documentation.
Two Additional Variables Beyond Self-Hosting
Exchange rates and supply: If hardware depends on imports, procurement costs may fluctuate with exchange rates and lead times. Delivery may change from “in stock” to a wait of several months. Cloud resources usually have an advantage in delivery certainty.
Alternative path: If business requirements do not mandate that data remain inside the enterprise network, compare self-hosting with calling a cloud-hosted large-model API:
Monthly self-hosting cost including depreciation ≈ one-time investment ÷ depreciation months + monthly operating costMonthly API cost = input tokens × input price + output tokens × output price
When actual monthly token volume is far below the break-even point, or when peak fluctuations are large, API billing is often more economical.
5. From “It Runs” to “It Runs Fast”: Five Key Practices
Quantization strategy: The official model introduces quantization-aware training from the SFT stage, so the combination of 4-bit weights and 8-bit activations during inference is training-aligned and has relatively controllable accuracy loss. When performing secondary quantization, evaluate quality regression rather than pursuing compression blindly.
Parallel partitioning: Shared attention components are suitable for tensor parallelism, while sparse experts are suitable for expert parallelism. In practice, the two are often combined, such as TP × EP, with partition granularity determined by node topology to minimize cross-node communication.
Communication optimization: All-to-all communication is the top performance killer in MoE inference. Keep communication inside high-speed interconnect domains, and reduce latency through computation/communication overlap and bucketed transfer. Do not expect ordinary Ethernet to handle high concurrency.
Decode acceleration: Multi-token prediction auxiliary heads, speculative decoding, and prefix caching can amortize per-token memory-access costs. Prefix caching is especially beneficial for shared system prompts and fixed few-shot examples.
Deployment shape: For high concurrency, use Prefill/Decode disaggregation. Allowing compute-intensive prefill and memory-intensive decode to scale independently makes it easier to meet SLOs than a monolithic deployment.
6. Decision Checklist and Benchmark Checklist
Decision checklist:
- Data compliance: Can the model and data leave the internal network? If not, choose local self-hosting. If yes, include API pricing in the comparison.
- Concurrency scale: Is it one user or more than 10 users? The cost gap across profiles can be several times larger. Do not pay for one profile while expecting another.
- Speed requirement: Can single-digit tokens/s be accepted? If yes, choose Profile 1. If not, at least choose Profile 2.
- Usage period: Will stable usage exceed two years with high utilization? If yes, prioritize self-hosting.
- Operations capability: Is there someone who can maintain a multi-node, multi-accelerator inference cluster? If not, prioritize fully managed services.
Benchmark checklist: After weights are released, verify the following step by step. The example below uses a benchmark script from an inference framework; the same approach applies when switching inference engines:
# 1. First test whether the model can load: try a single node or minimum cluster to confirm the weights fit# This framework usually specifies expert/GPU mapping through parameters; refer to official documentation for exact syntax# 2. Test throughput: the random dataset automatically generates variable-length inputs for load testing# Parameters: -n output tokens per request; --num-prompts total requests; --max-concurrency maximum concurrency# In the output, "Output token throughput" is aggregate output throughput; "Mean TPOT" is average time per output tokenbenchmark_serving.py random \--model /path/to/model --port 8000 \--num-prompts 100 --random-input-len 2048 --random-output-len 512 \--max-concurrency 1# 3. Increase concurrency gradually: 1 → 10 → 50; record aggregate throughput, P99 TTFT, and TPOT
At the same time, observe whether expert routing is balanced, CPU memory hit rate, and GPU utilization to locate the real bottleneck.
Summary: There is no single answer to “How much does local deployment cost?” It depends on which profile is required:
- Profile 1: about RMB 100,000, enough to run.
- Profile 2: about RMB 3 million–7 million, for single-user 25 tokens/s.
- Profile 3: about RMB 10 million–20 million one-time investment, plus ongoing operating costs, for service-level performance with 10 concurrent users and at least 25 tokens/s per user.
Every figure should be verified along the chain: weights → VRAM → communication → throughput → cost, rather than copying a post claiming that “eight GPUs are enough.” Define the SLO first, then choose hardware; complete benchmarks first, then talk about procurement.
