Local Deployment Cost Assessment for Trillion-Parameter MoE Models
This article breaks down the real cost of running a trillion-parameter Mixture-of-Experts (MoE) model on your own hardware. It explains why the storage footprint of the full expert set, not the small per-token activation count, is what determines your budget; how to estimate weight storage from parameter count and quantization format; and how costs diverge across three deployment tiers — bare minimum runnable, single-user at roughly 25 TPS, and 10 concurrent users all staying above 25 TPS — with the corresponding hardware shapes and budget orders of magnitude. It also covers the capacity/bandwidth/scheduling bottleneck triangle, and a staged measurement and testing approach to run before committing to hardware purchases.
1. Framing the problem correctly
What “local deployment” of a trillion-parameter MoE really means
Large-scale Mixture-of-Experts (MoE) models have become a major technical route in LLM development. Compared with conventional dense models, MoE trades off theoretical compute against model capacity by keeping the total parameter count very large while activating only a small number of parameters per token. When deploying locally, however, developers often fall into an obvious trap: they see that only a few experts fire per token and badly underestimate the true deployment cost.
The core tension in MoE deployment is not the parameter count of a single activation. It is that the complete parameter set must be stored, addressed, and scheduled. Inference may only compute a handful of experts, but before inference can start, every expert’s weights must fit somewhere you can address. During inference, a routing module decides which experts participate, and the system then dispatches those computations to the corresponding memory tier or compute unit. For a trillion-parameter-scale model, this creates three distinct cost pressures:
- Storage capacity pressure — every expert’s weights need a place to live.
- Data movement pressure — the weights of the activated experts must be fetched from system memory, VRAM, or storage.
- Scheduling and communication pressure — across multiple cards, multiple nodes, and multiple experts, frequent synchronization and routing are required.
So the right question is not “will it run?” but rather three separate questions with sharply different hardware, architecture, and budget implications:
- The model must be able to emit tokens, with no requirement on speed.
- A single user should see a stable output speed of roughly 25 TPS.
- Ten concurrent requests should be supported while each individual user still sees more than 25 TPS.
2. Capacity math: how much storage the weights minimally require
For any very large model, the first calculation is not compute but capacity. A general formula works well as a starting estimate:
weight footprint ≈ parameter count × average bytes per parameter + quantization metadata + non-low-bit parameters + runtime workspace + KV cache
Take an MoE model with roughly 2.8 trillion total parameters, about 896 experts, and 16 activated experts per token. With 4-bit weight quantization, the raw data floor is approximately:
2.8 trillion parameters × 4 bits ÷ 8 ≈ 1400 GB
Engineering reality does not stop at raw data. Low-bit quantization usually carries scale factors, calibration information, or block-level metadata. Using the common pattern of one 8-bit scale value per 32 four-bit elements, the effective average rises to about 4.25 bits per parameter:
2.8 trillion × 4.25 ÷ 8 ≈ 1487.5 GB
That figure still excludes:
- A small number of parameters that cannot be low-bit quantized, such as normalization layers, embedding layers, the output layer, and certain special tensors;
- Inference framework runtime overhead;
- VRAM or memory fragmentation;
- KV cache;
- Temporary activation data for concurrent requests;
- Multi-process and multi-card communication buffers.
In practice, budget 30% to 60% headroom. For a model of this scale, usable storage or memory capacity under 4-bit weights should generally not fall below 1.8 TB, and a comfortable configuration lands closer to 2.2–2.5 TB. With 8-bit weights, capacity requirements roughly double; with 16-bit or mixed precision, cost rises substantially further.
3. Cost tiers for the three deployment goals
3.1 Tier one: it only needs to run, speed is irrelevant
If the only goal is to load the model completely and produce tokens, the cheapest option is usually not “keep all weights resident in GPU VRAM across several cards” but rather a large-memory host plus a small number of accelerators plus tiered offloading.
The logic of this route:
- Put the compute-intensive, memory-access-heavy shared components on the accelerator cards;
- Put the expert weights — which occupy most of the capacity but activate only a few at a time — in server memory;
- Have a scheduler fetch expert parameters on demand to complete the forward pass.
A typical configuration for this route looks like:
- Very large server memory, usually multi-terabyte;
- Fast solid-state storage for weight loading and caching;
- One to two accelerators to host the attention modules, routing modules, shared compute paths, and some of the hottest experts;
- A high-core-count CPU to handle data preprocessing, expert dispatching, and dequantization.
In such a setup the model may run, but output speed could be only single-digit tokens per second. Its value is validating weight integrity, inference-pipeline correctness, and basic functional availability — not delivering a production-grade experience.
On budget magnitude, this kind of “just runnable” setup typically requires hardware investment on the order of hundreds of thousands of RMB (roughly tens of thousands of USD). The exact figure depends on memory capacity, accelerator model, storage capacity, and chassis form factor. Used equipment or staged expansion can lower the price, but stability and maintenance costs go up.
3.2 Tier two: single user, roughly 25 TPS output
Around 25 TPS for a single user is a typical threshold for interactive experience. Reaching it requires more than the ability to read data: the latency between data movement, computation, and scheduling has to be low enough.
Server memory alone may not be enough here. Each generated token can trigger expert reads, dequantization, routing computation, and cache writes. If most expert weights sit in ordinary memory while accelerator VRAM is insufficient, the system spends its time waiting on data movement. CPU memory bandwidth, the number of memory channels, processor cache, NUMA topology, and storage read throughput all become critical bottlenecks.
There are three main ways to raise single-user speed:
- Add accelerators or increase per-card VRAM so that hot layers, shared layers, and high-frequency experts stay resident in VRAM;
- Use higher-bandwidth server memory and optimize memory channel layout;
- Reduce cold starts and repeated transfers through expert caching, routing prediction, and inter-layer pipelining.
Configurations in this class typically include:
- Multiple accelerators forming an inference node;
- A larger VRAM pool to hold attention modules, part of the expert set, KV cache, and batching buffers;
- Large-capacity system memory acting as a second-tier cache;
- High-speed interconnects to reduce inter-card synchronization overhead;
- An optimized inference engine with expert prefetching, layered loading, and dynamic scheduling.
On budget magnitude, this tier is significantly more expensive than the bare-runnable setup and often requires hardware investment approaching one million RMB (roughly in the low hundreds of thousands of USD) or more. Demanding stable production behavior, long context, and low jitter pushes cost higher still.
3.3 Tier three: 10 concurrent users, each still above 25 TPS
At 10 concurrent requests, the question changes from “can this machine run it” to “can the system serve multiple requests simultaneously without them dragging each other down.”
MoE inference amplifies several pressures under concurrency:
- KV cache grows roughly linearly with concurrency and context length;
- Multiple requests may hit different experts at the same time, spreading routing thin;
- Data synchronization and result gathering across cards and nodes happen more often;
- Batch scheduling has to balance latency against throughput;
- Memory bandwidth and interconnect bandwidth become the primary bottlenecks.
A single large-memory machine can still function here, but it struggles to meet the performance target reliably. The more common approach is to scale the system into a distributed inference cluster with multiple nodes and multiple cards. Deployment architectures typically involve:
- Expert parallelism — distributing different experts across compute nodes;
- Tensor parallelism — splitting a single matrix computation across multiple cards;
- Pipeline parallelism — partitioning model layers across nodes to form a pipeline;
- Request scheduling — dynamically assigning work based on expert access heat, cache hit rates, and node load;
- High-bandwidth networking — ensuring routing decisions, activation values, and intermediate results move quickly.
Budgets for systems of this kind are usually far above single-machine setups. Requiring strong stability, low tail latency, and scalability often means investment on the order of several million RMB (several hundred thousand USD). Using elastic cloud compute converts the upfront capital expense into on-demand operating expense, but for sustained high concurrency the total cost still needs careful calculation.
4. Cost is driven by a triangle, not a single component
Developers often ask: is it the GPU that is expensive, the memory, or the network? For trillion-parameter MoE models, the usual answer is that the bottleneck depends on the target.
- For “it only needs to run,” the bottleneck is capacity.
- For single-user speed, the bottleneck is data movement and VRAM capacity.
- For multi-concurrent throughput, the bottleneck is scheduling, caching, and network interconnect.
Concretely, these metrics matter:
- Accelerator VRAM capacity — how much model state can stay resident;
- Server memory capacity — whether the entire expert set fits at all;
- Memory bandwidth — affects expert reads, dequantization, and cache updates;
- Inter-card interconnect bandwidth — affects multi-card tensor parallelism and expert routing;
- Inter-node network latency — affects distributed expert dispatching;
- Storage throughput — affects weight loading, model switching, and failure recovery;
- Inference engine scheduling capability — whether the hardware can actually be kept busy.
A key characteristic of MoE models: compute is not necessarily the bottleneck — communication and memory access usually hit the wall first. Every token triggers routing, the router decides which experts to activate, and the system then moves those experts’ parameters or results to the right place. If expert placement is poor, cache hit rates are low, or network latency is high, the system can end up “fast at computing but starved of data” even with ample accelerator FLOPS.
5. Engineering recommendations: build a verifiable test harness before the weights ship
For teams planning a local deployment, the recommendation is to establish a staged testing methodology before committing to hardware procurement. With a limited budget, this avoids blind capacity expansion.
5.1 Measure actual weight footprint first
Do not trust theoretical parameter counts alone. Measure the weight files after loading, memory usage after dequantization, runtime VRAM usage, and cache overhead. Record at minimum:
- Raw weight file size;
- Memory usage after loading;
- Per-card VRAM usage;
- Resident memory of the inference process;
- Memory growth at peak;
- KV cache behavior at different context lengths.
5.2 Run layered performance tests
Do not start by testing the full model. Begin with a single layer or a small subset of layers, then scale up to the complete model. Watch:
- Time to first token;
- Average output speed;
- Stability during long-form generation;
- Latency jitter under concurrency;
- Expert hit distribution;
- Memory bandwidth utilization;
- Accelerator utilization;
- Time spent waiting on communication.
5.3 Optimize the hot data first
In an MoE system, experts are not accessed equally often. Sample activation frequencies per expert, then place high-frequency experts on faster tiers closer to the compute units, and lower-frequency experts on higher-capacity but slower storage.
Common optimizations include:
- Keeping hot experts resident in VRAM;
- Caching cold experts in system memory;
- Predicting historical routing results;
- Session-level prefix caching;
- Merging batched requests;
- Dynamic batching;
- KV cache compression or paged management.
5.4 Do not ignore operational cost
Local deployment is not just a hardware purchase. In production you also deal with:
- Driver and runtime version compatibility;
- Inference engine updates;
- Model weight upgrades;
- Failed-node replacement;
- Power and cooling;
- Monitoring and alerting;
- Log auditing;
- Capacity planning.
For enterprise users, operational cost is routinely underestimated. If the team lacks experience operating large-scale inference systems, start with a small-scale validation and expand from there.
6. Budget guidance: buy tier by tier instead of over-investing up front
If a budget framework is required, the following structure works:
- Feasibility validation only — prioritize a large-memory server, a few accelerators, and fast storage; the goal is completed loading, working inference, and correctness verification;
- Usable for a single user — add accelerators or VRAM, optimize memory bandwidth and scheduling policy; the goal is an interactive experience;
- Multi-user production — move to multi-node distributed inference, investing mainly in high-speed networking, the scheduling system, the monitoring platform, and redundancy;
- Elastic hybrid — for workloads with pronounced peaks and troughs, combine a local baseline cluster with elastic cloud capacity to reduce long-term idle cost.
Note that these figures are order-of-magnitude references, not quotes. Final cost depends on model architecture, context length, concurrency requirements, quantization scheme, hardware generation, datacenter conditions, and operational maturity.
7. Conclusion: capacity first, bandwidth decides, scheduling sets the ceiling
For very large MoE models, local deployment is not a task you finish by buying a few accelerators. It is closer to systems engineering, requiring simultaneous attention to weight storage, memory bandwidth, compute acceleration, routing and scheduling, KV cache, and concurrent serving.
If the goal is merely “it runs,” a large-memory server with tiered offloading gets you there at a manageable cost. If the goal is 25 TPS for a single user, you need more efficient VRAM utilization, higher memory bandwidth, and expert caching. If the goal is 10 concurrent users with low latency, you need distributed inference, interconnect bandwidth, and request scheduling.
Before the weights and technical report are fully public, the safest course is not to buy hardware on speculation, but to estimate capacity with the general formulas above, validate bottlenecks with small-scale tests, and then build out in phases according to business goals. That is how you control cost while leaving enough room to optimize performance after the model goes live.
