# Local Deployment Cost and Hardware Sizing for Ultra-Large-Scale MoE Models
Local deployment of a 2.8-trillion-parameter sparse Mixture-of-Experts (MoE) model is not a matter of simply fitting weights into VRAM. This article breaks the problem down into three target tiers — "it just runs," 25 tokens/s for a single user, and 10 concurrent users each above 25 tokens/s — and works through the capacity math for each: raw 4-bit weight size with block-quantization overhead, the KV cache formula, and why 1.79% expert activation changes compute load but not storage requirements. For each tier it lists the dominant bottleneck, recommended hardware configuration, and a rough budget range in RMB, from roughly RMB 100,000 up to RMB 3,000,000 and beyond. It closes with the operating costs that a purchase quote hides (power, cooling, load time, ops), a comparison of short-term cloud validation versus buying an on-prem cluster, and a four-stage validation sequence — capacity, single-request latency, concurrency, long-run stability — to run before committing hardware.
1. The Real Problem: Loading a MoE Model Is Not Just “Fitting It in VRAM”
Local deployment of ultra-large sparse Mixture-of-Experts (MoE) models is often assumed to be a VRAM capacity question: a big enough card and you are done. The actual challenge is that storage capacity, memory bandwidth, expert routing communication, KV cache management, quantization precision, and concurrency scheduling all have to hold at the same time.
Using the model scale discussed here as a reference case — roughly 2.8 trillion total parameters, a sparse expert structure, a common configuration of 896 experts with 16 experts activated per token — three points deserve particular attention:
All expert weights must be resident on addressable storage.
The activation ratio affects compute volume, not storage volume. Each token may only touch a handful of experts during inference, but the system must still guarantee that every expert exists in addressable storage.A low activation ratio does not mean lower cost.
Activating 16 of 896 experts is roughly 1.79%. That reduces the compute pressure of a single token, but it does not turn a 1.5TB-class weight set into a few hundred GB.Different deployment goals lead to radically different hardware.
“It just produces tokens” and “one user sustains 25 tokens/s” are not the same problem. Adding a requirement for 10 concurrent streams at stable output brings KV cache, batching, communication bandwidth, and stability into play.
For that reason, cost estimation should not start from “how many cards do I need?” It should be split into three tiers:
| Target tier | Core metric | Main bottleneck | Where the budget goes |
|---|---|---|---|
| Just runs | Full model loads and emits tokens | Capacity, addressable space | Large memory, large storage |
| Usable by one user | ~25 tokens/s for a single user | Bandwidth, expert hit rate, latency | High-bandwidth memory, accelerator cards, caching strategy |
| Multi-user production | 10 concurrent streams, each ≥25 tokens/s | KV cache, communication, throughput, stability | Multi-card communication domain, scheduling, redundancy |
2. Weight Capacity Math: 4-Bit Is Not a Free Lunch
Many people reason intuitively: 2.8 trillion parameters at 4 bits is about 1400GB, so provisioning 1.5TB should be enough. That estimate only accounts for raw weights and ignores the overhead that accumulates in a real runtime.
2.1 Raw weight floor
Ignoring scale factors, non-quantized layers, metadata, and runtime overhead:
2.8 trillion params × 4 bit / 8 ≈ 1400GB
In practice, block quantization typically attaches one 8-bit scale factor alongside every 32 4-bit elements. The average bit width then becomes:
avg bit width = 4 + 8 / 32 = 4.25 bitweight footprint = params × 4.25 / 8result ≈ 1487.5GB
In other words, the weight component alone lands close to 1.5TB.
2.2 How much engineering headroom to reserve
A real deployment also has to account for:
- Tensors that are not 4-bit: some layers, normalization layers, routing layers, embedding or output layers may not be fully quantizable to 4 bits;
- Metadata and indices: expert routing tables, layer structure, load caches, version information;
- Runtime workspace: attention computation, activation caches, communication buffers;
- KV cache: grows quickly with multi-turn conversation, long context, and higher concurrency;
- Alignment and fragmentation: distributed loading, memory mapping, and page caches add further overhead.
So if the goal is “the model at least loads completely,” addressable capacity should not fall below 1.8TB. To be safe, plan for 2TB.
2.3 KV cache estimation formula
KV cache size depends strongly on concurrency, context length, layer count, attention head count, head dimension, and cache precision. A simplified formula captures the relationship:
KV cache bytes ≈concurrent requests × context length × layers × KV heads × head dim × 2 × bytes per element
The ×2 accounts for the two copies held for keys and values. KV cache quantization lowers bytes per element, but quality, compatibility, and stability need to be validated separately.
With 10 concurrent streams, long context, and higher output length, KV cache can easily grow from tens of GB to over a hundred GB. This is a major reason why the cost gap between “it runs” and “it serves concurrency” is so large.
3. Tier 1: It Just Runs — Focus on Large Memory and Heterogeneous Inference
If the only requirement is that the model loads and emits tokens, with no concern for speed, the cost driver is not the number of accelerator cards but addressable capacity.
3.1 Recommended approach
A general heterogeneous inference design works here:
- Keep the shared attention portion on accelerator cards, using high-bandwidth compute for attention and active computation;
- Store sparse expert weights in large-memory servers and load them dynamically according to routing results;
- The accelerator does not need to hold every expert — only the experts activated by the current request, plus attention compute and temporary buffers;
- System-level tuning then shifts to memory bandwidth, NUMA binding, expert cache hit rate, and storage read throughput.
3.2 Hardware configuration
| Component | Recommendation |
|---|---|
| CPU | Multi-socket server; core count is not the only thing that matters — memory channel count and memory bandwidth matter more |
| Memory | At least 1.5TB; 2TB to be safe |
| Accelerator | At least one high-VRAM accelerator card to host attention, active experts, and runtime buffers |
| Storage | High-speed SSD, capable of high-throughput large-file reads |
| Network | High-speed networking is not critical for a single node, but multi-node loading requires low-latency links |
3.3 Cost range
If the requirement is only that the model runs, without chasing speed:
- Reusing existing accelerator cards: cost concentrates on the server, memory, and storage, roughly in the RMB 100,000–250,000 range;
- Adding one high-VRAM accelerator card: may rise to RMB 200,000–400,000;
- Memory beyond 2TB, a higher-spec CPU platform, or stronger storage performance: cost continues upward.
The key point for this tier: do not blindly pursue multi-card parallelism just to get the model running. If the end result is only single-digit tokens/s, spending heavily on accelerator cards is unlikely to pay off.
4. Tier 2: 25 tokens/s for One User — Focus on Hot Expert Hits and Memory Bandwidth
25 tokens/s for a single user looks like a speed upgrade, but it actually moves the system into a low-latency inference regime. The bottleneck shifts from “does it fit?” to “how much data must be moved for each token generated, and from where?”
4.1 Why 25 tokens/s is not easy
Take the sparse expert structure: if the experts activated per token account for roughly 1.79% of all experts, the activated weight per token can still reach the tens-of-GB level. Assume each generated token requires reading about 25GB of weights and the target is 25 tokens/s, then the weight read bandwidth requirement is around:
25GB × 25 = 625GB/s
That is an idealized estimate only — it excludes attention, KV cache, communication overhead, fragmentation, and scheduling loss.
Which means:
- If expert weights are served mainly from ordinary system memory, memory bandwidth becomes a severe constraint;
- If hot experts are placed in accelerator VRAM, both VRAM capacity and VRAM bandwidth must be sufficient;
- If experts are cold-loaded on every routing decision, latency becomes highly unstable;
- If a hybrid offload approach is used, expert caching and hotness tiering are mandatory.
4.2 Two typical routes
Route A: Large memory + a few accelerator cards + expert cache
Suitable for cost-sensitive teams.
Configuration approach:
- Around 2TB of system memory;
- 1–4 high-VRAM accelerator cards;
- Frequently accessed experts cached in accelerator VRAM or high-bandwidth memory;
- Cold experts kept in system memory and loaded on demand;
- Scheduling policies that reduce repeated loading.
Pros:
- Relatively controllable cost;
- Can reuse general-purpose servers;
- Suitable for validation and small-scale pilots.
Cons:
- Speed depends heavily on cache hit rate;
- Latency can jitter under long context or complex routing;
- Places high demands on memory bandwidth and software scheduling.
Rough cost range: about RMB 300,000–900,000, depending on the number of accelerator cards, memory specification, and interconnect quality.
Route B: Keep as much as possible resident in VRAM / high-bandwidth memory
Suitable for scenarios that need stable latency.
Configuration approach:
- Place as much expert weight as possible in accelerator VRAM;
- Combine multiple accelerator cards into a high-speed communication domain;
- Use expert parallelism, tensor parallelism, and similar strategies;
- Minimize data movement across nodes and across memory domains.
Given 1.5TB of weights plus KV cache and runtime overhead, keeping the large majority of weights resident on accelerators may require close to 2TB of aggregate VRAM. Estimating at the 80GB-per-card class, that means at least twenty-odd accelerator cards to cover the bulk of the weights.
Pros:
- More stable latency;
- Friendlier to high concurrency;
- Easier to hit sustained throughput targets.
Cons:
- Cost rises sharply;
- High demands on high-speed interconnect, data-center power, cooling, and the scheduling stack.
Rough cost range: likely into the RMB million range; if multi-node communication domains, redundancy design, and long-context serving are involved, cost can reach into the low single-digit millions of RMB.
4.3 Key optimizations for this tier
- Track expert hotness and pin high-frequency experts to high-speed media;
- Reserve higher-bandwidth resources for the attention portion and shared computation;
- Apply quantization, paging, or compression strategies to the KV cache;
- Batch the routing layer to reduce jitter from expert switching;
- Asynchronously prefetch cold experts so the critical path never waits on them;
- Monitor hit rates across the three-level VRAM / memory / storage cache hierarchy.
5. Tier 3: 10 Concurrent Streams, Each ≥25 tokens/s — Focus on Throughput and Stability
Raising the target to 10 concurrent streams, each requiring 25 tokens/s or more, puts total throughput at roughly 250 tokens/s. At that point the problem is no longer single-request latency but throughput, stability, KV cache capacity, and communication efficiency.
5.1 New pressures from concurrency
10 concurrent streams mean:
- KV cache footprint grows close to 10×;
- Request scheduling, batching, preemption, and recovery logic become more complex;
- Expert routing communication frequency increases;
- Single-card or single-node hotspots become more likely;
- VRAM and memory capacity pressure rises noticeably;
- Any single bandwidth bottleneck gets amplified by concurrency.
In a sparse expert model, every request triggers routing. Different requests may hit the same batch of hot experts at the same time, or scatter across a large number of cold experts. With a poor caching strategy this produces the pattern of “more requests, more cold loads, less stable speed.”
5.2 Hardware approach
The recommendation is to move from single-node heterogeneous inference to a distributed inference cluster:
| Layer | Recommendation |
|---|---|
| Compute | Multiple high-bandwidth accelerator cards forming a high-speed interconnected communication domain |
| Memory | Large-capacity memory nodes handling cold experts, KV cache, and intermediate results |
| Storage | High-speed distributed storage supporting fast weight loading and cache warm-up |
| Network | Low-latency, high-bandwidth interconnect so cross-domain communication is not the bottleneck |
| Service | Continuous batching, priority scheduling, request isolation, and failure recovery |
For stable 10-stream output, “model loading capability” and “online serving capability” generally need to be designed separately. Training or offline batch jobs can tolerate slow loading, but online serving must keep the hot path low-latency.
5.3 Cost range
The 10-concurrent scenario is usually not solved by adding one more server; it moves into cluster deployment:
- A mid-sized heterogeneous cluster: roughly RMB 800,000–1,800,000;
- Multi-node high-speed interconnect, redundancy design, long-duration stable operation: potentially RMB 2,000,000–3,000,000 and above;
- Higher concurrency, longer context, or stronger availability requirements: cost continues to rise.
6. Cost Is Not Just the Purchase Price — Operating Costs Count Too
Beyond hardware acquisition, local deployment carries ongoing operating costs.
6.1 Power and cooling
Ultra-large model inference draws sustained high power. Accelerator cards, CPUs, memory, network, and storage all generate heat, and the facility needs stable power delivery and cooling. In an ordinary office environment, noise, temperature, and electrical capacity can all become problems.
6.2 Storage and load time
Weights at the 1.5TB scale cannot be loaded instantly. You need to evaluate:
- Cold-start loading time;
- Storage throughput when multiple instances load concurrently;
- Cache warm-up strategy;
- The cost of switching model versions.
6.3 Operations cost
This includes monitoring, failure recovery, quantized-version management, routing policy tuning, memory leak investigation, KV cache eviction, log analysis, and security isolation. For production use, this portion of the cost cannot be ignored.
6.4 Cloud validation versus building on-prem
For short-term validation, you can start with general-purpose cloud capabilities — for example the GPU instances, object storage, log service, and monitoring and alerting available from Baidu AI Cloud — and run small-scale load tests to observe model loading, single-request speed, concurrency bottlenecks, and resource utilization before deciding whether to buy a long-term on-prem cluster. This lowers upfront capital expenditure and lets you calibrate the cost model with real measurements.
7. Recommended Validation Steps
Before committing to a purchase, validate in this order.
Step 1: Capacity validation
Goal: confirm the model can load completely.
Metrics to watch:
- Weight file size;
- Peak memory after loading;
- Peak accelerator VRAM;
- Footprint of non-quantized layers;
- Whether memory overflow occurs;
- Load time.
Step 2: Single-request latency validation
Goal: confirm the single-user experience meets expectations.
Metrics to watch:
- Time to first token;
- Subsequent token generation speed;
- Stability over long generations;
- Routing jitter;
- Expert cache hit rate;
- Memory bandwidth utilization.
Step 3: Concurrency load validation
Goal: confirm speed holds at 10 concurrent streams.
Metrics to watch:
- Number of concurrent requests;
- Per-request output speed;
- Total throughput;
- KV cache growth;
- Peak VRAM / memory;
- Request queueing time;
- Tail latency;
- Error rate.
Step 4: Long-run stability validation
Goal: confirm the system can run for extended periods.
Metrics to watch:
- Continuous operation for 8 hours, 24 hours, or longer;
- Whether memory grows continuously;
- Whether VRAM fragments badly;
- Whether cache effectiveness degrades;
- Whether temperature and power draw remain stable;
- Whether the service recovers automatically.
8. Cost Conclusion: Three Tiers, Three Budgets
In one sentence: the cost of deploying an ultra-large MoE model locally is not determined by a single piece of hardware, but by which target tier you are aiming for.
Just runs
The core is large memory and high-speed storage, with the lowest cost. Suited to internal research, feasibility validation, and offline batch work. A rough budget lands in the RMB 100,000–400,000 range, depending on whether existing accelerator cards are reused.25 tokens/s for one user
The core is hot expert caching, memory bandwidth, and accelerator VRAM. Cost rises noticeably. With hybrid offloading, roughly RMB 300,000–900,000; keeping more weights resident on accelerators can push into the RMB million range.10 concurrent streams, each ≥25 tokens/s
The core is clustering, high-speed interconnect, KV cache, and the scheduling system. This is the most expensive tier, starting around RMB 800,000 and potentially reaching RMB 2,000,000–3,000,000 or more in a complex production environment.
The final recommendation: decide first whether the business goal is “just runs,” “usable,” or “production-ready,” then do the capacity math and hardware selection. Do not aim for a fully loaded high-end configuration from day one, and do not try to force a trillion-parameter model onto a single card. The path that actually works is to validate capacity first, then single-request latency, then gradually ramp to the target concurrency — and let measured data determine the final hardware scale and budget.
