Private Deployment Cost Estimation for 100B-Parameter MoE Models: VRAM, KV Cache, and Total Cost of Ownership
Open-source weights are only the entry ticket to privately deploying a large MoE model. This article presents a reusable framework for budgeting a 100B-parameter Mixture-of-Experts (MoE) model, starting from three model parameters—total parameters, activated parameters per token, and maximum context length. It breaks down the four VRAM consumers (weights, KV cache, runtime, and safety headroom), explains why memory bandwidth and concurrency matter more than raw parameters during decoding, and lays out two budget tiers—one for functional pilots and one for production-grade long-context service. It also covers hidden costs such as engineering talent and recurring operations, a prioritized list of cost-reduction techniques, and a pre-launch decision checklist. The bottom line: for a 100B-scale MoE model, FP8 "runs" on a single node at roughly RMB 3 million, while production-grade service for a full 1M-token context typically requires RMB 7.3–8 million on a dual-node setup, plus ongoing yearly operating expenses.
1. Open Source Isn’t Free: Draw the Cost Boundary First
When teams evaluate “privately deploying an open-source large model,” the first instinct is usually “the weights are free, so we just need to buy servers.” The real cost structure is far more involved. Even with a permissive license and public weights, taking a hundred-billion-parameter sparse (MoE) model to the point where it can serve traffic reliably means clearing four thresholds: VRAM capacity, compute bandwidth, network interconnect, and operational headcount.
This article doesn’t discuss any specific model or product. Instead, it abstracts a reusable estimation method. As long as you can obtain three key parameters of your target model—total parameter count, activated parameters per token, and maximum context length—you can plug them into the formulas below to derive your own budget range.
2. Three Decisive Parameters on the Model Side
2.1 Total Parameters Set the VRAM Floor
Take the hypothetical reference model in this article: a MoE architecture with about 743B total parameters that activates roughly 39B parameters per generated token, with native support for a 1M-token (about one million) context.
Note the order-of-magnitude difference between these two numbers: activated parameters are only about 5% of the total. This means compute cost (cost per token) is driven by the 39B figure, but VRAM usage is driven by the 743B figure. This is the easiest trap to fall into when estimating MoE models—many people size their compute from activated parameters but forget that the weights must reside entirely in VRAM, because routing is dynamic and no single card can predict which experts the next batch of tokens will hit.
2.2 Context Length Sets the KV Cache Ceiling
A 1M context is a “capability label,” but on the VRAM ledger it’s a “cost multiplier.” The KV cache grows linearly with sequence length and cannot be shared across concurrent requests (unless they share the same prefix). A long-context model can require over 100GB of extra VRAM even for a single request.
3. The VRAM Ledger: Breaking Down Four Overheads
Available VRAM per card is not the same as VRAM available to the model. Actual usage consists of four parts:
| Overhead | Description | Order-of-magnitude reference |
|---|---|---|
| Model weights | Parameter bytes after quantization | FP8 ≈ total parameters × 1 byte |
| KV cache | 2 × layers × KV heads × head_dim × precision bytes × sequence length | Linearly related to context and concurrency |
| Framework and runtime | Drivers, compute libraries, VRAM pool, communication buffers, activations | Usually reserve 10%–20% |
| Fragmentation and safety headroom | VRAM fragmentation, long-sequence spike buffers | Recommend reserving 15% headroom |
3.1 A Reproducible Formula for KV Cache
def kv_cache_bytes(num_layers, num_kv_heads, head_dim,seq_len, dtype_bytes=1, concurrency=1):"""KV cache for a MoE model depends only on the attention layers,not on the number of experts. With GQA/MQA, num_kv_heads ismuch smaller than num_attention_heads.dtype_bytes: FP8=1, BF16=2"""per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytesreturn per_token * seq_len * concurrency# Illustrative parameters (to demonstrate order of magnitude only,# not representative of any specific model)# 60 layers, 8 KV heads, head_dim=128, FP8per_token = 2 * 60 * 8 * 128 * 1print(f"KV cache per token: {per_token/1024:.1f} KB")print(f"1M context, single request: {kv_cache_bytes(60, 8, 128, 1_000_000)/1024**3:.1f} GB")
The output is roughly 120KB per token and about 114GB for a single 1M-context request. The significance of this number is that it already exceeds the VRAM capacity of most single cards, which means long context must be sharded across multiple cards using tensor parallelism, and it nearly exhausts all headroom, leaving very little room for concurrent requests.
3.2 VRAM Comparison Across Three Quantizations
- BF16/FP16: 743B × 2 bytes ≈ 1.4TB, requiring at least double-digit numbers of high-end accelerators, usually without economic viability;
- FP8: 743B × 1 byte ≈ 743GB, currently the most balanced engineering choice;
- 4-bit weight quantization: about 372GB, with low VRAM pressure, but MoE expert layers are sensitive to low-bit quantization, and the accuracy loss must be validated against business metrics—it can’t be generalized.
4. The First Budget Tier: It Runs, But It’s Only an “Entry Ticket”
Assume high-end accelerators with 141GB-class per-card VRAM; a standard 8-card server provides about 1128GB of VRAM.
Simple subtraction: 743GB of weights + about 10% runtime and communication overhead (about 110GB) leaves about 275GB for the KV cache. Based on the illustrative values above, a single 1M-context request consumes 114GB—which means the same machine can only support 2 concurrent full-length requests, and once concurrency climbs to a dozen or more, requests must be queued or rejected.
So the positioning of this tier is functional validation and internal pilots, not production service. The procurement budget for the whole machine (including CPU, memory, system disk, and power supply) lands around RMB 3 million, and with rack modifications plus network and storage accessories, first-year total investment typically exceeds RMB 3.5 million.
The conclusion is straightforward: being able to start up ≠ being able to serve. Treating “the model runs” as the end of the project is the most common misjudgment in projects like this.
5. The Second Budget Tier: Reliably Supporting the Full 1M Context
To give long context genuine high-concurrency serving capability, there are usually two paths:
- Scale horizontally to two nodes with 16 cards: VRAM doubles to about 2256GB, the available KV cache space expands to support several times the concurrency, and multi-machine tensor parallelism also lowers single-request latency;
- Compress the KV cache vertically: through GQA/MQA architecture optimization, KV quantization, sparse attention, or tiered caching, drive down the per-token cache cost.
Production-grade solutions often combine both. Preparing for a dual-node 8-card configuration, the hardware budget is about RMB 6 million to 7 million, plus:
- High-speed interconnect between nodes (RDMA NICs and low-latency switches—tensor parallelism for long context is extremely sensitive to bandwidth);
- An all-flash parallel file system or high-performance object storage (for weight loading, checkpoints, and archiving inference results);
- Racks, power distribution, cooling, and electricity costs under heavy load;
- Supporting engineering such as monitoring and alerting, logging services, stress testing, and canary releases.
The overall budget range is roughly RMB 7.3 million to 8 million, and this is just the initial investment; subsequent annual facility, power, and operations headcount costs are recurring expenses, so it’s advisable to reserve 15%–25% of the hardware investment per year.
6. The Compute Ledger: Why Concurrency Affects Experience More Than Parameter Count
MoE’s small activated parameters make compute requirements relatively friendly, but memory bandwidth is the real bottleneck during the decode phase. Token-by-token generation is a memory-bound scenario: for each generated token, the weights of the hit experts must be moved from VRAM into the compute units. So actual throughput is determined more by VRAM bandwidth, parallelism strategy, and batching efficiency than by paper FLOPS.
Three metrics that must be measured together:
- Time to first token (TTFT): most affected by prefill-phase compute and long context, especially pronounced with 1M inputs;
- Per-request output rate (TPOT): reflects bandwidth efficiency during decoding;
- Whole-machine throughput (tokens/s) and the latency curve under high concurrency: only by seeing the inflection point where latency rises with concurrency in stress tests can you know the cluster’s true capacity level.
A practical stress-testing method: fix four input lengths of 32K / 128K / 512K / 1M, run a concurrency gradient scan at each level, and plot a three-dimensional “concurrency–P99 latency–throughput” curve. Capacity planning should be based on P99, not the average; in long-context scenarios the gap between tail latency and the mean can often be several times.
7. Overlooked Hidden Costs
- Headcount: tuning distributed inference engines, choosing parallelism strategies, and troubleshooting VRAM overflow require engineers with high-performance computing experience, and hiring cycles are often measured in months;
- Model iteration: base models keep updating, and private deployment means every upgrade requires redoing accuracy validation and capacity stress testing;
- Lack of elasticity: hardware is procured for peak load, but average utilization may be only 30%–50%, and this idle capacity is a hidden loss;
- Compliance and security: the demand that data not leave the domain is the main driver of private deployment, but you must simultaneously build access control, auditing, and data masking capabilities.
8. Cost-Reduction Checklist, Sorted by ROI
- Prefix cache reuse: the KV prefixes of system prompts, long documents, and few-shot examples can be shared across requests, with the greatest benefit in long-context scenarios;
- Speculative decoding / draft models: improve decode throughput without degrading the output distribution, with clear effects in memory-bound scenarios;
- Prefill-decode disaggregated deployment: put the prefill of long inputs and token-by-token decoding into different resource pools to avoid P99 jitter caused by mutual interference;
- Context tiering on demand: summarize or retrieve-and-trim ultra-long histories—not every request needs a 1M window;
- Layered quantization strategy: use FP8 for weights while keeping sensitive attention and normalization layers at higher precision, trading a small accuracy cost for VRAM and bandwidth;
- Hybrid architecture: route bursty traffic to the cloud using general-purpose container platforms and elastic compute during peaks, and to self-built clusters during troughs, balancing cost and elasticity.
9. Decision Checklist
Confirm each item before the project starts:
- What is the real context-length distribution of the business? What is the P99 requirement, rather than “how much is supported”?
- What are the target concurrency and acceptable P99 latency?
- Has the VRAM ledger been fully accounted for as “weights + KV cache + runtime + 15% headroom”?
- Has tiered stress testing been completed, and are the scale-out triggers defined?
- Have the facility, power, and headcount costs beyond the first year been included in the TCO?
- Have you evaluated a hybrid “self-built + cloud elasticity” approach?
One-sentence summary: For a hundred-billion-parameter MoE model, “getting it running” under FP8 quantization requires a single-node investment on the order of RMB 3 million; production-grade service that reliably delivers a full million-token context typically puts the overall budget of a dual-node solution in the RMB 7.3–8 million range, with ongoing operating expenses every year. Open source is only the entry ticket—the real cost lies in turning a model into an SLA-backed service, so account for the VRAM ledger first, then the money.
