Cost and VRAM Estimation for Local Deployment of Ultra-Large Sparse Models

I. Bottom Line Up Front: It’s Not Just About “Buying a Few GPUs”

If you assume that locally deploying a trillion-parameter sparse MoE (Mixture-of-Experts) model simply requires assembling a few consumer-grade GPUs with large VRAM, the first misconception to correct is this: the primary bottleneck for these models is not compute power, but VRAM capacity and memory bandwidth.

Consider a MoE model with approximately 2.8 trillion total parameters and around 104 billion activated parameters per token. Although only a subset of experts is activated during a single forward pass, all weights must reside in addressable VRAM. Otherwise, reloading experts on the fly every time a route requests a non-resident expert will cause throughput to collapse entirely.

The precision design is even more critical. This model natively utilizes a 4-bit weight format (such as MXFP4, the official training and release precision) rather than post-quantizing FP16 down to 4-bit. This implies:

• The weight footprint is already at its “designed precision,” meaning there is no secondary quantization loss.
• Conversely, you cannot expect to compress it further into lower bit-widths to save VRAM without compromising model accuracy and stability.

Let’s do the basic math:

Model weight VRAM ≈ 2.8e12 × 4bit ÷ 8
                 ≈ 1.4 TB

Adding activation values, runtime overhead, and the KV Cache / state space cache, a single user effectively requires around 1.5TB of GPU VRAM.

This number dictates everything: this is no longer a workstation-grade task, but a server-tier project.

---

II. Where Does the VRAM Go: A Reusable Estimation Model

Many developers only calculate for the weights, only to find themselves short on VRAM after deployment. Below is a more comprehensive breakdown framework.

2.1 Weight VRAM (Static, Must Reside in Memory)

Item	Estimation Method	Magnitude
Main Weights	Total Parameters × Bits per Parameter ÷ 8	~1.4 TB
Quantization Scaling Factors/Metadata	Estimated based on grouped quantization overhead	Tens of GB
Embedding & Output Layers	Typically a small proportion but non-negligible	Several GB to Tens of GB

2.2 Runtime VRAM (Dynamic, Varies with Requests)

• Activations: Strongly correlated with batch size and sequence length.
• KV Cache: Proportional to concurrency × context length × number of layers × head dimension; grows extremely fast in long-context scenarios.
• Communication Buffers: Multi-GPU parallelism requires reserved VRAM for collective communications like AllReduce / AllGather.
• Framework Overhead: Inference engine graph capture, memory pools, temporary tensors, etc.

2.3 A Practical VRAM Budget Formula

Total VRAM ≈ Weights + KV Cache + Activations + Communication Buffers + Framework Reserves

Where KV Cache can be approximated as:

KV Cache ≈ 2 × Layers × Sequence Length × Hidden Dimension × Concurrency × Bytes per Element

With long context lengths and high concurrency, the KV Cache can expand from tens of GB to hundreds of GB. This is why “fitting the weights” does not equate to “running stable multi-concurrent services.”

---

III. Realistic Projections for Three Configuration Tiers

3.1 Tier 1: As Long as It Runs, Ignore the Speed

From a pure capacity perspective, historically, large VRAM cards could “squeeze it in,” such as roughly 19 x 80GB GPUs or 11 x 141GB GPUs. However, the problems are:

• These previous-generation GPUs cannot natively execute 4-bit weight formats and require runtime de-quantization.
• De-quantization introduces additional compute and VRAM overhead; fitting the model doesn’t mean it will run well.
• Actual throughput might be so low that it becomes unusable for interactive applications.

A more realistic minimum configuration is a single-node 8-GPU setup using next-generation large-VRAM GPUs (e.g., architectures that support native low-bit execution). This is not a biased opinion; official deployment guides for mainstream inference frameworks typically recommend 8-GPU parallelism as the starting point.

Conclusion: Merely “being able to run” already makes it a server-tier project, not a standard workstation.

3.2 Tier 2: Achieving 25 Tokens/s for a Single User

The hardware starting point remains at the 8-GPU node level. The reason is straightforward:

The primary bottleneck is not compute, but the fact that the 1.4TB of weights must be accommodated first. You cannot simply scale the server down to two or three GPUs just because only one person is using it.

As for whether an 8-GPU setup can stably achieve 25 Tokens/s, you cannot infer this solely based on VRAM capacity. Single-stream LLM inference typically operates at the “tens of tokens per second” magnitude. 25 Tokens/s is a reasonable target, but you must stress-test using your actual context length:

• Short contexts (e.g., 4K) might easily meet the target.
• Long contexts (e.g., 128K+) will cause KV Cache bloat and heavier attention computation, which may noticeably degrade throughput.

You must stress-test before going live; do not rely on guesswork.

3.3 Tier 3: 10 Concurrent Users at 25 Tokens/s Each

This requires stably providing:

10 × 25 = 250 Tokens/s aggregate output

Interestingly, the VRAM configuration may not necessarily need to increase beyond Tier 2. Under reasonable scheduling, an 8-GPU node can handle an estimated 40 concurrent requests. If the context isn’t in the millions, the theoretical number of resident requests could be even higher.

Therefore, 10 concurrent users alone won’t force you onto a second server. The real challenges are:

• Whether you can guarantee 25 Tokens/s for each stream when 10 are generating simultaneously.
• This is a throughput and scheduling performance metric that depends on batching strategies, KV Cache management, expert routing load balancing, etc.
• Again, this must be tested empirically, not just deduced from VRAM capacity.

---

IV. Parallelism Strategies: Why Start with 8 GPUs?

Deploying trillion-scale MoE models typically requires a combination of multiple parallelism methods:

Parallelism Method	Purpose	Typical Scenario
Tensor Parallelism	Splits single-layer weights	Single layer too large for one GPU
Pipeline Parallelism	Splits by layers	Many layers, scaling across nodes
Expert Parallelism	Places different experts on different GPUs	Core parallelism method for MoE
Data Parallelism	Replicates model to handle different requests	Boosts throughput

An 8-GPU node can typically execute a combination of Tensor Parallelism + Expert Parallelism within a single machine utilizing high-bandwidth interconnects, avoiding cross-node communication bottlenecks. Once you scale across nodes, communication overheads increase significantly, making throughput and latency guarantees much harder to maintain.

---

V. Cost Projection: Why Cloud Pricing is the Only Real Metric

These GPUs belong to the data-center tier and are difficult to procure through standard retail channels, so cost estimations must reference cloud platforms.

Using a cloud provider’s GPU instance as an example, assume the 12-month reserved price for a specific data-center GPU is $4.76/GPU/hour:

8-GPU Node Hourly Cost = 4.76 × 8 = $38.08/hour
Daily Cost ≈ 38.08 × 24 ≈ $913.92
Monthly Cost ≈ 913.92 × 30 ≈ $27,417.60
Annual Cost ≈ 27,417.60 × 12 ≈ $329,011

Note that this is strictly bare compute cost and does not include:

• Storage (weight loading, checkpoints);
• Network bandwidth;
• Operations and monitoring;
• Disaster recovery and high-availability redundancy.

If amortized based on “10 concurrent users at 25 Tokens/s each”:

Annual Cost / (10 Concurrency × 25 Tokens/s × 3600 × 24 × 365)
≈ 329,011 / (10 × 25 × 31,536,000)
≈ 329,011 / 7,884,000,000
≈ $0.0000417 / Token

This magnitude indicates that local/private deployment is suitable for high-frequency, stable, and data-sensitive workloads. For low-frequency use cases, utilizing APIs is far more cost-effective.

---

VI. Deployment Checklist

1. Calculate VRAM Before Compute: Weights + KV Cache + Activations + Buffers are all indispensable.
2. Verify Native Low-Bit Support: Runtime de-quantization will consume performance and VRAM.
3. 8-GPU Node as the Starting Point: Drop the illusion of handling trillion-scale MoE with two or three GPUs.
4. Stress-Test Across Context Lengths: Meeting targets on short contexts does not guarantee success on long contexts.
5. Evaluate Concurrency via Throughput and Scheduling: Sufficient VRAM doesn’t guarantee target Token/s per stream.
6. Amortize Costs Annually for Decision Making: Compare against API pricing to determine if self-hosting is worthwhile.
7. Reserve Redundancy: Failures, scaling, and peak loads will all eat into your budget.

---

VII. Conclusion

The local deployment of trillion-scale sparse MoE models is fundamentally a comprehensive challenge of VRAM capacity + memory bandwidth + parallel communication + scheduling efficiency. The 1.4TB weight threshold dictates that this is inherently a server-tier project. Meanwhile, 25 Tokens/s and 10 concurrent users are engineering metrics that must be validated through stress testing under realistic context lengths.

Clarify the VRAM budget before discussing hardware procurement or cloud migration; otherwise, you risk discovering midway through the project that the model “fits in memory but cannot run.”