Deploying a Trillion-Parameter MoE Model Locally: VRAM, Concurrency, and Cost Estimates

    This article walks through how to estimate the hardware requirements for running a 2.8-trillion-parameter MoE model on-premises. It covers VRAM budgeting for weights and KV cache, hardware configurations for three service tiers (bare-minimum inference, single-user at 25 tokens/s, and 10-way concurrency), a stress-testing methodology, annualized cost comparisons against API pricing, and a decision framework for deciding whether self-hosting makes sense. All figures are formula-based estimates, not benchmark results.

    Bottom Line First

    Running a 2.8-trillion-parameter MoE model locally requires, at minimum, one server node with eight high-VRAM data center GPUs. Weights dominate VRAM usage at roughly 1.44 TB, and this portion cannot be reduced regardless of whether your goal is “just getting it running” or “ten concurrent sessions.” The real differentiators are context length, concurrency level, and measured performance — all of which must be validated with realistic workloads before the project kicks off.

    All numbers in this article are derived from formulas, not actual measurements. The assumed model architecture parameters are for demonstration only; always refer to the model’s official configuration file.

    1. How to Calculate VRAM

    The formula for weight VRAM is:

    Weight VRAM = parameter count × bits per parameter ÷ 8 × (1 + format overhead)

    Where:

    • Parameter count: 2.8 trillion (2.8 × 10^12) — the model’s total parameters, not active parameters;
    • Bits per parameter: 4 bits, referring to the official 4-bit weight format released with the model;
    • Format overhead: the extra footprint from quantization scales and other metadata. This depends on the specific format and is typically a few percentage points; we use 3% here.

    Plugging in the numbers: 2.8 × 10^12 × 4 ÷ 8 × 1.03 ≈ 1.44 × 10^12 bytes, or roughly 1.44 TB.

    An MoE model activates only about 104 billion parameters per token, so per-step compute is determined by the active portion. However, since routing at every layer can select any expert, all expert weights must remain in VRAM to keep latency low.

    The KV cache formula is:

    KV cache = 2 × number of layers × KV heads × head dimension × bytes per element × context length × concurrency

    Assume a set of architecture parameters: 60 layers, 8 KV heads, head dimension of 128, and FP16 storage (2 bytes). The KV cache per token is approximately 2 × 60 × 8 × 128 × 2 ≈ 0.25 MB.

    • 8K context, 10 concurrent sessions: roughly 0.25 MB × 8192 × 10 ≈ 20 GB;
    • 32K context, 10 concurrent sessions: roughly 0.25 MB × 32768 × 10 ≈ 80 GB.

    The framework runtime, activations, and communication buffers typically require reserving several tens of GB more.

    2. What Fits on a Single Node

    Eight 192 GB GPUs provide 1,536 GB (1.536 TB) in total. After subtracting 1.44 TB for weights, only about 96 GB remains. If you reserve 40–80 GB for the runtime, the leftover space supports only a modest KV cache.

    This means:

    • Short context, single user: it runs, but with very thin headroom;
    • 8K context, ten concurrent sessions: KV cache is about 20 GB, which fits within the headroom, but real-world testing is required to confirm;
    • 32K context, ten concurrent sessions: KV cache is about 80 GB, essentially exhausting the headroom. You would need to reduce concurrency, compress the KV cache, or switch to GPUs with more VRAM.

    When do you need a second node? When “target concurrency × target context × per-unit KV cache footprint” plus the runtime reservation exceeds the node’s remaining VRAM, you need a second node or a higher-VRAM configuration.

    GPU selection also matters. Older-generation GPUs may not natively support 4-bit floating-point formats and must dequantize before computing — the model fits, but compute efficiency suffers. Verify GPU and inference framework support before choosing hardware.

    3. Hardware Assessment Across Three Tiers

    Tier 1: It Runs, Speed Doesn’t Matter

    If the only requirement is that the model runs, there are two paths.

    Path one: keep all weights in GPU VRAM — the eight-GPU node described above. This delivers the lowest latency and suits online serving.

    Path two: offload some experts to CPU memory or NVMe storage. This reduces the GPU count, but every miss forces a read from slow storage, significantly degrading latency and throughput. This approach only works for latency-insensitive tasks like offline batch processing — not for online services.

    Tier 2: Single User at 25 Tokens/s

    For a single-user scenario, the hardware baseline is the same as Tier 1. Fewer GPUs are not viable because the weights simply won’t fit.

    Whether 25 tokens/s is achievable has not been benchmarked here; treat it as a target to validate. A rough bandwidth sanity check: each token requires reading about 104 billion × 0.5 bytes ≈ 52 GB of active weights. Across eight GPUs in parallel, the theoretical ceiling is far above 25 tokens/s, so the actual bottleneck is more likely to be inter-GPU communication, scheduling overhead, and KV cache reads. This is only a coarse estimate — load testing is the final arbiter.

    Three factors most affect measured speed:

    • Context length: as the KV cache grows, more data must be read per step, slowing decoding;
    • Inter-GPU communication: insufficient interconnect bandwidth across eight GPUs makes communication the bottleneck;
    • Inference framework: kernel optimization varies significantly across frameworks and versions.

    Tier 3: Ten Concurrent Sessions at 25 Tokens/s Each

    Ten concurrent sessions at 25 tokens/s each means aggregate throughput of at least 250 tokens/s. Two distinct questions must be separated:

    • Capacity: can VRAM hold the KV caches of ten simultaneous requests?
    • Performance: can each session sustain 25 tokens/s while all ten decode concurrently?

    Capacity is generally fine at 8K context, but at 32K and above the math needs to be redone, as described in the previous section.

    Performance depends on the efficiency of continuous batching. A good scheduler merges decode steps across requests, amortizing weight-read costs so aggregate throughput scales with concurrency. But decoding is memory-bandwidth-sensitive, and multiple concurrent sessions increase bandwidth contention. MoE models under high concurrency may also suffer from hot-expert load imbalance, overloading some GPUs.

    4. Stress Testing Methodology

    “25 tokens/s per session” is a service-level commitment that must be validated under realistic load. At minimum, record the following during stress tests:

    • GPU model and count, driver and CUDA versions, inference framework and version;
    • Context length distribution — e.g., what share of traffic is 8K vs. 32K;
    • Input and output lengths, and concurrency levels;
    • Time to first token (TTFT), plus P50 and P95 per-session decode speed;
    • Aggregate throughput and peak VRAM usage.

    Do not substitute someone else’s benchmark scores or VRAM capacity estimates for your own stress testing.

    When troubleshooting performance, check in this order: hot-expert load distribution, inter-GPU communication time, and the continuous batching configuration (max batch size and max tokens per step).

    5. How to Calculate Cost

    Buying a full server as an individual is unrealistic; the closest reference is public cloud GPU instance rental pricing.

    Cost formulas:

    • Hourly cost of an 8-GPU node = 8 × per-GPU hourly price
    • Annual cost = hourly node cost × 8,760

    Per-GPU hourly pricing varies widely by cloud provider, GPU model, region, reservation term, and whether storage and networking are included. This article uses $5 per GPU per hour as an example; actual prices can vary across a wide range. The table below estimates three price tiers ($3, $5, $8) and excludes storage, egress traffic, and operational labor:

    Per-GPU hourly price 8-GPU node hourly Monthly cost (730 hrs) Annual cost (8,760 hrs)
    $3 $24 ~$17.5K ~$210K
    $5 $40 ~$29.2K ~$350K
    $8 $64 ~$46.7K ~$560K

    The key variable in the self-hosting vs. API comparison is utilization. API annual cost = annual token volume ÷ 10^6 × price per million tokens. Self-hosting annual cost includes hardware depreciation or rental, power, facilities, operations, and networking. The annual token volume at which the two are equal is the break-even point. Below that volume, APIs are usually more economical.

    Self-hosting only makes financial sense when strict data compliance requirements combine with stable, sufficiently large call volumes.

    6. Decision Framework

    1. Confirm necessity first: only consider self-hosting when daily token volume, data compliance requirements, and latency requirements all demand it;
    2. Confirm the hardware floor: an eight-GPU node with 192 GB per card is the minimum viable configuration, with thin headroom;
    3. Size the KV cache against your target context length and concurrency to determine whether you need more VRAM or a second node;
    4. Stress test with real context and concurrency patterns — treat measured numbers as authoritative;
    5. Run the annualized cost comparison against the API break-even point before committing.

    Running large models locally is technically feasible, and the cost is substantial. Working out VRAM, concurrency, and cost before investing is far cheaper than adjusting after going live.