Local Deployment of a 2.8-Trillion-Parameter MoE Model: VRAM and Budget Planning

    This article explains how to estimate VRAM and cloud costs for locally deploying a 2.8-trillion-parameter MoE model. It covers MXFP4 weight storage, runtime overhead and KV cache, FP4 hardware support, three deployment tiers, cloud pricing, and a pre-deployment checklist.

    1. Start with the Conclusion

    A large MoE model with roughly 2.8 trillion total parameters cannot be run on a few consumer-grade GPUs. The MoE architecture reduces computation per token through sparse activation: each token uses only about 104 billion parameters. However, all weights must remain resident in GPU memory. Computation is sparse; storage is not. GPU memory capacity is therefore the first barrier.

    Based on the estimates in this article, the minimum viable configuration is an 8-GPU data center server with about 1.5 TB or more of total GPU memory. This is already a server-class project, not a workstation-class one.

    2. How to Calculate GPU Memory

    2.1 Weight Storage

    This article calculates based on the model’s officially released MXFP4 format. MXFP4 uses 4 bits per parameter, and every 32 parameters share one 8-bit scaling factor, averaging about 4.25 bits per parameter.

    1. Total parameters × average bits per parameter ÷ 8
    2. = 2.8 trillion × 4.25 bit ÷ 8
    3. ≈ 1.49 TB (decimal)
    4. ≈ 1.35 TiB (binary)

    If the scaling factors are ignored, a 4-bit estimate gives about 1.4 TB. This article uses decimal TB throughout; hardware memory specifications should follow vendor ratings.

    If weights are stored in FP16, each parameter occupies 16 bits, which is 4 times the 4-bit case. The weights would be about 5.6 TB (about 5.1 TiB), far above the 1.49 TB of the 4-bit approach. Different precision strategies require very different hardware scales.

    2.2 Runtime Overhead and KV Cache

    Beyond weights, there are three additional overhead components:

    • Fixed overhead: inference framework, communication buffers, CUDA contexts, and similar costs. These are not strongly related to the number of users.
    • Activations: intermediate computation results for each layer. These vary with batch size and sequence length and need to be determined through measurement.
    • KV Cache: the key-value cache for attention. It grows linearly with context length and concurrency and is the most commonly underestimated part of long-context scenarios.

    The KV Cache formula is as follows, where the coefficient 2 represents the K and V caches:

    1. KV Cache bytes = 2 × layers × KV heads × head dimension × context length × concurrency × bytes per element

    Here is an example assuming the model has 80 layers, 8 KV heads, head dimension 128, uses BF16 (2 bytes) cache, has a 32K context, and serves a single request:

    1. 2 × 80 × 8 × 128 × 32768 × 1 × 2 = 10 GiB

    That is, each 32K-context stream uses about 10 GiB, and 10 concurrent streams use about 100 GiB. These parameters are only example assumptions, not the actual configuration of this model. Different architectures also calculate KV Cache differently. For example, attention mechanisms with KV compression require using the actual layer count and head count from the model configuration file.

    Therefore, this article takes the total requirement for a single-user short-context scenario as 1.5 TB: about 1.49 TB for weights, plus fixed overhead, with 1.5 TB as the lower bound. KV Cache is not included in this number and must be calculated separately using the formula above. The reserved GPU memory headroom should not use a fixed percentage; it should be determined by the KV Cache formula and measured framework overhead.

    2.3 Precision and Hardware Support

    Being able to fit the model is not the same as running it well. Data center GPUs fall roughly into two categories:

    • Cards without native FP4 compute support, such as Hopper-architecture H100 (80GB) and H200 (141GB).
    • Cards with native FP4 compute support, such as Blackwell-architecture B300, as well as AMD CDNA4-architecture MI350X and MI355X.

    On cards without native FP4 support, weights must be dequantized to a higher precision before computation. This adds extra compute overhead and memory bandwidth pressure, and it also consumes KV Cache space, significantly reducing throughput. If the weights are fully dequantized and kept resident in GPU memory, the memory requirement returns to about 5.6 TB, which is also impractical.

    The table below lists the minimum number of cards needed to satisfy 1.5 TB of weights. This is only a lower bound based on memory capacity. Actual deployments usually use an 8-GPU whole machine to leave room for KV Cache and meet parallel sharding requirements.

    GPU model Memory per GPU Minimum GPUs for weights only Native FP4 support
    H100 80GB 19 No
    H200 141GB 11 No
    B300 About 288GB 6 Yes
    MI355X About 288GB 6 Yes

    “Hardware supports the format” and “the framework is already optimized” are two different things. Hardware support for FP4 does not mean the inference framework version you use has implemented efficient kernels for this model. Before selecting hardware, confirm the model’s support status in the target framework documentation and rely on measured results. Some framework deployment examples use 8-GPU parallelism as a configuration example; refer to the documentation for the framework version you use for the specific configuration.

    3. Configuration Approaches for Three Requirement Tiers

    Tier Performance target Minimum GPU memory Suggested configuration Cost magnitude (example assumption) Measure before launch
    1: Just runnable Can load and generate normally About 1.5TB 8 GPUs with native FP4 support About $38/hour First-token latency, usable context length
    2: Single stream at 25 tokens/s Single-user decode speed meets target About 1.5TB plus KV Cache for target context Same as above Same as above Single-stream decode speed at target context
    3: 10 concurrent streams at 25 tokens/s each Aggregate about 250 tokens/s About 1.5TB plus KV Cache for 10 streams Same as above; confirm with stress testing Same as above Batch throughput, per-stream speed variation

    All three tiers use the same hardware; the difference is the stress-test target.

    Tier 1 is centered on capacity. All weights must fit. Under the assumptions in this article, configurations with fewer than 8 GPUs will find it difficult to satisfy both capacity and headroom requirements.

    Tier 2 requires judging 25 tokens/s together with the actual context. The decode phase is mainly limited by GPU memory bandwidth, so a theoretical upper bound can be estimated. The active weights that must be read per token are about 104 billion parameters × 4.25 bit ÷ 8 ≈ 55GB. If the combined effective bandwidth of 8 GPUs is about 64TB/s (about 8TB/s per GPU, subject to vendor specifications), the theoretical upper bound is far above 25 tokens/s. However, communication, scheduling, and KV Cache reads will significantly reduce actual speed. This theoretical value only shows that the target may be plausible in order of magnitude; it cannot replace measurement.

    Tier 3 focuses on scheduling. 10 concurrent streams mean the system needs to sustain about 250 tokens/s. Based on the example assumptions, the remaining memory of 8 GPUs can hold roughly 50 requests with 32K context, far exceeding the 10-stream requirement. Therefore, 10 concurrent streams do not necessarily require adding servers. The real challenge is whether each stream can maintain 25 tokens/s when 10 streams generate at the same time. This depends on:

    • The scheduling efficiency of the inference framework’s continuous batching;
    • The memory management strategy for KV Cache;
    • The allocation of GPU compute and memory bandwidth across multiple requests.

    These are throughput and scheduling metrics that can only be verified through stress testing, not inferred from GPU memory capacity alone.

    The calculation is as follows, and all values are example assumptions: 8 GPUs provide 2304GB of total memory. Subtract about 1490GB for weights, leaving about 814GB. After deducting about 10% (about 230GB) for framework and communication buffers, about 584GB remains for KV Cache. At about 10.7GB (10GiB) per stream, this theoretically accommodates about 54 streams. Considering fragmentation and peak fluctuations, actual reliable concurrency should be planned at a more conservative value.

    4. Cloud Costs and Procurement Boundaries

    Data center GPUs are not sold through consumer retail, and procurement channels, lead times, and warranties can all become issues. Cost estimates are therefore usually anchored to cloud platforms, or enterprises compare cloud and self-built options.

    The following calculation uses an example unit price of $4.76/GPU/hour. This unit price is assumed on a reserved-instance basis and is not a real-time quote:

    1. 8 GPUs × $4.76/GPU/hour = $38.08/hour
    2. $38.08 × 730 hours ≈ $27,800/month
    3. $27,800 × 12 ≈ $330,000/year

    Exchange rates and unit prices will change. Before use, recalculate with the target cloud provider, target region, and billing period’s actual quote.

    The applicable scenarios for different procurement approaches are as follows:

    Approach Suitable scenario Main risk
    On-demand billing Highly variable load, short-term validation High unit price; high long-term running cost
    Reserved instances Stable load, long-term availability required Payment required even when idle during the commitment period
    Managed or self-purchase Data compliance requirements, long-term large-scale use Long procurement cycle; high operations and data center costs

    Idle cost must be calculated separately. If actual utilization is only 30%, the effective unit price becomes $38.08 ÷ 0.3 ≈ $127/hour.

    In addition, storage (1.5TB-class weight files and their replicas), network, operations labor, and monitoring must be included. These are usually not included in the GPU unit price.

    5. Selection Checklist

    Check item What to verify Pass criterion
    GPU memory capacity Sum of weights, fixed overhead, and KV Cache for target concurrency Headroom remains, with headroom determined by stress testing
    Precision support Whether hardware natively supports the model’s FP4 (MXFP4) format Supported, and the framework has corresponding kernels
    Framework support Whether the inference framework version officially supports the model Documented clearly and version number fixed
    Interconnect and bandwidth Whether intra-node GPU interconnect meets expert-parallel communication needs Stress testing shows no obvious bottleneck
    Stress-test scenarios Actual context length, concurrency, request distribution Covers real business, records P95 latency and per-stream tokens/s
    Budget approval Cloud cost, utilization, storage, network, operations —