VRAM Requirements for Local Deployment of a 2.8T Open-Source MoE Model

    This article details the VRAM requirements for locally deploying a 2.8T open-source Mixture-of-Experts (MoE) model. It covers parameter scaling, expert routing mechanisms, and detailed VRAM estimation, including model weights and KV Cache. Additionally, it explores hardware planning, offloading strategies, quantization risks, parallelism strategies, and a practical deployment roadmap to help engineers effectively manage resources and optimize inference performance.

    1. Parameter Scaling

    This open-source MoE model has a total of approximately 2.8T parameters, with about 104B parameters activated per token during inference. The activated parameters account for roughly 3.7% of the total, meaning the theoretical compute required for a single token is about 1/27th of a dense model with the same total parameter size. This ratio only reflects parameter count; attention computation, routing, and communication overheads are excluded, so actual speed still requires empirical testing.

    Parameter scale directly determines the VRAM occupied by weights, calculated as follows:

    VRAM for Weights = Parameter Count × Bytes per Parameter

    • FP16 (2 bytes): 2.8T × 2 = 5.6TB
    • INT8 (1 byte): 2.8T × 1 = 2.8TB
    • INT4 (0.5 bytes): 2.8T × 0.5 = 1.4TB

    Weights represent a fixed inference overhead. Without offloading, all expert weights must reside in VRAM because any token can be routed to any expert.

    2. MoE Architecture and Expert Routing

    The described structure consists of 896 fine-grained experts and 2 shared experts. The router selects a subset of experts for each token. The text mentions that 16 experts are activated per token, though it is unclear if this includes the 2 shared experts. This should be confirmed with the official configuration.

    Sparse activation seems highly efficient for a single token, but the situation changes during batch inference. Different tokens route to different experts, meaning the number of experts involved in a batch increases with batch size. Assuming a uniform random distribution across 896 experts, with 16 experts selected per token per layer, 8 concurrent tokens would utilize approximately 120 distinct experts per layer, while 64 concurrent tokens would utilize around 613. In reality, routing is rarely uniform; popular experts are selected repeatedly. Therefore, these numbers serve only as an order-of-magnitude reference.

    This implies that the VRAM savings from “only loading currently active experts” diminish as batch size increases. The previously mentioned 208GB (104B × 2 bytes) represents the FP16 weights of the activated parameters for a single token, acting as a lower bound, and does not equate to the actual resident VRAM requirement.

    3. VRAM Estimation Breakdown

    Actual VRAM requirements consist of the following components:

    1. Model Weights: Depends on quantization precision and offloading strategy.
    2. KV Cache: Grows linearly with sequence length, batch size, and attention structure.
    3. Activations and Intermediate Buffers: Tied to batch size, sequence length, and the inference framework’s implementation.
    4. Routing Tables, Communication Buffers, and Framework Overhead: Internal operational costs.

    The KV Cache is calculated using the following formula:

    KV Cache Bytes = 2 × Num_Layers × Num_KV_Heads × Head_Dim × Seq_Length × Batch_Size × Bytes_per_Element

    The coefficient 2 represents the two separate caches for Keys and Values. Assuming the model has 60 layers, 8 KV heads (using Grouped-Query Attention, GQA), a head dimension of 128, a sequence length of 8192, a batch size of 1, and FP16 precision, the KV Cache would be approximately 2,013,265,920 bytes, or about 2.0GB. Under the same assumptions, a batch size of 20 would require around 40GB. If the model does not use GQA (where the number of KV heads equals the number of attention heads, e.g., 64), the KV Cache for a single 8k sequence would be about 16GB.

    The claim that an “8k context requires an extra 40-60GB” cannot be reproduced without configuration details like the number of layers and KV heads. Readers should extract these parameters from the official config file and recalculate using the formula.

    4. Hardware Planning for Local Deployment

    The following are calculated VRAM capacities, not empirical throughput results:

    • INT8 full residency requires at least ~35 80GB GPUs (2800GB ÷ 80GB), excluding KV Cache and other overheads.
    • INT4 full residency requires at least ~18 80GB GPUs (1400GB ÷ 80GB), though actual deployment typically requires more.

    Based on this, we can draw conclusions for three common configurations:

    Configuration Total VRAM INT4 Weights (1.4TB) Full Residency Applicable Scenario
    1 node 8×80GB 640GB No, requires offloading to RAM or SSD Functional verification
    2 nodes 8×80GB 1280GB No, requires offloading to RAM or SSD Small-scale debugging
    4 nodes 8×80GB 2560GB Theoretically yes, ~1.1TB left for KV Cache etc. Near-production testing

    An INT8 deployment cannot achieve full residency on 4 servers because 2560GB is less than 2800GB. Achieving full INT8 residency requires at least 5 similar servers.

    5. Costs of Offloading Strategies

    When VRAM is insufficient to hold all weights, weights can be stored in CPU RAM or on SSDs, keeping only the currently required experts in VRAM. The speed of this approach depends on three factors: the bandwidth of the PCIe or other interconnect, the expert cache hit rate, and the effectiveness of the prefetch strategy. A lower hit rate means more data must be read from slow storage per token, increasing latency.

    Offloading is suitable for verifying that the model runs correctly and outputs meet expectations. Whether it is viable for production should be decided after measuring throughput and latency on target hardware under real workloads.

    6. Quantization Risks and Evaluation

    Both INT8 and INT4 require a calibration dataset. Calibration data should closely resemble actual business data, otherwise precision loss may exceed expectations. Some schemes perform sensitivity analysis on different layers or experts, retaining higher precision for sensitive components.

    Handling shared experts requires a trade-off. Keeping shared experts in FP16 helps reduce quantization errors, but it increases VRAM usage and complicates tensor parallelism partitioning. An alternative fallback strategy is to use INT8 for the attention layers and shared experts, INT4 for routed experts, and adjust based on evaluation results.

    Post-quantization, at least the following should be checked:

    • General language capability benchmark results compared to the unquantized version.
    • Success rate on target business tasks, such as QA accuracy or code pass rates.
    • Format accuracy and parameter accuracy for tool calling.
    • Information retrieval accuracy in long-context scenarios.
    • Inference latency and peak VRAM.

    Precision loss is highly task-dependent; there is no fixed metric applicable to all scenarios.

    7. Choosing Parallelism Strategies

    Inference for ultra-large-scale MoE models typically requires a combination of parallelism strategies:

    • Tensor Parallelism (TP): Splits the weights within a single expert across multiple GPUs by row or column. Suitable for multi-GPU within a single node due to low communication overhead.
    • Expert Parallelism (EP): Places different experts on different GPUs or nodes. The routing results dictate data flow.
    • Pipeline Parallelism (PP): Splits the model by Transformer layers, reducing the frequency of cross-node communication, but introduces pipeline bubbles that require micro-batch scheduling to mitigate.

    Consider 8 GPUs using pure Expert Parallelism: the 896 experts are evenly distributed, meaning each GPU hosts 112 complete experts, with no tensor splitting.

    Consider 2 nodes with 4 GPUs each, utilizing 4-way TP intra-node and 2-way EP inter-node. Each expert group corresponds to one node and is responsible for 448 experts. Each GPU holds 1/4 of the shards for those 448 experts. In terms of total weight volume, each GPU is equivalent to 112 complete experts, but it does not hold 112 complete experts. This distinction can easily cause misunderstandings; implementation should be based on the actual VRAM usage of the shards. The cross-node communication overhead of Expert Parallelism must also be included in the evaluation.

    8. Required Inference Engine Capabilities

    An inference engine capable of supporting this model needs at least the following features:

    • Dynamic Expert Loading and Multi-level Caching: Place popular experts in VRAM, sub-popular experts in RAM, and cold experts on SSD. Cache eviction strategies can use LRU (Least Recently Used) or its variants, with prefetching based on actual routing records.
    • Batching Grouped by Expert Sets: Grouping requests that use the same combination of experts for computation can improve tensor core utilization.
    • Sparse Attention and Sliding Window Attention Fusion: This is only meaningful if the model architecture actually employs such attention mechanisms; it should not be assumed by default.

    Engine versions, startup parameters, and the quantization toolchain all affect the outcome. When selecting an engine, confirm whether the specific version supports the model’s MoE structure and offloading features, and record the version number and parameters for reproducibility.

    9. LoRA in Fine-Tuning

    The purpose of LoRA is to freeze the base weights and only train and save low-rank adapters. Therefore, LoRA does not reduce the size of the base weights. During deployment, the complete base model must still be loaded, but the gradients and optimizer states during training only cover the adapter parameters, significantly reducing training VRAM.

    For MoE models, a common practice is to freeze the shared experts and router, adding adapters only to experts relevant to the target domain. Determining which experts are domain-relevant requires first analyzing routing statistics using domain data. This approach may limit the model’s adaptability, so its effectiveness should be checked using an evaluation set.

    The extent of training VRAM savings depends on the base precision, adapter rank, optimizer type, and sequence length. It should be measured on the target hardware rather than relying on a fixed ratio.

    10. Multimodal Deployment

    Estimates place the visual encoder at around 2 to 3 billion parameters, though this figure should be verified against the official model card. The visual encoder is deeply coupled with the text backbone. Deploying the visual encoder separately on another set of devices can free up GPU VRAM for text generation, but it adds an extra service call, increasing latency for image inputs. Whether to split the deployment depends on the proportion of image requests and latency requirements.

    Curriculum learning was reportedly used during training, starting with pure text tasks before gradually introducing multimodal and reasoning tasks. The effectiveness of this training strategy should be referenced in official technical reports; this article makes no judgment on it.

    Sensitivity-Aware Quantization is a class of methods whose core idea is to first measure the sensitivity of different layers or experts to quantization error using calibration data, and then allocate different bit-widths accordingly. The benefits of this approach require empirical testing based on the task, as differences between models and datasets are significant.

    CXL-based memory expansion may allow GPUs direct access to large-capacity RAM. Currently, this capability depends on specific hardware and drivers, and the bandwidth and latency are far lower than VRAM. It cannot be considered a direct solution for VRAM bottlenecks.

    Introducing structured sparsity during training has the potential to make expert allocation during inference more regular, thereby reducing routing overhead. This direction is still under exploration, and there is currently insufficient public evidence to support its effectiveness.

    12. Implementation Sequence

    It is recommended to proceed with local deployment in the following order:

    1. Read the official configuration file and recalculate the VRAM for weights and KV Cache using the formulas.
    2. Run the model on a small-scale configuration to confirm it loads and outputs correctly.
    3. Conduct precision evaluation under the target quantization scheme and compare it with the unquantized version.
    4. Adjust parallelism strategies and caching parameters, recording throughput and latency.
    5. Integrate the service layer to handle batch scheduling, timeouts, and load balancing.