Local Deployment of a 2.8T MoE Model: VRAM and Cost Estimation

    This article provides a technical estimation and strategy for locally deploying a 2.8 trillion parameter Mixture-of-Experts (MoE) model. It covers VRAM calculation including weights, KV Cache, and overheads, hardware requirements, communication topology impacts, and hardware cost estimation. It also outlines a phased deployment path from cloud verification to local scaling, along with the deployment process, acceptance metrics, and risk disclaimers.

    1. VRAM Depends on Total Parameters, Activated Parameters Only Affect Compute

    This model has a total of 2.8 trillion parameters, containing 896 expert modules. Each token only activates 16 experts, an activation ratio of about 1.79%. Many people mistake this ratio for the proportion of VRAM required, which is a misconception. The activation ratio determines the computation required per step, but the VRAM must accommodate the weights of all experts. The routing mechanism selects which expert to compute during inference, but all other experts must already be in VRAM. If weights had to be loaded from external storage on every route, the latency would be unacceptable.

    The model uses Quantization-Aware Training (QAT), with weights in MXFP4 format and activations in MXFP8 format. In MXFP4, every 32 4-bit elements share an 8-bit scaling factor, so the average memory footprint per element is 4 + 8/32 = 4.25 bits. The lower bound for weight storage is calculated as follows:

    2.8T × 4.25 ÷ 8 ≈ 1487.5 GB

    This figure only includes the quantized weights themselves. Unquantized tensors, metadata, memory alignment padding, runtime workspace, and KV Cache must be accounted for additionally. A common empirical overhead buffer is 15% to 25%, corresponding to approximately 1711 GB to 1859 GB. This is just an empirical value and needs to be calibrated with actual test results. The GB unit used in this text follows the decimal system, consistent with the nominal VRAM capacity of graphics cards.

    2. KV Cache Formula and Sensitivity

    KV Cache grows with context length and concurrency. The calculation formula is as follows:

    KV Cache Bytes = 2 × L × n_kv × d_h × S × B × b

    Where L is the number of layers, n_kv is the number of KV heads, d_h is the dimension per head, S is the context length, B is the number of concurrent sequences served, and b is the number of bytes per cache element (2 for BF16, 1 for FP8). The coefficient 2 represents the two caches for K and V.

    These parameters need to be read from the model’s configuration file (e.g., config.json). This article does not provide the specific values for this model; instead, a set of hypothetical values is used below to demonstrate the calculation process: L=60, n_kv=8, d_h=128, S=32768, B=16, b=2. The result is:

    2 × 60 × 8 × 128 × 32768 × 16 × 2 ≈ 128.8 GB

    This result shows that if the concurrency or context length doubles, the KV Cache doubles, while the weight footprint remains unchanged. If the model uses MLA (Multi-head Latent Attention) or Sliding Window Attention, the cache structure is different, and the above formula must be replaced with the corresponding architectural formula.

    Calculated by the weight lower bound, 1487.5 GB divided by 80 GB per card is about 18.6 cards, meaning the weights alone require at least 19 cards. Adding the typical overhead of 1711 to 1859 GB, about 22 to 24 cards are needed. KV Cache and communication buffers will further increase the card count. If the total requirement is set between 2000 GB and 2200 GB, actual parameters must be substituted into the above formula for verification. Until verification is complete, this range is only an estimate.

    3. Communication Topology Determines Performance Beyond GPU Count

    In distributed inference, each token is routed and sent to the device where the target expert resides, and the result is sent back to the original device after computation. This process relies on all-to-all communication, and communication latency accounts for a high proportion of end-to-end latency. The bandwidth and latency of inter-GPU interconnects often determine throughput more than sheer VRAM volume.

    Deployment documents from some vendors typically start with supernodes containing 64 or more accelerators, connected internally by high-bandwidth, low-latency interconnects (such as NVLink domains or similar technologies). Specific recommended solutions should be based on the model publisher’s deployment documentation, which should be requested before procurement. If a loosely coupled cluster stitched together via multi-node 10GbE is used, the communication overhead may offset the computational savings of the MoE architecture.

    4. Three Configuration Tiers: Choose by Use Case, Not Just VRAM

    Configuration Accelerators Total VRAM Applicable Scenarios Main Limitations
    Functional Verification 32 cards, 80GB each 2560GB Verify loading, functional correctness, small-scale testing Limited VRAM headroom, small KV Cache space, cannot be used as a throughput reference
    Standard Production 64-card supernode 5120GB Internal enterprise multi-team concurrent use Requires supernode interconnect; procurement cycle and power conditions must be confirmed in advance
    High-Concurrency Scaling 128 cards or more Determined by scale For a large number of external users Requires multi-tier storage, expert load balancing, and O&M systems

    The total VRAM for 32 cards is 2560GB, higher than the previously estimated range of 2000 GB to 2200 GB, leaving a margin of 360 GB to 560 GB. It can be trialed for single-user interaction, but the throughput and latency measured for multi-user scenarios will not be representative.

    5. Hardware Costs: Itemize and Replace Estimates with Quotes

    This article does not provide purchase prices or monthly rental ranges. Hardware prices fluctuate based on accelerator models, procurement volumes, supply cycles, taxes, and exchange rates, and the quoting standards from different sources vary significantly. A figure without a specified standard can easily mislead procurement decisions. Cost estimation should at least include the following items:

    • Accelerators and complete supporting servers
    • High-speed interconnect equipment (switches, cables, backplanes)
    • Storage systems (model weights, business data, logs)
    • Power and cooling, annual electricity cost ≈ total system power (kW) × annual operating hours × PUE × electricity price
    • Data center and network infrastructure
    • Maintenance, spare parts, and O&M labor
    • Taxes, shipping, and exchange rate conditions

    When renting cloud GPUs, confirm instance specifications, billing methods, storage and public network traffic costs, and the conditions for reserved or spot instances.

    The break-even point between self-building and renting can be estimated using the following formula:

    Break-even Months = One-time Investment ÷ (Monthly Rental Cost − Monthly Operating Cost)

    If the monthly rental cost is not higher than the monthly operating cost, self-building cannot recover the investment through rental expenses. This formula does not account for the cost of capital, depreciation, or the risk of hardware obsolescence after model iterations.

    6. Phased Deployment Path and Decision Gates

    The first stage is cloud verification. Rent supernode instances and pay by the hour to complete weight verification, performance benchmarking, and business adaptation. The goal of this stage is to confirm that the model’s effectiveness matches the hardware configuration and to accumulate operating parameters.

    The second stage is hybrid deployment. Keep core services on high-performance cloud nodes while procuring a small number of accelerators to set up a local development and testing environment for daily tuning and experimental iteration. This model controls initial capital expenditure.

    The third stage is local scaling. Only purchase local clusters in phases when the following conditions are met simultaneously: daily request volume and peak QPS are stable; monthly cloud costs are approaching the self-build break-even point; data compliance requirements prohibit data from leaving the domain, or cloud latency cannot meet the SLA.

    Each stage should retain a fallback plan. For example, retain cloud contracts or standby nodes, fix the model version and inference engine version, and gradually roll out new versions.

    7. Deployment Process

    Deployment can be carried out in the following four steps:

    1. Obtain the weight files and verify their integrity. Publishers usually provide SHA256 checksum files, which can be verified using the following command:
    1. sha256sum -c SHA256SUMS
    1. Set up the inference service and configure the combination of tensor parallelism and expert parallelism. Specific parameters depend on the inference engine and should follow the engine’s documentation.
    2. Load the quantized weights, set the KV Cache allocation strategy, and define the maximum context length.
    3. Run benchmark tests and record Time To First Token (TTFT), throughput, and VRAM usage.

    During operation, you can use the following command to monitor NVIDIA GPU VRAM and utilization. If using accelerators from other vendors, switch to the corresponding vendor’s monitoring tools:

    1. nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu --format=csv -l 5

    8. Acceptance Metrics

    Metric Measurement Method Acceptance Criteria
    Model Load Time Time from startup to first service availability Determined by the O&M window
    TTFT (Time To First Token) Recorded step-by-step by benchmark script based on concurrency gradients Business SLA
    Output Tokens per Second Calculate steady-state throughput Business goals
    Peak VRAM Continuously sample VRAM usage Peak must retain a margin to avoid overflow under pressure
    Communication Hotspots Analyze expert routing load distribution and communication time Load differences among cards are within an acceptable range

    During acceptance, the system should run under sustained high-concurrency pressure to observe whether VRAM overflows and whether performance jitter occurs due to communication hotspots. Before launching the service externally, it is recommended to observe with real business traffic for at least two weeks and complete tuning before officially opening it.

    9. Risk Disclaimers

    Quantization accuracy, KV Cache parameters, quotes, and supply cycles will all affect the final conclusion. The VRAM figures in this article are estimates and should be based on actual configuration tests. Hardware and cloud service prices change rapidly, and current quotes must be obtained before procurement. The version combination of the inference engine, drivers, and accelerator firmware may also affect performance, requiring item-by-item verification before launch. As quantization technology and inference engines continue to evolve, the deployment cost for equivalent models still has room to decrease. Major procurement decisions should be preceded by continuous tracking of technological trends.