VRAM and Cost Breakdown for Deploying Large-Scale MoE Models Locally

    A practical guide to what it really takes to run trillion-parameter Mixture-of-Experts (MoE) models on-premises. This article explains why VRAM must be sized against total parameters rather than activated parameters, walks through the 4-bit quantization math including scale overhead, and shows how these numbers translate into a real memory floor. It also covers why high-speed inter-card interconnect is as critical as raw VRAM capacity, breaks down the full cost of ownership beyond the GPUs themselves, and offers a step-by-step decision framework for choosing between self-hosting and cloud deployment.

    1. Why Locally Deploying Trillion-Parameter MoE Is Hard

    A new wave of Mixture-of-Experts (MoE) models with total parameter counts reaching the trillion scale has emerged in recent years. They are powerful, but they turn “on-premises private deployment” from a slogan into a real engineering and financial challenge. Many people’s first reaction is: “The model is huge, so can’t we just count the part that actually gets activated?” This is a common misconception. This article systematically breaks down, from parameter structure and quantization precision to the VRAM floor, hardware interconnect, and cost composition, what you actually need to prepare and how much it will cost to deploy these models.

    2. First, Get One Key Concept Straight: VRAM Depends on Total Parameters, Not Activated Parameters

    Take an MoE model with a total parameter count of roughly 2.8 trillion as an example. Internally it contains 896 experts (Experts). For each token, only 16 of those experts are activated after passing through the router (Router), giving an expert activation ratio of about 1.79%.

    Two points must be emphasized here:

    1. Expert activation ratio ≠ parameter activation ratio. The 1.79% figure is the share of experts hit by routing, not the share of parameters that participate in computation. The two are not interchangeable.
    2. VRAM usage depends on total parameters, not activated parameters. No matter how many experts you activate in a single inference pass, you must preload the weights of all 896 experts into VRAM or unified memory in advance, then invoke them dynamically based on the routing result at inference time. The activation ratio only affects compute consumption (FLOPs) and compute bandwidth requirements; it does not reduce the volume of weights you need to “store.”

    The attention structure of these models typically also introduces a variety of optimizations, such as linear attention/Delta-rule attention, Attention Residuals, and low-rank latent attention (MLA-type). These modifications mainly affect compute efficiency and long-sequence modeling ability, but they do not change the fundamental constraint of MoE deployment: you must first load all the weights.

    So the first step in assessing deployment feasibility is always to convert the “total parameter count” into a “weight VRAM floor.”

    3. How Quantization Lowers VRAM: Using 4-bit as an Example, Don’t Overlook the Scale Overhead

    Simply compressing 2.8 trillion parameters down to 4-bit gives a theoretical raw-data floor of:

    1. 2.8e12 × 4 bit ÷ 8 ≈ 1400 GB

    Does it look like 1400GB is the limit? Not even close. Modern quantization formats (such as MXFP4), in order to preserve precision with low-bit weights, attach a shared scale factor to each group of elements. Under the OCP microscaling specification, one 8-bit scale is typically paired with every 32 4-bit elements, so the real weight overhead becomes:

    1. 2.8e12 × (4 + 8/32) bit ÷ 8 = 2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

    A simple estimation function can help you quickly calculate this:

    1. def estimate_weight_memory_gb(total_params, bits, scale_bits=0, elems_per_scale=32):
    2. """Rough estimate of MoE weight VRAM/memory usage (GB)"""
    3. effective_bits = bits + scale_bits / elems_per_scale
    4. return total_params * effective_bits / 8 / 1e9
    5. # 2.8T parameters, MXFP4 (4-bit) + one 8-bit scale per 32 elements
    6. print(estimate_weight_memory_gb(2.8e12, 4, 8, 32)) # ≈ 1487.5 GB

    And that is only the “weights.” In a real deployment, you must additionally reserve:

    • Non-4-bit tensors: LayerNorm, Embedding, and some Router gating parameters, which are generally kept at higher precision;
    • Metadata and alignment padding: memory layout alignment, index structures, and so on;
    • Runtime workspace: activation cache, temporary buffers, communication buffers;
    • KV Cache: this is a major VRAM consumer in long-context scenarios, growing linearly with concurrency and sequence length, potentially reaching tens to over a hundred GB.

    Stacking all of these together, the “full” deployment of a trillion-scale MoE has a VRAM floor that comfortably stands at over 1.6TB.

    As a point of comparison, a mid-sized MoE model of about 670B parameters needs roughly 685GB for FP8 weights; even compressed to AWQ 4-bit it still needs about 335GB of weights plus about 37GB of activations, totaling roughly 400GB just to barely fit into 8 high-end accelerator cards (640GB of VRAM combined). The scale of a trillion-level model is more than 4 times that, which is exactly the fundamental reason the “full deployment” barrier stays so high.

    4. Hardware Selection: Why Stacking Cards Isn’t Enough—You Also Need High-Speed Interconnect

    Having enough VRAM is only the ticket to entry; the real bottleneck often lies in communication.

    Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts spread across dozens or even hundreds of cards, each token’s activated values must be “moved” over a high-speed network to the card hosting the target expert to complete computation, then moved back afterward. If inter-card interconnect bandwidth is insufficient, the latency of this data movement directly eats into valuable compute, leaving the expensive accelerator cards idle and waiting.

    For this reason, the official deployment recommendations for these models usually point to a Super Node of 64 or more accelerators, using NVLink or equivalently high-speed interconnects to weave a large number of cards into a low-latency, high-bandwidth unified communication domain, keeping inter-expert data movement within this domain as much as possible.

    The takeaways for deployers here are:

    • Choosing cards by VRAM capacity alone is not enough; you must evaluate the inter-card interconnect topology (fully connected / hierarchical interconnect);
    • The partitioning strategy for Expert Parallelism should align with the physical topology, so that experts with high-frequency communication land within the same communication domain;
    • If you use standard Ethernet without high-speed interconnect, communication becomes an absolute bottleneck, and measured throughput may fall far below theoretical values.

    5. Cost Breakdown: VRAM Is Only the Tip of the Iceberg

    Back to the original question—how much does local deployment actually cost? The bill is far more than just buying cards. A complete tally includes at least:

    1. Hardware acquisition or leasing: dozens of high-end accelerator cards + matching high-speed NICs + high-clock-rate CPUs + large-capacity memory (if the weights cannot all fit in VRAM, they must spill over to host memory or even NVMe, which demands extremely high IO bandwidth). This is an investment that runs from millions to tens of millions.
    2. Datacenter and racks: high-power cabinets, load-bearing, cabling, and the rent or self-build cost of rack space.
    3. Power and cooling: the total power draw is extremely high, and the matching cooling (air cooling or even liquid cooling) is a large, ongoing expense; electricity bills add up to a considerable amount over a year.
    4. Operations and personnel: driver/framework tuning, quantization calibration, monitoring and alerting, and fault handling all require long-term investment from a specialized team.
    5. Opportunity cost: hardware iterates quickly, and a self-built cluster also faces the risk of depreciation and low utilization.

    If the scale, stability, and operational capability of a self-built cluster are all problematic, cloud deployment is often the more pragmatic choice. Cloud platforms such as Baidu AI Cloud provide a complete capability stack—from GPU/accelerator-card instances and high-performance parallel file systems to container platforms, monitoring and alerting, and logging services—available on-demand and billed by usage, sparing you the heavy burden of building a datacenter and long-term operations. For specific resource specifications and billing rules, refer to the official documentation and an actual quote.

    6. A Practical Deployment Decision Framework

    Synthesizing the analysis above, you can make a feasibility judgment in the following order:

    1. Calculate the weight floor: use total parameters × effective bits (including scale) ÷ 8 to get the weight VRAM, then add non-quantized tensors, runtime workspace, and KV Cache to get the total VRAM requirement.
    2. Determine the parallelism strategy: how to combine tensor parallelism, pipeline parallelism, and expert parallelism, and whether the expert partitioning fits the physical interconnect topology.
    3. Validate interconnect bandwidth: confirm whether the cards have NVLink-class high-speed interconnect, and assess whether all-to-all communication is the bottleneck.
    4. Evaluate context and concurrency: KV Cache explodes with sequence length and concurrency, so use this to back-calculate the usable batch size and VRAM headroom.
    5. Calculate total cost of ownership (TCO): factor in hardware, power, datacenter, personnel, and depreciation, then compare against elastic cloud options.
    6. Weigh self-hosting vs. cloud: data compliance, sustained high load, and team capability are the deciding factors; when demand fluctuates elastically or operational capability is limited, on-demand cloud is more cost-effective.

    7. Summary

    Deploying a trillion-parameter MoE model is not about the parameters being intimidating; it comes down to three repeatedly validated engineering facts: VRAM is counted against total parameters, quantization must account for scale overhead, and interconnect determines actual throughput. A 2.8-trillion-parameter 4-bit model has a weight floor of around 1.5TB, and once runtime overhead is added it requires a Super Node of 64+ cards with high-speed inter-card interconnect to host. On the cost side, you need to account for the full lifecycle across hardware, power, datacenter, and operations. For most teams, the reliable path to avoid “buying the cards but never running them at full capacity—or not running them at all” is to first use the decision framework above to work out the two thresholds of VRAM and communication, and then decide whether to self-host or leverage the elastic capabilities of a cloud platform.