How to Calculate the Real Cost of Self-Hosting Large MoE Models

    Self-hosting an open-weight Mixture-of-Experts (MoE) model involves far more than downloading the weights. This article breaks down the real cost of running a large MoE model (roughly 743B total parameters, ~39B activated per token, and native support for ~1M-token contexts) on your own hardware. It explains why VRAM—not compute—is the dominant cost factor, how weights, framework overhead, and KV Cache consume memory, and how these scale with precision, context length, and concurrency. It then compares two hardware configurations, itemizes the full bill that goes far beyond GPUs, and offers a decision framework for choosing between building in-house and renting cloud compute.

    1. First, a common misconception: open source ≠ zero-cost private deployment

    Many teams assume that once a model releases open weights, all you have to do is download it and run it. In reality, truly bringing a large MoE model in-house means the weight download is only the first step. After that comes hardware selection, VRAM planning, inference framework tuning, long-context and concurrency capacity evaluation, and long-term operations—a whole chain of engineering problems. Think of access to the weights as an “entry ticket”: it gets you on the field, but you’re still a long way from stable, production-grade serving.

    This article uses a next-generation MoE model as its sample: roughly 743B total parameters, a mixture-of-experts structure that activates only about 39B parameters per token, and native support for ultra-long contexts on the order of 1M tokens. These characteristics determine its VRAM footprint and form the basis for all the cost estimates below. For easy comparison, GPUs are referred to throughout as “flagship AI accelerators,” using around 140GB of memory per card as a representative configuration.

    2. Understanding MoE: total parameters determine whether it “fits,” active parameters determine whether it “runs fast”

    The defining trait of a Mixture-of-Experts (MoE) model is “large total parameters, small per-token activation.” 743B total parameters means every weight must stay resident in VRAM to be routed on demand—an incompressible, hard cost. Activating only about 39B parameters per token, however, means compute (FLOPs) consumption and inference latency are far lower than those of a dense model with the same total parameter count.

    Two frequently confused concepts are worth separating:

    • VRAM usage is determined by total parameters (743B)—no matter how few you activate, all expert weights must sit in memory, ready on demand.
    • Compute throughput is determined by active parameters (~39B)—this governs token generation speed and per-unit compute cost.

    The conclusion is straightforward: MoE trades “large VRAM + small compute” for cost-efficient inference, but it also concentrates the cost pressure on VRAM. So the cost of self-hosting is, in essence, a VRAM bill.

    3. What actually consumes VRAM: weights, framework, and KV Cache

    Self-hosting uses more than just the model weights—at least three things:

    1. Model weights: A deterministic, estimable cost. VRAM usage at different precisions is roughly:

      • FP8 (8-bit float, 1 byte/parameter): ~743B × 1B ≈ 743 GB
      • BF16/FP16 (16-bit, 2 bytes/parameter): ~1.5 TB
      • INT4 (0.5 byte/parameter): ~372 GB

      As you can see, weight size is almost entirely determined by precision, which is why quantization is the first choice for cutting costs.

    2. Framework and runtime overhead: CUDA context, the inference framework’s memory pool, temporary activation buffers, communication buffers, and so on—typically requiring 10%–20% of the weight size in reserve. This is often underestimated and can amount to over 100 GB on very large models.

    3. KV Cache: The “invisible money pit” in long-context scenarios, and the key factor in determining concurrency.

      The size of the KV Cache can be estimated with a general formula:

      1. KV Cache memory ≈ 2 × num_layers × num_KV_heads × head_dim × sequence_length × concurrency × bytes_per_element

      (The factor of 2 accounts for the two caches—Key and Value; if the model uses GQA/MQA grouped-query attention, the number of KV heads drops significantly, and the cache shrinks accordingly.)

      Key takeaway: The KV Cache grows linearly with sequence length and the number of concurrent requests. At a 1M context, a single long request’s KV Cache alone can consume several hundred GB; if you also need to support multiple concurrent streams, its overhead quickly exceeds the model weights themselves.

    Illustrative estimate (values vary with the specific architecture and only show the order of magnitude): On a single node with 8 cards and about 1120GB total memory, after subtracting ~743GB of weights and over 100GB of framework overhead, only about 200–300GB remains for the KV Cache. A single request approaching 1M tokens could already fill that margin. In other words, “being able to load the model” and “being able to serve ultra-long contexts at high concurrency” are two entirely different thresholds.

    4. Two configurations: from “just runs” to “handles production”

    Based on the VRAM math above, self-hosting naturally splits into two tiers:

    Tier 1: FP8, single node, 8 cards (entry / validation grade)

    • Goal: Load the full model and get it running, supporting moderate concurrency or shorter contexts.
    • Limitation: At 1M context with high concurrency, the KV Cache margin is insufficient, easily triggering out-of-memory errors or forcing you to limit concurrency.
    • Reference cost: Around RMB 3.5 million for a complete machine (8 flagship accelerators + server + high-speed inter-card interconnect + supporting CPU/memory/storage).

    Tier 2: FP8, dual node, 16 cards (stable production grade)

    • Goal: Stably support the full 1M context with some concurrency capacity.
    • Approach: Use tensor parallelism / expert parallelism to split weights and KV Cache across nodes, trading double the VRAM for capacity and redundancy.
    • Reference cost: Around RMB 7.3–8 million, including dual-node compute, high-speed inter-node networking (interconnect gear sensitive to latency and bandwidth), and the corresponding racks and supporting infrastructure.

    Note that these figures are rough estimates based on current market conditions and will fluctuate with accelerator supply, memory capacity, interconnect choices, and purchase volume. They’re meant only to establish a sense of scale, not as precise quotes.

    5. Breaking the cost all the way down: a machine is more than “the cost of GPUs”

    When you actually budget, hardware procurement is only the tip of the iceberg. A complete self-build cost list includes at least:

    • Compute hardware: Accelerators, servers, CPU/memory/system disks, and data disks (model weights alone can run from hundreds of GB to over 1TB).
    • High-speed interconnect: In multi-card, multi-node setups, bandwidth between cards and nodes directly determines parallel efficiency—a component that is often overlooked but not cheap.
    • Power and cooling: A single flagship accelerator server can draw on the order of 8–10kW; the electricity bill for 24/7 operation and the data center’s cooling capacity both need to be budgeted in advance.
    • Racks and data center: Cabinet space, UPS, power distribution, load-bearing capacity, fire safety, and other infrastructure.
    • Operations headcount: Model deployment, quantization tuning, monitoring and alerting, and incident response all require an engineering team with large-model inference experience.

    6. Don’t forget hidden costs: RMB 3.5 million is just the entry ticket

    Once you tally everything, a gap of orders of magnitude emerges between the entry-level “just loads” configuration and actual production investment:

    • Utilization cost: If the accelerators only occasionally run and lack a steady business load, they sit idle most of the time, and the per-inference cost becomes astonishingly high.
    • Iteration cost: Model version upgrades, re-quantization and validation, and context-capability expansion all bring repeated investment.
    • Headcount cost: Large-model inference engineering is a scarce skill, and the long-term cost of maintaining a team is often comparable to hardware depreciation.
    • Opportunity cost: Self-building requires a full cycle of procurement, delivery, racking, and debugging—and the business waiting during that period carries a cost too.

    7. Build or rent: a decision table

    For most teams, fully building in-house isn’t the only answer. You can split workloads by their characteristics:

    Dimension Better suited to self-hosting Better suited to renting cloud compute
    Workload profile Stable, predictable, high-intensity 24/7 Volatile, intermittent, with clear peaks
    Data compliance Strong private-deployment / data-cannot-leave-domain requirements Within an acceptable compliance boundary
    Cash flow Ample one-time capital expenditure Prefer pay-as-you-go, asset-light
    Team capability Have a large-model operations team Want a quick start

    If you lean toward the cloud, platforms like Baidu AI Cloud generally offer general-purpose capabilities such as GPU cloud servers, container platforms, object storage, and logging and monitoring/alerting, letting you obtain compute elastically on demand and turn the burden of heavy-asset procurement and operations into manageable operating expenses. Treat this only as a general understanding of the product categories; specific specs and pricing are subject to the official documentation.

    8. A practical checklist for decision makers

    Before making the final call, answer the following questions first, then decide on a configuration tier:

    1. Which precision: Can you accept the accuracy loss from quantization? Does FP8 meet your business requirements?
    2. How large a context: What context length does the business actually need? Is 1M really necessary?
    3. How much concurrency: The number of simultaneous long requests at peak directly determines KV Cache needs and card count.
    4. Can quantization cut costs: As long as accuracy is acceptable, quantizing to a lower precision can significantly reduce weight VRAM.
    5. Do you need elasticity: Is the load stable? If it’s volatile, a hybrid of “self-built baseline + cloud elasticity” is often more cost-effective.

    In one sentence: “Open source” for MoE models lowers the barrier to access, but not the barrier to serving. What truly determines the cost of self-hosting is the VRAM structure, long-context concurrency needs, and long-term hidden expenses like power and operations. Only after you’ve worked out this VRAM bill and total cost bill can you sensibly discuss whether it’s RMB 3.5 million or RMB 8 million—without paying tuition beyond the “entry ticket.”