Self-Hosting Open-Weight LLMs: The Real Cost of Deploying a 743B MoE Model Locally

    Open model weights don't mean cheap deployment. Using a 743B-parameter MoE model—39B activated per token and a native 1-million-token context—as a case study, this article breaks down the VRAM, hardware, and total cost of ownership (TCO) for local deployment, from a single-node 8-GPU entry point at roughly RMB 3.5 million to a dual-node 16-GPU setup at RMB 7.3–8 million for high-concurrency, long-context serving. It explains how weights, KV cache, framework runtime, and safety margins consume VRAM, why FP8 is effectively the entry precision, and how context length and concurrency drive GPU count. It also provides a reusable estimation script, five cost-driving variables, a build-versus-cloud utilization analysis, and checklists for cost reduction and pre-launch acceptance.

    Introduction: Open weights don’t mean cheap deployment

    As major models release their weights, many teams form an intuitive assumption: the weights are free to download, so private deployment should be cheap too.

    Take an open-source MoE model with roughly 743B total parameters, about 39B parameters activated per generated token, and native support for a 1-million-token context. Do the math carefully and you’ll find that RMB 3.5 million is only the entry ticket. It buys the minimum configuration that can boot up and produce results—not a service that reliably handles high concurrency with long context.

    This article works through the numbers using a “break down VRAM first, then price the hardware, then look at total cost of ownership” approach. Prices reflect approximate public market rates and are for reference only; actual figures depend on procurement quotes.

    1. Three numbers to understand first: 743B, 39B, and 1M

    Before estimating costs, be clear about what each number controls:

    • 743B (total parameters): determines how much VRAM the weights occupy. This is a fixed cost you pay regardless of concurrency.
    • 39B (parameters activated per token): determines the compute per generated token, affecting throughput and decode speed.
    • 1M (context window): sets the upper bound on KV cache size. The longer the context, the greater the VRAM usage and memory-access pressure.

    In a Mixture-of-Experts (MoE) architecture, each token only passes through the few experts selected by the routing mechanism. For example, if each token is routed to 2 experts on average and each expert has ~19B parameters, the activated amount is 2 × 19B ≈ 39B, about 5% of the total. So “39B activated” must not be read as “only 39B of VRAM is needed.”

    Model property Value Primary impact
    Total parameters ~743B Weight VRAM (fixed cost)
    Activated parameters per token ~39B Compute and decode speed
    Context window Native 1M KV cache size and time-to-first-token
    Weight precision (FP8) ~743GB VRAM and multi-GPU layout
    Weight precision (BF16) ~1.5TB VRAM and server cost

    Note: The table above is an order-of-magnitude estimate; VRAM is calculated with 1GB = 10⁹ bytes.

    2. What actually consumes VRAM: four cost buckets

    A common mistake when sizing a deployment is choosing GPUs based only on model weights—then the service collapses as soon as concurrency kicks in. Actual VRAM usage includes at least four parts:

    1. Model weights: a fixed overhead;
    2. KV cache: grows linearly with “context length × concurrency,” and is the part most likely to spiral out of control;
    3. Inference framework runtime: communication buffers, operator workspaces, memory pools, and so on;
    4. Safety margin: VRAM fragmentation and peak fluctuations; a 10%–15% reserve is recommended.

    The first two determine “the minimum number of GPUs needed”; the last two determine “how many GPUs are actually needed.”

    2.1 Weight quantization precision sets the entry bar

    The same model occupies very different amounts of VRAM at different quantization precisions:

    Weight precision VRAM usage Suitable for local/private deployment?
    FP16/BF16 ~1.5TB High VRAM cost; usually only for accuracy cross-checks
    INT8 ~743GB Optional; verify accuracy loss first
    FP8 ~743GB Balances accuracy and VRAM; the mainstream choice

    In other words, FP8 is effectively the “starting precision” for this class of model. Mainstream inference frameworks typically support generating FP8 quantized weights offline from training or calibration data, then loading them directly at deployment time.

    2.2 KV cache is the real VRAM hog for long context

    The KV cache stores the keys (K) and values (V) of historical tokens in attention, avoiding recomputation for every token. Its VRAM usage can be estimated as:

    1. KV VRAM ≈ layers × 2 × KV heads × head dim × tokens × concurrency × bytes per element / parallelism

    For illustration, assume 90 layers, 8 KV heads, and a head dimension of 128, with the KV cache stored in FP8. That gives roughly 2 × 90 × 8 × 128 × 1B ≈ 0.18MB per token.

    This means:

    • A single 1M-context request needs about 180GB for the KV cache alone—more than the entire VRAM of a 140GB-class GPU;
    • If the KV cache is stored in BF16, a single 1M-context request needs about 360GB;
    • For long-document analysis, code-base understanding, and similar scenarios, the KV cache is often a scarcer resource than the weights themselves.

    2.3 Don’t forget the framework and the margin

    The inference framework needs extra VRAM for request scheduling, operator execution, and inter-GPU communication; fragmentation also appears during real operation. A reserve of 10%–15% on top of the combined weights and KV cache is recommended. These overheads don’t show up on the model card, but they directly affect how much concurrency you can actually run.

    3. Option 1: Single node, 8 GPUs—the RMB 3.5 million entry ticket

    Using GPUs with 140GB VRAM each, 8 cards per node (~1.12TB total VRAM), estimate with FP8 weights:

    Item Value
    Total node VRAM 8 × 140GB ≈ 1120GB
    Weights (split via TP=8) ~93GB per GPU
    Remaining VRAM per GPU ~47GB
    1M-context requests the node can hold ~1–2 (after ~15% reserve)

    “Holding” isn’t the same as “serving.” Assume ~500 TFLOPS of effective compute per GPU and FP8 weights fully resident in VRAM, then run a full prefill over a complete 1M context. A rough estimate (excluding the extra attention compute):

    GPUs Prefill time Single-step decode ceiling (weight memory access only)
    8 ~20 seconds ~650 tokens/s
    16 ~10 seconds ~1300 tokens/s

    Factor in the fact that inference engines can’t saturate peak compute, plus communication synchronization, and real numbers will be noticeably lower. If multiple long-context requests queue up, waiting time stacks on top. Also, 1M is the model’s capability ceiling—whether it can be used stably over the long term in real business depends on concurrency scale and latency requirements.

    Cost breakdown (estimate):

    • GPUs: 8 × ~RMB 400K ≈ RMB 3.2 million;
    • Full node, memory, networking, and other accessories: ~RMB 300K;
    • Total: ~RMB 3.5 million.

    This RMB 3.5 million is enough to get the model running—suitable for small-scale pilots and offline tasks. If the business needs to make service-level commitments to users, it’s only the starting point.

    4. Option 2: Dual node, 16 GPUs—RMB 7.3 million to 8 million class

    To reliably handle high concurrency at 1M context, the VRAM budget works out like this:

    • Weights: ~743GB;
    • Target: carry 6 concurrent 1M-context requests, requiring ~1080GB of KV cache;
    • Combined: ~1.8TB; with a 15% reserve, ~2.1TB of VRAM is needed;
    • In practice, that’s ~16 GPUs of the 140GB class, i.e., two 8-GPU servers forming a dual node.

    Parallel strategy: tensor parallelism within a node (TP=8) and pipeline parallelism or expert parallelism between nodes is recommended. Cross-node tensor parallelism communicates at every decode step, making it highly sensitive to network bandwidth and latency. Going straight for “cross-node tensor parallelism” requires a high-speed RDMA network, which raises the budget significantly.

    Cost breakdown (estimate):

    • GPUs: 16 × ~RMB 400K ≈ RMB 6.4 million;
    • Two full nodes and accessories: ~RMB 800K;
    • High-bandwidth networking (NICs, switches, cables): ~RMB 500K–1 million, depending on parallel strategy;
    • Storage and datacenter accessories: ~RMB 200K–500K;
    • Total: ~RMB 7.3 million–8 million.

    Note that the dual-node approach mainly solves “stability” and “carrying concurrency.” A single request’s decode speed is still constrained by model size, GPU count, and memory bandwidth. For latency-critical workloads, optimize separately with techniques like model distillation and workload tiering—don’t expect to fix it by simply piling on hardware.

    5. It’s not just about GPUs: hidden costs and 5-year TCO

    The budgets above only cover hardware procurement. What enterprises actually pay is the total cost of ownership (TCO). For the dual-node option, the 5-year TCO looks roughly like this:

    Cost item 5-year estimate Notes
    Hardware depreciation RMB 7.3M–8M Depreciated over 5 years, per accounting policy
    Power and cooling RMB 500K–800K ~12kW per node, PUE 1.4–1.5, electricity RMB 0.6–1.0/kWh
    Storage and datacenter RMB 200K–500K Weights ~743GB, plus data, logs, and multi-version backups
    Spare parts and maintenance ~RMB 1M ~3% of hardware cost per year
    Staffing RMB 2M–4M Deployment, tuning, monitoring, and incident handling
    Total (excl. staffing) ~RMB 9M–10.3M ~RMB 1.8M–2.1M per year
    Total (incl. staffing) ~RMB 11M–14.3M Depends on dedicated vs. part-time effort

    Beyond the table, a few easily overlooked costs:

    • Datacenter conditions: the node’s power density and weight are high; ordinary office datacenters may not meet power, load-bearing, and cooling requirements, and retrofits can cost hundreds of thousands of RMB;
    • Power redundancy: production services typically need a UPS or dual power feeds;
    • Failure risk: GPUs, power supplies, NICs, and other components all fail. A single-node setup lacks redundancy—one hardware failure can take down the whole service—while a dual node can back up the other.

    6. A reusable estimation script

    Rather than eyeballing estimates each time, fix the key parameters into a script—change a few numbers and see the corresponding GPU count and budget:

    1. # local_llm_cost.py: Estimate VRAM and hardware budget for self-hosting an ultra-large model (illustrative script)
    2. # All parameters are examples only; replace them with your actual model structure, GPU type, and service targets
    3. import math
    4. def estimate(weights_gb, layers, kv_heads, head_dim, ctx_len, concurrency,
    5. card_mem=140, card_price=40, node_cost=40, cards_per_node=8,
    6. kv_bytes=1, reserve=0.15):
    7. """
    8. weights_gb : total weight VRAM (GB); ~equal to total params in FP8, ~2x in BF16
    9. ctx_len : context length per request (tokens)
    10. concurrency : number of concurrent requests (leave headroom at 1.2~1.5x peak concurrency)
    11. kv_bytes : bytes per element of the KV cache
    12. reserve : runtime overhead and fragmentation reserve ratio
    13. card_price : per-GPU price (in RMB 10K units); node_cost is full-node + accessories price (in RMB 10K units)
    14. """
    15. kv_per_token = 2 * layers * kv_heads * head_dim * kv_bytes
    16. kv_total_gb = kv_per_token * ctx_len * concurrency / 1e9
    17. required_gb = (weights_gb + kv_total_gb) / (1 - reserve)
    18. cards = math.ceil(required_gb / card_mem)
    19. cards = max(cards, cards_per_node) # at least one full node
    20. cards = math.ceil(cards / cards_per_node) * cards_per_node
    21. nodes = cards // cards_per_node
    22. return {
    23. "KV cache (GB)": round(kv_total_gb, 1),
    24. "Required GPUs": cards,
    25. "Nodes": nodes,
    26. "Hardware budget (RMB 10K)": cards * card_price + nodes * node_cost,
    27. }
    28. if __name__ == "__main__":
    29. result = estimate(
    30. weights_gb=743, # example: FP8 weights ~743GB
    31. layers=90, kv_heads=8, head_dim=128,
    32. ctx_len=1_000_000, concurrency=6,
    33. )
    34. for k, v in result.items():
    35. print(f"{k}: {v}")

    Run this script, swap ctx_len and concurrency for your own business targets, and you’ll quickly get the matching GPU count and budget—avoiding “buying cards on a hunch.” You can also reduce kv_per_token through KV cache quantization, context compression, and similar techniques; that’s the main lever for later optimization.

    7. Five variables that drive cost

    1. Weight and KV precision: FP8 vs. BF16 weights directly halves weight VRAM; KV cache quantization also opens up multiples of concurrency headroom, provided it’s backed by accuracy evaluation;
    2. Context length and concurrency: together they determine KV cache size, and are the main reason the GPU count jumps from 8 to 16;
    3. Interconnect and parallel strategy: cross-node tensor parallelism demands a lot of the network; the wrong parallel strategy can significantly inflate the budget;
    4. Business peaks and valleys: if peaks occupy only a few hours a day, provisioning to peak 24/7 wastes a lot of capacity;
    5. Power and datacenter: differences in electricity price and PUE can amount to hundreds of thousands of RMB over a five-year period.

    8. Build vs. cloud: it comes down to utilization

    If keeping data on-premises isn’t a hard requirement, or if workloads have clear peaks and valleys, on-demand cloud usage is often the more pragmatic choice. On a cloud platform (Baidu AI Cloud, for example), these ultra-large models can be provisioned on demand via GPU cloud servers or heterogeneous compute instances, deployed with container services, with object storage managing the weight files and logging and monitoring services tracking operation. For specific specs, regional availability, and billing, refer to the provider’s official documentation.

    A rough comparison (for order-of-magnitude reference only): assume RMB 400K per GPU over five years of depreciation—averaged out, the hardware portion is about RMB 876 per GPU per day. In other words, as long as a GPU’s daily cloud rental cost is clearly below this figure, the cloud beats self-hosting even at 24/7 full load—and the lower the load, the wider the gap. During supply shortages, cloud prices may also temporarily exceed self-hosting.

    A more sensible approach is to layer workloads:

    • Always-on workloads: core business with strict data-compliance requirements and a need for long-term stability suits self-hosting;
    • Elastic workloads: testing and validation, event peaks, and temporary long-context tasks suit on-demand cloud.

    Mixing the two controls fixed investment while preserving elasticity.

    9. Cost-reduction checklist: save tens to hundreds of thousands of RMB

    • Weight quantization: use FP8 or INT8, and run before/after accuracy comparisons;
    • KV cache quantization: compress the KV cache within acceptable accuracy to directly boost concurrency capacity;
    • Context tiering: retrieve or summarize ultra-long context first, and keep the full KV cache only for the necessary segments;
    • Prefix cache reuse: reuse the KV cache for repeated prefixes (system prompts, knowledge bases) to cut redundant prefill cost;
    • Request scheduling optimization: use continuous batching, dynamic batching, and similar strategies to raise GPU utilization;
    • Distillation for latency-sensitive workloads: use a larger model to generate data and train a smaller model to handle high-frequency requests.

    10. Pre-launch acceptance checklist

    • Function and accuracy: compare before/after quantization on the same set of business samples and confirm the loss is acceptable;
    • Performance: stress-test both “short input, short output” and “long input, short output” scenarios, watching time-to-first-token (TTFT) and time-per-output-token (TPOT);
    • Stability: run sustained stress tests for hours to days, watching whether VRAM and memory keep growing;
    • Failure drills: simulate single-GPU anomalies, process crashes, and so on, and verify that recovery time meets the service-level objective (SLO);
    • Cost accounting: measure daily token volume per the real business distribution, convert it to per-token cost, and validate the budget assumptions.

    Conclusion: Open source is just the start; serving is where the cost lives

    Back to the opening question: how much does it cost to self-host an MoE model with ~743B parameters (39B activated) and 1M context?

    • FP8 single-node, 8 GPUs: ~RMB 3.5 million—it runs, but can’t reliably serve; essentially the entry ticket;
    • FP8 dual-node, 16 GPUs: RMB 7.3M–8M—needed to have real high-concurrency capacity at 1M context;
    • 5-year TCO: ~RMB 11M–14.3M for the dual-node setup, with staffing the largest and most easily underestimated part.

    “Open source” lowers the adoption barrier, not the serving cost. What really determines this bill is context length, concurrency scale, quantization strategy, and utilization. Quantify these clearly and work through them with the estimation script, and you’ll avoid the awkward situation of “bought the cards but can’t serve well, and only discovering you miscalculated after going live.”