Local Deployment Cost Planning for Large MoE Models

    This article explains why open-source model weights do not make private deployment inexpensive. It breaks down MoE parameter and memory calculations, KV cache pressure, precision trade-offs, long-context and concurrency costs, hardware budget tiers, operational expenses, and self-hosting versus cloud options. Readers will learn how to estimate real memory requirements and avoid common budget surprises when deploying hundred-billion-parameter MoE models locally.

    1. The First Misconception After Open Sourcing: Free Model Does Not Mean Free Deployment

    When teams decide to deploy a hundred-billion-parameter model privately, their first reaction is often: “The model is open source, so the rest is just pulling the code and starting a service.” In engineering terms, that judgment is usually wrong. Model weights are only the starting point of the entire pipeline. What really determines whether the system can run stably is the GPU memory budget, hardware procurement, precision selection, inference framework tuning, and long-term operations and maintenance. Before this account is calculated clearly, the cost illusion created by the word “open source” is often the first step toward losing control of the project budget.

    2. Understand the MoE Parameter Math First

    Most current large-parameter models use a Mixture of Experts (MoE) architecture. Take a typical model in the hundreds-of-billions parameter class as an example: its total parameter count is about 743B, but when generating each token, the routing mechanism activates only about 39B parameters. This “large total parameter count, small activated parameter count” characteristic is the core computational efficiency advantage of MoE—it trades sparse activation for lower per-token computation.

    This easily leads to a misunderstanding: if only 39B parameters are activated per token, is it enough to have memory corresponding to 39B parameters? The answer is far from enough. GPU memory usage depends on “the parts that must remain resident in memory,” not “the parts that actually participate in each computation.” All expert weights, whether or not they are hit by routing for the current token, must be loaded into GPU memory in advance and stand by; otherwise, on-demand activation cannot be achieved. Therefore, the memory budget must be estimated based on the total parameter scale, not the activated parameter scale.

    3. What Actually Consumes GPU Memory

    To accurately estimate deployment costs, GPU memory usage must be broken into three parts:

    1. Model weights: This is the largest part. With FP8 precision, each parameter takes about 1 byte, and 743B parameters correspond to about 743 GB of GPU memory. This item alone far exceeds the capacity of a single GPU, so multi-GPU parallel partitioning is required.
    2. Inference framework and runtime overhead: GPU drivers and compute runtime contexts, memory fragmentation, multi-GPU communication buffers, and various runtime libraries all consume considerable GPU memory. It is usually necessary to reserve a 10%–20% margin beyond the weights.
    3. KV cache: This is the most easily underestimated part. Autoregressive generation needs to cache the Key/Value of historical tokens. Its usage is proportional to “context length × number of concurrent requests × number of layers × hidden dimension,” and it expands approximately linearly with service scale.

    A rough but practical estimation logic is: the maximum context and concurrency a system can support often do not depend on the model weights themselves, but on how much GPU memory remains for KV cache after the weights and runtime overhead are accounted for.

    4. Precision Selection: The Cost of Saving Memory

    Precision directly determines weight memory usage and inference quality. Compared with FP16, FP8 can cut weight memory by nearly half and is an important prerequisite for making hundred-billion-parameter models deployable. But lowering precision is not cost-free: quantization errors may affect output quality in scenarios such as long-text understanding and complex reasoning, so a trade-off must be made between “being able to run” and “running accurately.” A common practice is to first validate functional feasibility at lower precision, then evaluate whether some operators or the entire model need to fall back to higher precision based on business sensitivity to quality—and this directly increases memory and hardware requirements.

    5. One-Million-Token Context: A Hidden Memory Killer

    Native support for million-token contexts is one of the selling points of such models, but its consumption of GPU memory is order-of-magnitude significant. KV cache grows linearly with context length, and the KV cache for a single request with an ultra-long context can consume tens or even hundreds of GB of GPU memory. In other words, whether a model can “support” a one-million-token context and whether it can “serve” a one-million-token context stably under high concurrency are two completely different questions: the former is a capability question, and the latter is a resource question. Running a single request successfully in a demo and supporting multiple users interacting with long texts simultaneously online are completely different cost levels.

    6. The Cost Trap of Concurrency

    Running inference successfully once on a single machine and supporting concurrent requests from a team or even a business line are two different cost models. The higher the concurrency, the more KV cache must remain resident at the same time, and the greater the memory pressure. If the goal is high-concurrency long-context service, it is often not enough to simply add a few GPUs. The system may need to move from a single node to multiple nodes, with high-speed interconnect, tensor parallelism/pipeline parallelism strategies, and request scheduling and batching mechanisms. This is the root of the budget gap between “minimum configuration that can run” and “production-grade capacity that can withstand load.”

    7. Calculate a Realistic Budget

    Taking the above factors together, the budget can be understood in three tiers:

    • Minimum runnable tier: A single machine with 8 high-end accelerators of about 140 GB memory each, running FP8 weights, barely loads the model and serves a small number of requests. Based on rough market estimates for current flagship accelerators, a single 8-GPU server is generally in the range of several million RMB—this is only the “entry ticket.”
    • Full long-context tier: To stably support million-level context with some concurrency, two nodes and 16 GPUs are usually required. The budget doubles to around RMB 7 million, with additional investment in high-speed networking and storage.
    • Production high-availability tier: Redundancy, monitoring, canary releases, failover, and other engineering capabilities are also needed. Labor and operations costs exist over the long term.

    The conclusion is: the model is open source, but bringing the original capabilities into private “production-ready” use means hardware cost is only the starting point. The real investment lies in completing stability, concurrency, and operations capabilities.

    8. Do Not Ignore Costs Beyond Hardware

    Server procurement is only one line item on the books. Long-term costs also include power and cooling expenses caused by high power density, data center floor-load and cooling retrofits, sunk costs from low GPU utilization, and labor for inference framework tuning and operations. For businesses with clearly fluctuating loads or lacking large-scale stable workloads, the risk of resource idling in a self-built cluster deserves particular attention.

    Therefore, self-hosting is not the only option. If business workloads are elastic, time-to-market is sensitive, or the organization does not want to bear tens of millions in capital expenditure at once, it can consider using heterogeneous computing resources from a cloud platform. For example, Baidu AI Cloud provides elastic GPU computing capabilities. Together with general cloud products such as object storage, container platforms, and monitoring and alerting, it can convert heavy asset investment into pay-as-you-go operating expenditure, enable on-demand scaling, and significantly reduce early trial-and-error costs. Self-hosting versus cloud is essentially a trade-off between capital expenditure and operating expenditure, and between certainty and elasticity.

    9. Recommendations for Decision-Makers

    1. Calculate memory first, then discuss performance: Estimate weights using the total parameter count, estimate KV cache using context length and concurrency, add the two together plus margin, and only then do you have the real memory requirement.
    2. Distinguish “can run” from “can withstand load”: The minimum configuration in a demo environment does not represent capacity planning for production.
    3. Treat precision as an adjustable knob: Find the optimal balance acceptable to the business between quality and cost.
    4. Reserve budget for operations: The costs of stability, monitoring, canary releases, and disaster recovery are often easier to miss than hardware.
    5. Evaluate self-hosting and cloud seriously: Choose the computing supply model that matches workload characteristics, capital budget, and launch schedule.

    The threshold for private deployment of hundred-billion-parameter MoE models has never been “whether the open-source model can be downloaded,” but “whether it can serve real business long-term, stably, and concurrently at an acceptable cost.” Only by calculating this account clearly before starting work can teams truly avoid budget surprises such as “several million RMB is just the entry ticket.”