Local Deployment Costs for Large Sparse Mixture-of-Experts Models
Deploying a very large language model locally costs more than its parameter count and GPU count suggest. This article explains how sparse Mixture-of-Experts architectures change the picture. It covers weight storage, KV cache, context length, multi-GPU parallelism, power, cooling, and operations, and it ends with a phased method for estimating total cost of ownership.
When deploying very large language models locally, the most common mistake is to focus only on how many parameters the model has and how many GPUs are needed. In practice, deployment cost depends on much more than model size. It also depends on whether the parameters are sparse, the weight precision, the context window, the number of concurrent requests, the inference speed target, data security requirements, and how the system is operated.
For Mixture-of-Experts (MoE) models, the publicly stated parameter count usually includes the parameters of all expert networks, but a single inference pass activates only a subset of them. As a result, the total parameter count and the compute per forward pass are not equivalent. However, all expert weights still need to be loaded and kept in GPU memory or unified memory. A sparse architecture therefore reduces computational load but does not reduce storage and memory requirements by the same proportion. Cost estimates must account for both the total weight footprint and the active compute per token.
1. Distinguish total parameters, active parameters, and weight size
Assume a model has (P) total parameters, of which an average of (A) parameters are activated per inference, and its weights are stored at (b) bytes per parameter. Considering only the model weights, the theoretical storage requirement can be approximated as:
[
S=P \times b
]
High-precision formats need more bytes per parameter, while low-bit quantization significantly reduces the size per parameter. The actual file size, however, also depends on several other factors:
- quantization granularity and grouping method;
- scaling factors, zero points, and metadata;
- number of weight shards;
- alignment and indexing overhead;
- whether backup weights in other precisions are stored alongside the main weights;
- whether the tokenizer, configuration files, and additional modules must be loaded.
The theoretical calculation is therefore suitable only for a rough first estimate. In engineering deployments, use the total file size after the model files are downloaded or exported as the basis for the storage budget. Reserve at least 15% to 30% additional space for temporary files, caches, logs, and version switching.
Lower-bit weights do not necessarily mean that runtime GPU memory drops by the same proportion. Inference also generates intermediate activations, attention caches, communication buffers, and runtime workspaces. Some operators may use higher compute precision than the weights, so actual GPU memory requirements can exceed the size of the weight files.
2. GPU memory budgeting cannot rely on model file size alone
Local deployment typically needs to satisfy several categories of memory demand at once:
Model weight memory. This holds all or part of the network parameters. A sparse expert model may need to store many expert weights even though only a few experts are active at any given time.
KV cache. In long-context scenarios, the KV cache often becomes a major source of memory use. Longer context windows, more concurrent sessions, and deeper networks all make the cache grow faster.
Activations and temporary tensors. These include intermediate layer outputs, routing results, batch tensors, and operator workspaces.
Communication buffers. In multi-GPU runs, expert parallelism, tensor parallelism, and pipeline parallelism all require data exchange between devices, so space must be reserved for communication.
Runtime headroom. Memory should not stay at 100% utilization for long periods. Otherwise, request fluctuations, changes in sequence length, or memory fragmentation can cause tasks to fail.
A simplified budget formula is:
[
M{\text{total}} \approx M{\text{weight}}+M{\text{cache}}+M{\text{runtime}}+M{\text{communication}}+M{\text{reserve}}
]
The cache term depends heavily on the following factors:
[
M_{\text{cache}} \propto L \times T \times B \times H \times D
]
Here, (L) is the number of layers, (T) is the context length, (B) is the batch or concurrency level, and (H) and (D) are the number of attention heads and the head dimension. Cache layouts differ between models, so this formula is mainly useful for understanding trends. It does not replace actual measurement.
3. Long context windows significantly increase deployment cost
Long context is an important capability for LLM applications, but it adds cost in three ways.
First, cache usage increases. Even if the model weights stay the same, each session can keep occupying GPU memory as the input and conversation history grow. If the system handles multiple parallel requests, cache overhead grows roughly in proportion to concurrency.
Second, scheduling becomes harder. When short and long requests are mixed, memory fragmentation, batching efficiency, and response latency become harder to manage. Simply increasing batch size does not always raise throughput, and it can even cause out-of-memory errors.
Third, testing takes longer. Long context requires validating truncation strategies, cache reclamation, timeout control, and error recovery. In production, it is generally not advisable to allow the maximum context for every request. Instead, set different limits based on the business use case.
A more reasonable approach is to tier context requirements. For example, general Q&A might use a shorter window, code analysis and long-document processing a medium window, and very long windows only when they are truly needed. This balances user experience, memory use, and overall cost.
4. Main cost components of multi-GPU deployment
When a single server cannot hold the model weights, a multi-GPU setup is needed. Multi-GPU deployment usually involves three parallelism strategies:
- Tensor parallelism splits the computation of the same layer across multiple GPUs. It reduces per-GPU memory pressure but requires high inter-device communication bandwidth.
- Pipeline parallelism assigns different layers to different devices. It suits deeper models but can create pipeline bubbles and request scheduling issues.
- Expert parallelism distributes different experts across devices. It fits sparse MoE architectures, but routing and cross-device communication can affect throughput and latency.
As the device count grows, cost does not scale linearly with the number of GPUs alone. Also consider:
- multi-GPU server motherboards and chassis;
- power supply capacity and redundancy design;
- cooling and data center climate control;
- high-bandwidth device interconnects;
- network switching equipment;
- remote management and hardware replacement;
- rack space and power allocation.
For low-concurrency offline tasks, you can accept lower throughput and longer response times, and adopt more aggressive quantization to reduce the device count. For real-time services used by many people, focus on time to first token, sustained generation speed, and peak concurrency. The device count usually increases significantly in this case.
5. Cost estimation should cover four layers
1. One-time hardware cost
One-time investment includes accelerators, servers, memory, solid-state storage, network equipment, and data center facilities. For very large models, do not purchase hardware based only on whether the weights just barely fit. Plan for upgrade headroom and fault tolerance.
If the weights are distributed across multiple GPUs, plan at minimum for how to handle the failure of one device. Deciding whether to reduce service capacity and keep running, or to keep a spare device for fast switchover, directly affects how many units to buy.
2. Storage cost
Local deployment requires at least three types of storage:
- original model weights;
- converted or quantized runtime versions;
- temporary space for downloads, checksum verification, and version rollback.
With sharded files, old and new versions may coexist during an update, so usable space should be clearly larger than the size of a single model copy. Keep model storage separate from logs and caches so that log growth does not interfere with model loading.
3. Power and cooling cost
High-performance computing equipment typically runs continuously, so electricity costs cannot be ignored. Monthly power cost can be roughly estimated as:
[
C{\text{power}}=P{\text{avg}}\times 24\times D\times R
]
Here, (P_{\text{avg}}) is average power draw, (D) is the number of operating days in the month, and (R) is the unit electricity price. In practice, also account for cooling, power conversion losses, and data center infrastructure overhead. If equipment runs at high load for long periods, the cooling system can become a key factor in stability, not just an added expense.
4. Software and operations cost
Local deployment also requires engineering effort for:
- model format conversion and quantization;
- multi-GPU communication configuration;
- inference service packaging;
- request queuing and rate limiting;
- logging, metrics, and tracing;
- model version management;
- access control and auditing;
- failure recovery and capacity planning.
For a personal experiment, software cost may be close to zero, but time cost still exists. Enterprise production deployments should include development, testing, operations, and security auditing in the total budget.
6. Different deployment goals call for different budgets
Personal experimentation
Individual users usually care more about whether the model can run at all than about maximum throughput. They can use low-bit quantization, shorter context windows, and single-user mode, and accept slower generation. For this setup, confirm that:
- the weights are complete;
- the runtime supports the target architecture;
- GPU memory is enough for a minimal request;
- multi-GPU communication works correctly;
- the system stays stable over long runs.
Small team internal service
A
