Local LLM Deployment Costs: From Hardware to Quantization

    This article breaks down the cost of running a 2.8-trillion-parameter Mixture-of-Experts (MoE) model locally. It covers how to define use-case requirements, estimate VRAM, compare three deployment paths, weigh quantization trade-offs, and account for total lifecycle cost.

    More technical teams are evaluating whether to run large language model (LLM) capabilities in local environments. Private deployment keeps data inside the internal network and gives teams more control over service stability and customization. In practice, hardware selection, architecture compatibility, quantization accuracy, and cost accounting come up one after another. This article uses a model with 2.8 trillion parameters and a MoE architecture as a reference to break down the cost structure and key decisions of local deployment.

    1. Define the Use Case First

    Before discussing budget, three metrics need to be fixed: per-user output speed, number of concurrent users, and quantization precision. These three factors set the order of magnitude of hardware investment. The gap between scenarios can exceed one order of magnitude.

    Scenario Target output speed Concurrent users Recommended path
    Feature validation, pipeline testing, teaching demos Not speed-sensitive, as long as output is correct 1 CPU inference or memory mapping
    Researchers tuning prompts, developers integrating apps About 25 tokens per second 1 CPU + GPU heterogeneous inference
    Packaging as an internal service About 25 tokens per second per user 10 or more Multi-GPU cluster or larger scale

    The first tier prioritizes basic functionality. Internal feature validation, data pipeline testing, and teaching demos usually fall here. These scenarios do not depend on token generation speed. The model only needs to process inputs and outputs correctly. Lower-spec hardware is enough, and CPU inference with memory mapping can run the workflow at minimal cost.

    The second tier targets smooth single-user interaction. When researchers tune prompts or developers integrate the model into an application, output needs to reach about 25 tokens per second. That is the speed at which text reads smoothly to a person. CPU inference alone rarely reaches this rate. At least one mid-to-high-end GPU must take part in the computation.

    The third tier is multi-user concurrent service. If the model is packaged as a service that must handle 10 or more concurrent requests while keeping each user at about 25 tokens per second, the difficulty rises sharply. VRAM capacity, memory bandwidth, and inter-GPU communication efficiency all need to be high.

    2. VRAM Estimation: Calculate Weights First, Then Add Overhead

    The model has 2.8 trillion total parameters, including 896 expert modules. Each token activates 16 experts. MoE sparse activation reduces compute but does not reduce VRAM usage. The router decides which experts each token uses, but all 896 experts must stay resident, because any token can be routed to any expert.

    The model was trained with quantization-aware methods. Weights use MXFP4 format, and activations use MXFP8. Each MXFP4 element is 4 bits, but every group of 32 elements shares one 8-bit scaling factor. That makes the average cost per element 4.25 bits. At this rate, the total weight size is 2.8 trillion × 4.25 ÷ 8 bytes, or about 1,487.5 GB. If only the raw 4-bit data is counted, the lower bound is about 1,400 GB. The difference comes from the scaling factors.

    This figure is still incomplete. The following items must be counted separately:

    • Non-4-bit tensors: embedding layers, normalization layers, and similar components usually keep higher precision. Their share depends on the model implementation.
    • KV cache: its size scales with the number of concurrent requests and the context length. Reserving 100 GB to 200 GB brings the total with weights to about 1.6 TB to 1.7 TB.
    • Runtime workspace and tensor-parallel alignment padding: these vary with the inference framework and GPU count, so they cannot be calculated directly from a formula.

    Therefore, 1.6 TB to 1.7 TB is only a lower bound. It excludes non-4-bit tensors, runtime workspace, and alignment padding. Before deployment, measure actual VRAM usage with the target inference framework on the target number of GPUs.

    3. Three Deployment Paths

    1. Full-GPU Supernode: Who It Suits

    All parameters reside in VRAM, and high-speed interconnects such as NVLink handle tensor parallelism and expert parallelism. MoE inference depends on all-to-all communication. After routing, each token’s activations are sent to the GPU that hosts the target expert, and the results are sent back after computation. This communication pattern is highly sensitive to bandwidth, so the full-GPU approach delivers the best performance.

    A single 8-GPU server cannot hold this model. Eight H100 GPUs provide 640 GB in total. Even with 4-bit quantization, the weights far exceed that capacity. The reference supernode in this article uses 64 accelerators with roughly 5 TB or more of combined VRAM. It also requires liquid-cooled racks, a high-speed switching network, and dedicated storage. The system price, networking, liquid cooling, power, and operations each need a separate quote. Total investment typically reaches several million RMB or more. This option suits leading cloud providers and large research institutions. Small and mid-sized companies will find it hard to afford.

    2. CPU + GPU Heterogeneous Inference: How to Choose

    The idea behind heterogeneous inference is to run compute-intensive parts, such as shared attention, on the GPU. Expert parameters take up most of the storage, but each token uses only a few experts. So the expert parameters stay in CPU memory, and the CPU feeds the required experts to the computation.

    The key constraint is memory capacity. The 4-bit weights total about 1.5 TB. To keep all of them resident in memory, the server needs more than 1.6 TB of RAM. If memory is insufficient, memory mapping lets the system read expert parameters from disk on demand. However, speed drops noticeably. Each generated token requires moving the corresponding expert parameters from memory, so the number of memory channels and their frequency directly determine output speed. The match between PCIe bandwidth and CPU compute also sets the performance ceiling.

    On speed: a claim of 25 tokens per second must specify the model version, quantization format, batch size, input and output lengths, CPU memory channels and frequency, GPU model, and the source of the test. Without these conditions, you can only say that the speed may be reached under specific optimizations. This path suits the needs of the first and second tiers. It is the most practical entry point for mid-sized teams.

    3. Small Multi-GPU Cluster: Challenges of Concurrent Workloads

    The typical target for this path is supporting 10 concurrent requests, each at about 25 tokens per second. The 4-bit weights total about 1.5 TB, so the first step is to confirm that VRAM is sufficient.

    Take accelerators with 192 GB each as an example. Eight cards provide 1,536 GB, only about 50 GB more than the weights. That leaves almost no room for KV cache. An 8-card setup can run only if the context length is strictly limited. To support 10 concurrent requests stably with longer contexts, roughly 16 cards, or about 3 TB of VRAM, are usually needed. The exact card count, interconnect, and system pricing depend on the selected hardware model and must be confirmed with vendors. This article does not provide budget figures.

    4. Quantization Precision: Affects Both Cost and Output Quality

    The minimum quantization level should be set by the business’s tolerance for accuracy, not by the hardware budget. Common options include 4-bit weight quantization, such as AWQ- and GPTQ-style methods, and native low-bit formats, such as MXFP4. A natively low-bit model is adapted to low precision during training. It needs no extra dequantization at inference time, so accuracy loss does not accumulate.

    When post-training quantization is applied to a model that was not trained with quantization awareness, activations usually need to keep higher bit widths to avoid numerical overflow. This pushes actual VRAM usage above the theoretical calculation, so measured results should take priority.

    If the business needs high accuracy and high concurrency, consider 8-bit weights. 8-bit weights take about 2.8 TB or more, roughly double the 4-bit size. The number of cards needed on the same hardware doubles as well. Hardware investment should be calculated from quoted prices for each card model. A fixed percentage should not be used as an estimate.

    Accuracy also cannot be measured by VRAM alone. Before going live, cover at least the following validation items:

    • Long-text tasks: check whether answers stay stable