How Much Does It Cost to Locally Deploy a 100B+ Parameter LLM?

With open-source large language models (LLMs) scaling into the hundreds of billions of parameters, a pressing question for many developers and teams arises: “Can I deploy this on my own hardware, and how much will it cost?” Using an open-source Mixture of Experts (MoE) model with 671B parameters as a reference, this article clarifies capacity estimation, route selection, hardware lists, performance expectations, and decision frameworks in a single comprehensive guide.

1. The Capacity Math: Parameters × Quantization Bit Width

For models at the 100B+ scale, the primary constraint for local deployment is not compute power, but whether the weights can physically fit into memory. The estimation formula is straightforward:

Weight Size ≈ Total Parameters × Bits per Parameter ÷ 8

Q4_K_M is a mixed-precision quantization scheme in the 4-bit tier, averaging about 4.5 bits per parameter. Based on this, we can outline a “memory bill” (using the 671B model as an example):

Quantization Tier	Bits per Parameter	Weight Size
BF16 / FP16	16 bit	≈ 1342 GB
8-bit	8 bit	≈ 671 GB
Q4_K_M	≈ 4.5 bit	≈ 377 GB
3-bit	≈ 3.2 bit	≈ 268 GB
2-bit	≈ 2.6 bit	≈ 218 GB

Three key conclusions:

1. Full-precision BF16 is completely impractical for individuals and small teams. A capacity requirement of over 1.3TB directly exceeds standard single-machine configurations.
2. 4-bit quantization is the cost-effectiveness sweet spot. 377GB is currently the mainstream tier for community deployment of models at this scale.
3. Compressing to 2~3 bits saves an additional 30%~45% of memory, but leads to perceptible quality degradation. This is especially true for multi-step reasoning and long-chain tasks, making it an inadvisable first choice.

Beyond weights, you must also allocate memory for the KV Cache and runtime overhead. Modern models generally use Grouped-Query Attention (GQA) structures, which have significantly compressed the KV Cache. However, running 32K~128K long contexts can still require tens of extra gigabytes. In engineering practice, it is recommended to plan total memory as “Weight Size × 1.1~1.2”. A simple estimation script:

params = 671e9        # Total parameters
bits   = 4.5          # Average bits per parameter for Q4_K_M
weight_gb = params * bits / 8 / 1e9    # ≈ 377 GB
reserve   = 1.2                        # KV Cache + framework overhead margin
print(f"Recommended total memory: {weight_gb * reserve:.0f} GB")
# Output: Recommended total memory: 453 GB

2. Two Technical Routes: Full GPU Cluster vs. CPU+GPU Hybrid Inference

Route A: Full GPU Cluster. Loading 377GB of weights requires at least 8 80GB-class GPUs, not accounting for the KV Cache. Factoring in high-speed interconnects, complete machine design, and redundancy, the investment typically falls in the million-dollar range. This buys high-concurrency, low-latency production-grade serving capabilities, suitable for online businesses serving multiple users.

Route B: CPU+GPU Hybrid Inference (MoE Offloading). The key characteristic of the MoE architecture is that while the total parameters are massive, the parameters actually activated per token are only in the tens of billions. Attention layers and shared parameters are used every time, while expert layers are sparsely activated. This gives rise to a clever divide-and-conquer strategy:

• Attention computation, KV Cache, and hot expert caching → placed on a single 24GB VRAM GPU.
• All expert weights → reside in CPU memory, fed to the CPU for computation via multi-channel memory bandwidth.

Among open-source hybrid inference solutions in the industry, a widely cited public benchmark is: a dual-socket server-grade CPU (approximately 32 cores and 8 memory channels per socket) + a single 24GB VRAM GPU, running the Q4_K_M quantization of a 671B model (~377GB). This setup achieves a long-context prefill speed of over 200 tokens/s and a single-route decode speed of about 14 tokens/s.

It must be emphasized that these numbers come with constraints: they were measured under specific expert selection strategies (where expert routing was constrained or pre-filtered), representing an “upper bound for that strategy” rather than an “average for general scenarios.” The correct approach is to benchmark with your own real business workloads and leave a discount margin from official demo numbers.

3. A Practical Reference Configuration and Cost Estimate

The selection principles for the hybrid inference route differ entirely from typical desktop machines. The four core principles are:

1. For CPUs, prioritize memory channel count over core count. The bottleneck during the decode phase is memory bandwidth. On a dual-socket platform, you must populate all 8 DDR5 channels per socket to max out the bandwidth.
2. Start with 512GB of memory. This covers the 377GB weights plus KV Cache margin. Standard frequency RDIMMs are sufficient; capacity takes priority over extreme frequencies.
3. A 24GB-class GPU is sufficient. It only handles attention, KV Cache, and expert caching, so there is no need to chase high-end flagship models.
4. Do not skimp on storage and power supply. Model files approach 400GB; a 2TB NVMe SSD is the recommended starting point. The machine’s full-load power draw is substantial, so a 2000W-class redundant power supply is more stable.

Cost tier estimation (market prices fluctuate, for reference only):

Component	Budget Tier
Dual-socket server platform (incl. CPU)	$4,000 ~ $7,000
512GB DDR5 RDIMM	$2,000 ~ $3,500
24GB VRAM discrete GPU	$1,500 ~ $2,000
2TB NVMe SSD	$300 ~ $500
Case / PSU / Cooling	~$700
Total	~$8,500 ~ $13,500

Hidden costs must also be accounted for: full-load power consumption is around 1kW, costing hundreds of dollars annually for 24/7 operation; continuous updates to the OS, drivers, and inference services require labor; and hardware depreciation and single-point failure risks should be amortized into the total cost of ownership.

4. Managing Performance Expectations: Fast Prefill ≠ “Fast in Practice”

Prefill and decode are two different things. Prefill is the one-time speed to “ingest” the input context. Over 200 tokens/s means a long document of thousands of words can be encoded in seconds, which is highly favorable for RAG and long-document summarization scenarios. Decode is the speed of generating tokens one by one, directly determining the conversational experience. 14 tokens/s is roughly the speed of a person skim-reading—smooth for single-user, single-route usage. However, once concurrency exceeds 1, throughput degrades almost linearly, which is the ceiling for single-machine hybrid solutions.

The root cause of slow decode is that every token requires reading data from expert weights residing in CPU memory, constrained by memory bandwidth. Viable optimizations include: NUMA affinity binding (keeping the GPU and compute threads on the same CPU socket), populating all memory channels, caching frequently routed “hot experts” into VRAM, and controlling context length based on VRAM budget. Furthermore, for any official benchmark, it is recommended to conduct three types of self-tests: long-text summarization, multi-turn dialogue, and code generation. Record prefill and decode separately to replace lab distributions with real-world distributions.

5. On-Premise vs. Cloud: A Decision Framework

Signals favoring on-premise: Strong compliance requirements mandating data cannot leave the local network; stable and high utilization of model calls by internal tools (amortizing hardware costs over the long term); the need for deep customization and secondary development of the inference pipeline.

Signals favoring the cloud: The project is in the validation phase, and you want to avoid heavy capital expenditure; traffic shows obvious tidal patterns with frequent scaling; the team lacks dedicated operations staff. Cloud services like Baidu AI Cloud provide general capabilities including GPU compute rental, LLM inference services, and managed platforms. These convert one-time procurement expenses into pay-as-you-go operational expenses. For most small and medium teams, “validating business value on the cloud first, then deciding whether to build on-premise” is a more prudent path.

A hybrid strategy is also common: sensitive data and core pipelines remain in a small-scale local deployment, while elastic peaks and external services are handled by the cloud, bridged by a unified API interface.

6. Pre-Launch Operational Checklist

• Ensure all memory channels are fully populated and memory frequency is configured correctly in the BIOS;
• Bind the inference process and GPU to the same NUMA node to avoid cross-socket access;
• Reverse-engineer the context length setting based on “VRAM = model resident portion + KV Cache + expert cache”;
• Use a standard HTTP API format for service encapsulation to facilitate integration with mainstream client ecosystems. Pseudo-code example:

# Pseudo-code example: Typical startup parameters for a hybrid inference service
hybrid-infer serve \
  --model /models/moe-671b-q4km \
  --attn-device gpu:0 \
  --experts-on cpu \
  --expert-cache-gb 12 \
  --ctx-len 32768 \
  --port 8000

7. Conclusion: Three Cost Tiers, Three Answers

Returning to the title’s question, “How much does it cost to locally deploy a 100B+ parameter model?” actually has three tiers of answers:

1. Pay-as-you-go API: Zero startup cost, pay per token, ideal for getting started;
2. CPU+GPU hybrid on-premise: An investment of around $8,500 ~ $13,500 buys a single-route experience of about 14 tokens/s and complete data sovereignty, suitable for single users or small internal teams;
3. Full GPU cluster: A million-dollar investment buys high-concurrency production-grade capabilities, suitable for external services.

When making a decision, simply ask yourself three questions: Must the data stay local? What is the expected concurrency? Can utilization remain high over the long term? Plugging these three answers into the cost framework above turns “how much does it cost and is it worth it” from a mystical question into a clear engineering calculation.