How Much Does It Cost to Run a 100B-Parameter LLM Locally?

Introduction: Two Questions That Need Straight Answers

“Running a 100B-parameter LLM on my own hardware” is a common goal for developers and enterprise teams alike. The appeal is obvious: data never leaves the internal network, there are no usage-based fees, and the model can be freely modified. But before turning that goal into reality, two questions must be answered: What kind of machine can actually run it? And how much does that machine cost?

This article uses a typical 671B total-parameter MoE (Mixture of Experts) architecture — one that activates only about 37 billion parameters per token — as a concrete example to work through both questions in detail.

1. Storage First: How Big Is the Model, Really?

The first principle of LLM deployment is simple: weight volume ≈ parameter count × bytes per parameter.

• FP16/BF16 (training precision): 2 bytes per parameter, so 671B parameters require about 1.3 TB.
• INT8: 1 byte per parameter, roughly 670 GB.
• 4-bit quantization: approximately 0.5–0.6 bytes per parameter (including scaling factors and other overhead), putting the weight files at around 370–400 GB. A commonly cited 4-bit format in public tutorials measures roughly 377 GB.

Beyond the weights, two additional categories of storage overhead must be reserved:

1. KV Cache: scales with context length and concurrency. Long-context scenarios can consume anywhere from several GB to tens of GB.
2. Runtime overhead: activations, framework memory usage, and related allocations.

This leads to the first hardware rule: system RAM should be at least 1.2× the weight volume, with 512 GB as a practical starting point. You also need at least one GPU with 24 GB of VRAM to handle attention computation and KV Cache.

2. Why CPU-GPU Hybrid Inference?

A 24 GB GPU cannot hold 370 GB of weights. A pure-GPU approach would require over 640 GB of aggregate VRAM — for example, eight 80 GB data-center GPUs — pushing the machine into the million-dollar range. The MoE architecture, however, offers a third path:

An MoE model has an enormous total parameter count, but only a small subset of parameters is activated per token. The parts used on every token — attention layers, shared components, routing, and embeddings — can stay resident in GPU memory. The sparsely used expert weight matrices can live in large system RAM, loaded on demand according to token routing. This is the principle behind the CPU-GPU hybrid inference approach widely adopted in public tutorials.

Decode Speed Is Bottlenecked by Memory Bandwidth

During decoding, every generated token requires reading all activated weights from memory. This gives rise to a classic estimation formula:

decode speed ceiling ≈ effective memory bandwidth ÷ bytes of weights read per token

For our example: 37B activated parameters × roughly 0.58 bytes ≈ 21 GB per token. A dual-socket server platform with all 24 DDR5 memory channels populated has a theoretical aggregate bandwidth of about 900 GB/s. Real-world effective bandwidth typically runs at 60–70% of that, or roughly 550–650 GB/s. Then 21 GB ÷ 600 GB/s ≈ 35 ms per token, giving a theoretical ceiling of about 28 tokens/s. After accounting for scheduling, kernel overhead, and attention computation, public benchmarks report roughly 14 tokens/s — the same order of magnitude and internally consistent with the estimate.

Public Benchmark Results and an Important Caveat

Using a dual-socket server-class CPU (all memory channels populated, sufficient capacity for the full quantized weights) plus a single 24 GB flagship GPU, a public tutorial ran the 671B model with 4-bit quantization and reported: prefill at over 200 tokens/s, decode at roughly 14 tokens/s.

It must be emphasized that these results come with a caveat: the setup limits the number of activated experts per layer rather than strictly following the original routing for all experts. This is effectively a slight trade-off in generation quality for faster decoding. If you require fully native routing behavior, expect lower speeds. Anyone reproducing this setup should also run sampled quality comparisons against the baseline.

What 14 tokens/s Actually Feels Like

That is roughly 10–20 Chinese characters per second, slightly above average human reading speed. For a single user in conversational use, it feels smooth. Long-form writing requires some waiting. Multi-user concurrency is untenable — multiple requests divide those dozen-or-so tokens/s, and with tens of thousands of tokens of context, time-to-first-token during prefill can stretch into minutes. The correct positioning for this setup is a single-user workstation for individuals or small teams, not an online service.

3. Cost Breakdown: What a Machine That Can Run It Actually Costs

Based on typical e-commerce pricing (prices fluctuate; these are order-of-magnitude estimates):

Component	Description	Estimated Cost (USD)
GPU	24 GB flagship consumer graphics card ×1	$1,800–$2,500
CPU + motherboard	Dual-socket server-class CPU and matching motherboard	$3,500–$7,000
Memory	64 GB DDR5 RDIMM ×8 (512 GB total)	$2,100–$3,500
Storage	2 TB NVMe SSD for weights and quantized artifacts	$150–$300
PSU / chassis / cooling	1 kW+ power supply, tower case, air cooling	$400–$700

Total system cost: approximately $7,000–$14,000, with a typical build landing around $8,500–$11,000. By comparison, a pure-GPU multi-card setup that can hold the entire model often approaches seven figures — which is precisely where the hybrid approach delivers its value.

Hidden costs are equally important:

1. Electricity: a fully loaded system draws about 1 kW. Running 24/7 means roughly 720 kWh per month. At $0.10–$0.30 per kWh, that is $70–$220 per month.
2. Noise and heat: server-class platforms are loud and run hot. Placement matters.
3. Maintenance and troubleshooting: large-memory platform compatibility tuning, system optimization, and crash recovery all consume time.
4. Depreciation: amortizing a $10,000 system over 4 years comes to about $210 per month.

All in, keeping this machine running costs roughly $280–$430 per month in electricity plus depreciation.

4. Three Alternatives: You May Not Need to Buy Hardware

1. Go smaller: if your use case only requires a 70B dense model or a several-hundred-billion-parameter MoE, 4-bit quantization brings weights down to 40–130 GB. A machine with a single 24 GB GPU and 128 GB RAM can be built for $2,000–$3,000. The capability gap is acceptable for many scenarios.
2. Rent on demand in the cloud: for short-term experiments or burst capacity, rent a large-memory or GPU instance and release it when done. On-demand costs typically run from a few hundred to a few thousand dollars per month, making this a good fit for teams with irregular usage.
3. Use a hosted API: managed LLM services charge per token, require no infrastructure maintenance, and scale elastically. For applications with moderate daily call volumes, API costs are almost always lower than self-hosting.

Decision guide: data must be fully offline and usage is intensive → build a hybrid inference workstation. Occasional use, mainly for evaluation → rent on demand in the cloud. Business-facing delivery with concurrency and stability requirements → use a hosted API or managed service.

5. Implementation Checklist: Six Steps from Zero to Running

1. Define the workload profile: single-user interactive or batch offline? How long is the context? What level of concurrency? These determine RAM, VRAM, and whether multi-GPU is needed.
2. Hardware selection: RAM ≥ 1.2× weight volume, populate all memory channels to maximize bandwidth, and prioritize ECC memory and server-class stability.
3. Obtain model and quantized weights: choose a well-maintained 4-bit quantized format and verify file hashes and integrity.
4. Configure hybrid inference: the core idea is attention on GPU, expert weights in RAM. A sample configuration (pseudocode; see your framework’s documentation for exact parameters):

model: moe-671b-4bit        # 4-bit quantized weights, ~377 GB
device_map:
  attention: gpu            # attention layers and KV Cache in VRAM
  experts: cpu              # sparse expert weights in system RAM
experts_per_token: 6        # limited active experts: speed/quality trade-off
context_length: 8192

1. Benchmark: record prefill and decode speeds, time-to-first-token for long contexts, and concurrent throughput. Compare output quality against native full-expert activation behavior with sampled evaluations.
2. Stability and operations: monitor temperatures and memory usage, configure automatic restart on power failure and log rotation, and keep off-site backups of quantized artifacts.

6. Conclusion

Back to the original question: locally deploying a 671B-parameter MoE LLM requires an entry-level investment of roughly $8,500–$11,000 for a hybrid inference workstation (512 GB RAM + 24 GB GPU + dual-socket server platform), plus about $280–$430 per month in electricity and depreciation. In return, you get approximately 14 tokens/s of single-user decode speed and over 200 tokens/s of prefill throughput — with the important caveat that these numbers depend on limiting the number of activated experts.

This setup suits individuals and teams who need fully offline data, have intensive usage, and can accept single-user speeds. For everyone else, scaling down to a smaller model, renting cloud capacity on demand, or using a managed LLM service is often the more cost-effective choice. Before spending money on deployment, do the math first — that is the most important step.