How to Choose Hardware for Local LLM Deployment
Local LLM deployment is less about GPU count and more about the data path from storage to RAM to VRAM. This article covers VRAM capacity and bandwidth estimates, PCIe idle downclocking, memory bandwidth for MoE offload, NUMA, storage, and a practical checklist.
1. The question is no longer “can it run,” but where the config is stuck
When people deploy large models locally, the first instinct is to count GPUs. That matters, but after the machine is built, experience is decided by a less visible path: data moving from storage to RAM, RAM to VRAM, and between VRAM and the chip.
While tuning a local inference setup, two components were suspected:
- The PCIe link between GPU and motherboard. A GPU status tool showed the idle link negotiated to Gen1. Under load it returned to Gen3 x16.
- CPU memory bandwidth. The platform was a dual-socket 2U server with 512 GB DDR4-2400, four memory channels per CPU and two DIMMs per channel. The model was an open-source MoE (Mixture of Experts) model with hundreds of billions of parameters.
The first was a misdiagnosis. The second was the bottleneck. The rest of this article follows estimate, inspect, then verify.
2. Capacity estimates: VRAM is the hard gate
2.1 Can the model fit
VRAM must hold at least three parts:
Total VRAM ≈ model weights + KV cache + inference framework overhead
Weight size is driven by parameter count and precision:
| Weight precision | Bytes per parameter | 7B model | 70B model |
|---|---|---|---|
| FP16 / BF16 | 2 bytes | ~14 GB | ~140 GB |
| INT8 | 1 byte | ~7 GB | ~70 GB |
| 4-bit (INT4/NF4) | ~0.5 bytes | ~3.5 GB | ~35 GB |
Two notes:
- Weight files are usually slightly larger than raw tensors because of embeddings, normalization parameters, and metadata.
- 4-bit weights still need dequantization at compute time, which costs extra VRAM and compute and some accuracy. Speed versus quality is a trade-off.
KV cache grows with context length and concurrency:
KV cache bytes ≈ 2 × layers × KV heads × head dim × sequence length × concurrency × bytes per element
Example: a 32-layer model with about 4096 bytes of KV per token. One 32K context is about 128 MB; 128 concurrent requests balloon to about 16 GB. For long documents or multi-user serving, KV cache often fills VRAM before weights do.
2.2 Will it be fast
Inference is memory-bandwidth bound. In decoding, each generated token requires a full read of the active weights, with low arithmetic intensity. Single-request decode speed can be estimated as:
Theoretical max speed (token/s) ≈ effective VRAM bandwidth (GB/s) ÷ bytes read per decode step (GB)
For a 70B model in 4-bit:
- Each decode step reads about 35 GB.
- A consumer GPU with ~900 GB/s VRAM bandwidth has a theoretical ceiling of ~26 token/s.
- A typical consumer GPU at ~400 GB/s is closer to ~11 token/s.
Framework overhead, sampling, and some compute still reduce real speed. In this regime bandwidth dominates FLOPs: raising core clocks barely helps, while a same-generation card with less VRAM often has much lower bandwidth.
Conclusions:
- Capacity decides whether it can run.
- Bandwidth decides how fast decode is.
- Compute decides the ceiling for long-context prefill and high-concurrency throughput.
Pick GPUs on all three, not core count or clocks alone.
3. Link inspection: idle downclock is not a fault
3.1 Symptom and cause
A status tool once showed a GPU PCIe link at Gen1 (2.5 GT/s), which looked like a seating problem.
Under load the link returned to Gen3 x16 (8 GT/s). Firmware link power saving was at work:
- Idle links downclock or enter low-power states.
- Traffic restores higher generations step by step.
That is a feature, not a bug. Check link state under load, or disable link power saving in firmware if you want the link to stay at full speed.
3.2 How to compute per-generation bandwidth
PCIe single-lane rates are 2.5, 5, 8, 16, and 32 GT/s for Gen1–Gen5. Encoding differs:
- 2.5 and 5 GT/s use 8b/10b (80% efficiency).
- 8 GT/s and above use 128b/130b (~98.46% efficiency).
Effective bandwidth per lane (GB/s) = lane rate (GT/s) × encoding efficiency ÷ 8x16 unidirectional bandwidth = per-lane bandwidth × 16
| Generation | Per-lane effective | x16 unidirectional |
|---|---|---|
| Gen3 | ≈ 0.985 GB/s | ≈ 15.75 GB/s |
| Gen4 | ≈ 1.97 GB/s | ≈ 31.5 GB/s |
| Gen5 | ≈ 3.94 GB/s | ≈ 63.0 GB/s |
PCIe is full duplex; bidirectional totals are about twice the unidirectional figures.
3.3 Is that enough
For MoE:
- When expert layers are offloaded to CPU memory, the GPU only pulls the few selected experts.
- Those shards cross PCIe; gating already limits per-step volume.
- Intermediate copies are typically tens to hundreds of MB.
At a few to tens of tokens per second, Gen3 x16 unidirectional bandwidth far exceeds the transfer need. 15.75 GB/s could theoretically move more than 30 GB of weights per second. The bottleneck is VRAM and DRAM bandwidth, not this “highway.”
3.4 When PCIe really is the bottleneck
- Multi-GPU tensor parallelism: every layer needs inter-GPU exchange. If GPUs talk only over PCIe or across CPU interconnects, without high-bandwidth direct links, decode communication cost rises.
- Pipeline parallelism: less traffic, but only some devices work at once, so throughput is limited and imbalance hurts more.
- Virtualization: passthrough may split a link (x16 into x8+x8) or share uplink with NICs and storage.
- Slots and physical path: chipset-attached slots, risers, vertical mounts, or oxidized contacts can negotiate x8 or x4 even under load.
4. Memory bandwidth: the hidden ceiling for MoE inference
4.1 Why MoE is sensitive to DRAM bandwidth
MoE models contain many expert subnetworks; each token activates only a few:
- Compute is much lower than a dense model of similar size.
- All expert parameters still must reside in VRAM or CPU memory.
If they do not fit in VRAM, experts are offloaded. Each token moves activated experts from DRAM into VRAM and back, so speed is locked to memory bandwidth. Replace VRAM bandwidth with DRAM bandwidth in the token/s formula:
Theoretical max speed (token/s) ≈ effective memory bandwidth (GB/s) ÷ bytes read from memory per step (GB)
If each step reads ~30 GB and measured bandwidth is 120 GB/s, the ceiling is ~4 token/s. Headroom comes from:
- Reading less per step (more aggressive quantization, expert caching).
- Raising measured bandwidth.
- Raising concurrency to amortize fixed overhead and lift total throughput.
4.2 Theoretical vs measured bandwidth
Theoretical memory bandwidth = channels × transfer rate (MT/s) × 8 bytes per channel
Dual-socket example, 4 channels per CPU, 8 channels total:
- DDR4-2400: 8 × 2400 × 8 ÷ 1000 = 153.6 GB/s
- DDR4-3200: 8 × 3200 × 8 ÷ 1000 = 204.8 GB/s
Some single-socket servers now offer 12 DDR5 channels. At 4800 MT/s: 12 × 4800 × 8 ÷ 1000 = 460.8 GB/s.
That is a ceiling. Streaming copies typically reach 60–80% of theoretical due to refresh, timings, and bus turnaround. Inference also has compute, scheduling, and routing, so never convert theoretical bandwidth directly into token/s—measure.
4.3 DIMM population often matters more than faster DIMMs
The platform had two DIMMs per channel:
- Fully populating a channel can make the memory controller drop the operating rate. Rated 2400 may run slower, shrinking theoretical bandwidth.
- Filling all channels usually beats fewer, larger, faster sticks. Eight channels with eight large DIMMs typically beat four sticks with half the channels empty.
General rules:
- Follow the motherboard: populate the first DIMM per channel before the second.
- Keep pairs in a channel matched in capacity, speed, and rank.
- Prefer CPU-direct DIMM slots.
4.4 NUMA: do not take the long path
On dual- or multi-socket systems each CPU has its own memory controller. Local access is fast; remote access goes over the CPU interconnect and loses bandwidth and latency.
Common issues:
- The inference process is scheduled on one socket while memory is allocated on the other.
- Default allocation spreads the model across nodes so every request is remote.
- PCIe devices hang off one CPU while the process runs on the other.
Mitigations:
- Inspect topology with
lscpuandnumactl --hardware. - Bind inference to the CPU that owns the GPU and the high-bandwidth memory.
- Put loading and preprocessing on the other socket.
# Bind inference to NUMA node 0 and prefer local memory# Replace with your engine's launch commandnumactl --cpunodebind=0 --membind=0 <inference engine start command>
4.5 Verify and improve
Use a STREAM-style bandwidth test, comparing all threads on one NUMA node versus per-node local access. If local is clearly faster, remote access is hurting inference.
Structural improvements:
- Use a CPU or platform with more channels per socket.
- Reduce DIMMs per channel so rated frequency is restored.
- Cut bytes per token: more aggressive quantization, keep hot experts in VRAM.
The last is often cheapest: halving bytes per step can double speed, more than adding a few DIMMs.
5. Other parts that must not become the bottleneck
5.1 Storage: cold-start experience
A 70B 4-bit checkpoint is about 35 GB; a hundreds-of-billions-parameter MoE can be hundreds of GB. Sequential read speed sets load time:
- 200 GB at 3 GB/s NVMe: ~67 seconds.
- SATA at 550 MB/s: ~6 minutes.
- HDDs: tens of minutes.
Recommendations:
- Keep model files on NVMe SSDs.
- Stripe multiple disks (RAID 0) for large, frequently used models; accept the redundancy risk.
- With enough RAM, OS page cache can make the second load nearly instant.
- Keep a local copy for offline and internal-network use.
5.2 CPU cores and RAM size
The CPU handles tokenization, sampling, operator scheduling, copies, and dequantization. Do not blindly add cores:
- Single-request decode has little CPU work.
- High concurrency, multi-GPU, and CPU offload need enough cores and PCIe lanes.
Estimate RAM and leave 20–30% headroom:
RAM ≥ resident weights in host memory + peak KV cache + framework and OS overhead
Insufficient RAM causes paging or OOM kills—one of the most common local-deployment failures.
5.3 Power, cooling, and stability
- Size the PSU for peak system draw with margin; prefer higher-efficiency units.
- Keep GPU intake clear; add case fans or skip slots in multi-GPU builds.
- Thermal throttling cuts clocks, compute, and even link stability.
- Dual-socket servers are loud and power-hungry; plan rack, power, and cooling, not a desk.
6. Configuration order: model first, then parts
Do not start by stacking flagship GPUs. Decide in this order:
- Model and precision: estimate weights and KV cache; check if one GPU can hold them.
- Deployment shape: if not, weigh a multi-GPU VRAM pool versus CPU offload, and whether memory bandwidth can hit the target speed.
- VRAM capacity and bandwidth: given capacity, prefer high-bandwidth memory; decode speed tracks it.
- Memory channels and population: fill channels per the manual, prefer first DIMMs, verify with STREAM.
- Storage: NVMe; stripe for huge models.
- System: PSU margin, airflow, NUMA affinity.
After install, inspect:
| Check | Method | Pass criterion |
|---|---|---|
| GPU link | Query under load | Expected generation and width under load |
| Memory bandwidth | Streaming benchmark | ~60–80% of theoretical |
| NUMA affinity | Local vs remote tests | Local clearly better |
| VRAM use | Curves vs context and concurrency | Headroom, no wild swings |
| Sustained performance | Long soak; temps, clocks, latency | No throttle, no timeouts |
Know the limits:
- Consumer GPUs often cannot hold the largest models on one card.
- Offload speed is capped by DRAM bandwidth.
- Multi-GPU without high-bandwidth interconnect scales poorly.
If you need elastic concurrency, faster model iteration, or less hardware ops, cloud GPU instances and containers can be requested and released on demand. For sensitive offline data, isolated networks, or long-term cost control, local still makes sense—match the path to the workload instead of chasing specs.
7. Closing
Local LLM hardware is about moving data fast enough:
- VRAM capacity decides whether the model fits.
- VRAM and DRAM bandwidth decide decode speed.
- Idle PCIe downclock is usually power saving; failure to recover under load is the real problem.
- NUMA affinity and DIMM population often beat buying faster RAM.
Estimate, inspect, then verify—more effective than blindly stacking parts.
