How to Build a Local LLM Machine: VRAM Estimation and Quantization Choices
This guide walks you through planning a local large language model (LLM) setup, starting with VRAM and memory estimation, then comparing two practical configurations: a pure-CPU machine with 64GB of RAM and a desktop with a 16GB GPU. It covers how to calculate weight memory, KV cache, and framework overhead, explains quantization trade-offs (FP16, INT8, INT4), evaluates dense vs. MoE models, provides rough token/s speed estimates based on memory bandwidth, and includes troubleshooting tips for out-of-VRAM errors, slow inference, and quality degradation after quantization, plus a pre-purchase checklist.
1. Estimate Memory First, Then Pick Hardware
When running LLMs locally, the model weights must fit into memory that the compute device can access directly. For pure GPU inference, weights need to reside in VRAM; for pure CPU inference, they live in system RAM. Capacity therefore determines which model sizes are possible at all, while memory bandwidth and compute determine generation speed.
Weight memory ≈ parameter count (B) × bytes per parameter
Typical bytes per parameter: FP16 ≈ 2 bytes, INT8 ≈ 1 byte, INT4 ≈ 0.5 bytes. Real INT4 model files also contain grouped scale factors, so 0.55–0.6 bytes per parameter is a more realistic estimate.
Beyond weights, account for two additional sources of overhead:
- KV cache: stores attention keys and values for previous tokens; it grows with context length and concurrent sessions.
- Framework and driver overhead: the inference engine and CUDA runtime themselves consume some VRAM.
Actual usage ≈ weights + KV cache + framework overhead. Leave a 20–30% buffer. Desktop systems and display output also consume VRAM, so the usable memory on a 16GB card is usually below the nominal value.
KV cache can be estimated with:
KV Cache ≈ 2 × layers × KV heads × head dim × context length × concurrency × bytes per element
The factor of 2 accounts for both the K and V caches. Take a 13B-class model without grouped-query attention (GQA): 40 layers, 40 KV heads, 128 dimensions per head, FP16 cache at 2 bytes per element. Each token occupies roughly 2 × 40 × 40 × 128 × 2 bytes, or about 0.8MB. With a 4k context and a single session, the cache is about 3.1GiB. Double the context and the cache doubles. Models using GQA have far fewer KV heads and use much less cache at the same context length, so always check the specific model’s config file.
2. Pure-CPU Setup: RAM Capacity Sets the Ceiling, Memory Bandwidth Sets the Speed
A pure-CPU setup requires no discrete GPU; the entire model lives in system RAM. Model size is limited only by capacity, at the cost of speed constrained by memory bandwidth.
Rough capacity guidelines: 32GB RAM suits 7B–13B-class INT4 models. 64GB can hold the weights of an INT4-quantized 30B-class model (about 16–18GB) while leaving ample room for context.
MoE (mixture-of-experts) models are a good fit for this approach. An MoE model contains multiple expert subnetworks, and each token activates only a subset. With 30B total parameters and 3B activated, for example, the activated parameters determine per-token compute and the amount of weights that must be read. However, all expert weights must stay resident in memory, so capacity is estimated from total parameters.
The theoretical speed ceiling can be approximated as:
Theoretical ceiling (tokens/s) ≈ effective memory bandwidth (GB/s) ÷ weight bytes read per token (GB)
Dual-channel DDR4-3200 has a theoretical bandwidth of 3200 MT/s × 8 bytes × 2 channels ≈ 51.2GB/s. This is the peak; achievable bandwidth is usually lower.
- A dense 30B INT4 model reads about 17GB of weights per token, giving a theoretical ceiling of roughly 51.2 ÷ 17 ≈ 3 tokens/s.
- A 30B-total / 3B-active INT4 MoE model reads about 1.7GB of weights per token, giving a ceiling of roughly 30 tokens/s.
These are theoretical ceilings only. Real-world speed will be lower and depends on the inference engine implementation, thread count, memory channels, and memory frequency. Always validate with your own benchmarks.
Total cost for a pure-CPU build depends on the prices of the CPU, motherboard, and RAM, as well as whether you buy used parts. This article does not list specific prices—check current local pricing before purchasing.
This approach suits offline batch jobs, long-document summarization, and other tasks that don’t require real-time responses. Submit the job in the background and wait for the result.
3. 16GB GPU Setup: What You Can Run Depends on Weights, Context, and Quantization
The advantage of a 16GB GPU is its much higher memory bandwidth compared to system RAM. At 300GB/s of VRAM bandwidth, reading 7.5GB of weights per token (a 13B INT4 model, for example) yields a theoretical ceiling of roughly 300 ÷ 7.5 ≈ 40 tokens/s. VRAM bandwidth varies widely between cards—always check the specs of the specific model.
Using the estimation method above, a 16GB card roughly supports:
- 7B–8B INT4 (~4–5GB weights): plenty of headroom; 4k–16k context usually runs fine. GQA models can go even longer.
- 7B FP16 (~14GB weights): weights nearly fill the card; only very short contexts work and out-of-VRAM crashes are likely. Not recommended.
- 13B–14B INT4 (~7–9GB weights): with 4k context and one session, total usage is about 9–12GB, fitting entirely in VRAM. Pushing context beyond 8k significantly grows the cache and eats into the remaining headroom.
- 30B-class INT4 (~16–18GB weights): exceeds 16GB of VRAM and cannot fit entirely; some layers must be offloaded to system RAM. It can run, but speed drops noticeably.
Offloaded speed can be understood as follows. Every token passes through all layers sequentially, so latency is approximately: (bytes of layers in VRAM ÷ VRAM bandwidth) + (bytes of layers in RAM ÷ RAM bandwidth). The more layers sit in system memory, the closer performance gets to a pure-CPU setup.
Recommended configuration checklist:
- GPU: 16GB of VRAM; pay attention to memory bandwidth, cooling, and power draw.
- System RAM: 32GB or more, to load model files and leave room for offloading and the OS.
- Storage: model files range from a few GB to over 20GB—reserve enough SSD space.
- Power supply: size it according to the GPU vendor’s wattage recommendation, with headroom.
- Inference engine: common options include llama.cpp, Ollama, and vLLM. Choose based on support for the quantization formats and GPU backends you need.
4. Comparing the Two Approaches
| Dimension | Pure CPU (64GB RAM) | 16GB GPU (32GB+ RAM) |
|---|---|---|
| Main bottleneck | Memory bandwidth | VRAM capacity; beyond that, reliance on RAM offload |
| Runnable models | 30B-class INT4 and below; more RAM widens the range | Comfortable with 7B–14B INT4; 30B-class needs offload |
| Speed magnitude | Low for dense large models; MoE models can be faster | Significantly faster for small and mid-size models |
| Best suited for | Offline batch jobs, long-document summarization | Interactive chat, tasks needing real-time responses |
| Cost structure | Depends on CPU, RAM, and the used market | The GPU is usually the dominant expense—budget separately |
5. Troubleshooting Common Issues
Out of VRAM
- Re-estimate with the formula: weights + KV cache + framework overhead, and check whether it exceeds 70–80% of actually available VRAM.
- Shorten the context length or reduce concurrency.
- Switch to a lower-precision quantization, e.g., from Q8 to Q4. Lower precision can degrade answer quality—verify this as well.
- Close other programs using VRAM, such as browser hardware acceleration and games.
- Check whether large batch sizes or high concurrency settings were accidentally enabled.
Slow Inference
- Confirm all model layers are on the GPU. Inference logs usually show the number of offloaded layers; even a few layers on CPU noticeably reduce speed.
- Confirm the engine is actually using the GPU backend rather than silently falling back to CPU.
- Check temperature and power. Thermal throttling or a power limit will lower clock speeds.
- On a pure-CPU setup, make sure the thread count matches the physical core count, confirm the RAM is running in dual-channel mode, and verify XMP or EXPO is enabled in BIOS so memory runs at its rated frequency.
- Shorten the context and see whether speed recovers, to determine whether cache reads are the bottleneck.
Degraded Answers After Quantization
- Compare the original precision (FP16) and the quantized version on the same set of questions, record the differences, and judge whether they’re acceptable.
- Smaller models are more sensitive to quantization; quality drops are usually more pronounced at the same precision.
- Check whether the chat template and system prompt match the model. A template mismatch degrades answers and has nothing to do with quantization.
- Standardize temperature, top_p, and other sampling parameters before comparing, so parameter differences aren’t mistaken for quantization loss.
- Verify the source and version of the model file to avoid corrupted or engine-incompatible quantized files.
6. Pre-Purchase Checklist
- List the models you plan to run and compute weights, KV cache, and framework overhead using the formulas above.
- Confirm the VRAM fits the target configuration with a 20–30% buffer.
- Test the target quantization with the same set of questions and assess whether answer quality is acceptable.
- If you need real-time interaction, prefer models that fit entirely in VRAM.
- If you only run offline batch jobs, a pure-CPU build can be a low-cost option.
- Verify local pricing and the risks of used parts before buying. All speed figures in this article are theoretical ceilings for reference only.
