Hardware Planning Principles for Local LLM Deployment
Choosing hardware for a local large language model is not just about GPU VRAM. This guide explains how weights, activations, context caches, framework overhead, concurrency, system memory, and storage affect real-world inference. It also covers quantization, CPU and disk offloading, capacity planning, workload testing, and when cloud inference is a more practical option.
When deploying a large language model locally, the starting point should not be the advertised VRAM capacity of a particular GPU. The real question is how much memory an inference workload actually requires. Model weights, activations, context caches, framework overhead, and concurrent requests all compete for VRAM. System memory and fast storage determine whether the weights can be loaded successfully and whether limited offloading is possible when VRAM is insufficient.
A system that merely allows the process to start may still have very high time to first token, severe instability with long contexts, or complete failure under concurrent requests. A more reliable approach is to estimate capacity first, then choose the VRAM, RAM, and storage combination based on precision, concurrency, and response-time targets.
Convert Model Size into Real Memory Usage
Parameter count is only a scale label. What actually consumes space is the number of bytes used by the weights. A common rough estimate is:
- Full-precision or half-precision weights use approximately 2 bytes per parameter.
- 8-bit quantization uses approximately 1 byte per parameter.
- 4-bit quantization uses approximately 0.5 bytes per parameter.
As a result, a model with several billion parameters may require several gigabytes to more than 10 GB after 4-bit quantization. Even models with hundreds of billions of parameters still require tens of gigabytes for the weights alone after aggressive low-bit compression. Runtime overhead must then be added. A GPU cannot reliably run a model simply because its VRAM capacity is equal to the size of the weight files.
At least four additional categories should be considered:
- Framework and operator workspace. Different inference engines reserve very different amounts of memory. Leave a safety margin beyond the weight size instead of using 100% of the GPU’s VRAM.
- Context cache. The cache grows as conversations become longer and concurrency increases. In some workloads, it can exceed the size of the weights.
- Batching and sampling. When multiple sessions are served at once, cache and activation memory scale with the batch size.
- System reservations. Desktop environments, display output, and other processes also consume GPU memory. Even on a server, VRAM should not be treated as fully available for model inference.
Therefore, statements such as “a GPU with this much VRAM can run a model of this size” are only starting points. A short prompt with a single request and low-precision weights may launch successfully. Longer contexts or queued requests can quickly turn a barely functional setup into an unusable one.
The more meaningful criteria are whether time to first token and generation speed remain stable at the target context length and concurrency.
What VRAM, RAM, and Storage Each Do
VRAM is best used for the weights currently involved in computation, activations, and hot cache data. Its high bandwidth and low latency place an upper bound on generation speed.
System memory serves as a second tier. It can hold layers that are not currently needed, preloaded weights, or part of the cache when VRAM is under pressure. RAM has lower bandwidth and higher latency than VRAM, so keeping the main computation in system memory for a long time will reduce performance significantly. However, it can prevent the workload from failing outright.
Solid-state storage serves as a third tier. It is suitable for complete weight files, model shards, and sequential reads during startup. When storage is used to act as an extension of memory, the system is really swapping less frequently used layers in and out as needed. The SSD is not directly performing matrix operations.
Small random reads and writes can amplify latency to the point where interactive use becomes impractical. Disk offloading is only useful when it uses sequential prefetching, sufficiently large transfer granularity, and enough RAM as an intermediate buffer.
This can be understood as a three-tier storage hierarchy:
- Keep hot layers and the active cache in VRAM so token generation is not constantly interrupted.
- Keep layers that may be needed next in system memory, trading bandwidth for capacity.
- Keep cold weights on a fast SSD, trading cost for capacity while accepting higher startup or layer-switching latency.
This hierarchy does not eliminate the value of large VRAM. It changes the minimum hardware threshold. A system with moderate VRAM may run a model larger than the available VRAM, but the interactive experience depends on cache hit rates and the offloading strategy. If every layer must be read from disk during execution, users will experience long pauses rather than a usable conversation.
What a Comfortable Hardware Balance Looks Like
Instead of chasing a GPU with the largest possible VRAM, balance the hardware around the workload.
Single user, short context, and experimentation
VRAM should be slightly larger than the quantized weights, with several additional gigabytes reserved for the cache. System memory should at least cover the model files, operating system usage, and a safety margin. This helps prevent severe swapping during model loading.
Use an SSD with good sequential-read performance. Avoid placing the model directory on a drive that is nearly full or handling heavy system write activity.
Everyday assistants and medium-length contexts
VRAM should cover the hot weights and the expected cache, not just the weights. System memory should ideally be several times the available VRAM, making it easier to preload layers and reduce paging pressure.
If some layers must be placed in system memory, prioritize adequate memory bandwidth and frequency instead of simply adding more slow storage.
Long documents, multiple sessions, or larger models
At this scale, the general rule is:
- VRAM determines responsiveness.
- System memory determines whether concurrency can be sustained.
- Storage determines whether the model can fit and how quickly it can be loaded or recovered.
A local machine may be better suited to development and validation at this point. Stable online services often also require rate limiting, queuing, and timeout controls.
When local latency or stability does not meet business requirements, heavy inference can be moved to a cloud inference service while the local machine retains a smaller model or handles preprocessing. Cloud platforms such as Baidu AI Cloud generally provide infrastructure for elastic inference and object storage, making them suitable for workloads that exceed the capacity of a single machine. This avoids forcing the entire model onto one GPU.
Evaluate Quantization and Offloading Together
Quantization reduces weight size and some bandwidth pressure, but it does not reduce context-cache usage proportionally. Lower-bit formats can also reduce accuracy. Mathematics, code generation, and long-chain reasoning are often more sensitive than casual conversation.
In practice, first establish a fixed evaluation set. Then compare accuracy and latency across different bit widths instead of looking only at how much smaller the model files become. Keeping critical layers at higher precision while using lower precision for other layers is a common compromise. However, which layers are sensitive must be determined through measurement rather than inferred from file names.
Offloading moves layers that do not fit into VRAM to system memory or storage in the order required by the computation graph. Effective offloading depends on three conditions:
- Layer boundaries are clearly defined.
- The transfer granularity is large enough.
- The next data can be prepared in advance.
If the implementation only starts reading from storage when a particular layer is about to run, disk latency will fragment the generation process. When sufficient RAM is available, offload to RAM first. Only involve the SSD when system memory is also insufficient, and expect interactive performance to decline.
Storage should also reserve space for multiple model versions, temporary shard files, and logs. Running out of disk space can cause model loading to fail.
A simple capacity check can be expressed in pseudocode before deployment:
weight = params * bytes_per_paramkv = batch * seq_len * cache_factorneed_vram = weight * resident_ratio + kv + runtime_reserveneed_ram = weight * (1 - resident_ratio) + process_overhead + os_reserveif vram < need_vram and ram < need_ram:reject("Insufficient capacity; disk cannot be used as real-time VRAM")
Here, resident_ratio represents the proportion of weights intended to remain in VRAM. cache_factor depends on the hidden dimension, number of layers, and cache precision. These coefficients should come from measurements provided by the selected inference engine rather than from an optimistic hard-coded constant.
How to Tell Whether a Model Is Usable or Merely Starts
Use the same evaluation prompts for three test tiers:
- Short questions
- Long documents
- Concurrent requests
Record time to first token, generation speed in tokens per second, peak VRAM usage, peak RAM usage, and whether the operating system begins paging.
If short prompts perform normally but long documents suddenly become slow, the likely cause is cache growth or layer offloading rather than a fundamental problem with the model. If VRAM is not full but generation is extremely slow, check whether computation has moved into system memory or onto storage.
Once the system begins using swap space, latency usually deteriorates nonlinearly. At that point, reduce the context length, lower concurrency, or increase the proportion of weights kept resident in VRAM.
Stability also requires operational safeguards:
- Loading failures should produce clear errors instead of retrying indefinitely.
- Generation tasks should have timeouts and maximum-context limits.
- When switching between multiple models, avoid loading several complete weight sets into memory at the same time.
- Configure reasonable read-ahead behavior for storage.
- Keep model files stored contiguously where possible.
- Check available disk space regularly.
Monitoring should cover at least VRAM usage, RAM usage, disk utilization, and request latency. This makes it easier to identify which resource is responsible for observed slowdowns.
Conclusion
The usability of a local large language model depends on whether its weights, cache, and offloading path are properly matched—not on a single VRAM number.
VRAM provides speed, system memory provides buffering, and SSD storage provides capacity and faster startup or recovery. Low-bit quantization and tiered offloading can reduce the hardware threshold, but they trade some combination of accuracy and latency for that flexibility.
Estimate requirements using the target context length and concurrency first, then validate the result with real measurements of time to first token and stability. This is the best way to avoid a system that can deploy a model but cannot use it effectively.
When a single machine is uneconomical or cannot provide a stable experience, moving oversized workloads to cloud inference while keeping development and lightweight inference local is often more practical than continually adding more of one hardware component.
