Hardware Requirements and Quantization Guide for Running LLMs Locally

1. The Core Challenge of Local LLM Deployment: From “Can It Run” to “Can It Run Well”

Deploying large language models locally is no longer as simple as downloading a set of weights and executing inference. As model sizes grow from 7 billion parameters to hundreds of billions or more, hardware compatibility, quantization trade-offs, and inference framework selection all directly affect deployment success and user experience. This article walks through a real deployment case and presents a systematic methodology covering model selection, memory estimation, and measured results across different quantization precisions.

Before starting a deployment, developers need to answer three fundamental questions: How large is the model? What is the target inference throughput? What is the available GPU memory budget? These three factors are interconnected and determine every subsequent technical decision. For example, a 7B-parameter model in FP16 requires roughly 14GB of memory just for the weights. A model with hundreds of billions of parameters can require more than 200GB, far exceeding the capacity of a single consumer GPU. This order-of-magnitude gap is why techniques such as quantization and model parallelism become essential.

2. Prerequisite: A Theoretical Model for Estimating GPU Memory

GPU memory usage is not simply parameters × bytes per parameter. During inference, memory consumption mainly consists of three parts:

1. Model weights: storage cost of the parameters at a given precision. For a 110B-parameter model, FP32 requires 440GB, FP16/BF16 requires 220GB, INT8 requires 110GB, and INT4 requires 55GB.
2. KV Cache: key-value pairs cached for past tokens during inference. Its size depends strongly on sequence length, batch size, number of layers, and number of attention heads. A simplified formula is: 2 × layers × attention heads × head dimension × sequence length × batch size × bytes per value.
3. Runtime overhead: intermediate activations, temporary buffers, CUDA context, and related resources. This typically accounts for 10% to 30% of the weight memory.

A more rigorous estimate is therefore: total memory ≈ quantized model weights + KV Cache + runtime overhead (estimated as 0.2 × weight memory). For a 110B model in FP16 with a sequence length of 2048 and batch size of 1, the weights require 220GB, the KV Cache takes several gigabytes, and runtime overhead is about 44GB, giving a total of roughly 270GB. This estimate closely matches real deployment results.

3. Case Study: Quantized Deployment of a 100B-Class Model

To validate the estimation model, we deployed a 110B-parameter Chinese conversational model on a high-performance cloud computing instance. The hardware setup included 8 high-end data center GPUs with 80GB of memory each, providing 640GB in total, which is sufficient for loading the unquantized model.

3.1 Baseline: FP16 Without Quantization

With no quantization, the model was loaded in FP16. Actual memory usage was as follows:

• Model weights: 220.4GB
• KV Cache (sequence length 2048, batch size 1): 12.8GB
• Runtime overhead: 45.2GB (approximately 2GB CUDA context per GPU, with activations allocated dynamically across layers)
• Total usage: 278.4GB distributed across 8 GPUs, with peak usage around 35GB per GPU

Measured inference speed was about 12 tokens/s with an input of 64 tokens and an output of 128 tokens. Memory utilization was approximately 43.5%, leaving significant headroom. Deployment on only 4 GPUs (320GB total) would have caused an out-of-memory error and failed to start.

3.2 INT8 Quantization Results

We applied per-tensor 8-bit quantization, compressing weights from FP16 to INT8. The key changes were:

• Model weights: reduced to 110.2GB, about a 50% reduction
• KV Cache: unchanged at 12.8GB because quantization does not affect KV Cache precision
• Runtime overhead: reduced to about 28.6GB, since INT8 computation requires smaller activations
• Total usage: approximately 151.6GB, with peak usage around 19GB per GPU

In this configuration, the model could be deployed comfortably on 4 GPUs with 320GB total memory, and batch size could be increased to 4 for throughput optimization. Measured inference speed reached 48 tokens/s, a 4x improvement over FP16. Some precision-sensitive operators such as softmax were still computed in FP16. Compared with FP16 inference, the BLEU score difference was less than 0.3%.

3.3 INT4 Quantization Stress Test

Using a GPTQ-style 4-bit quantization method, weights were further compressed to 55.1GB. However, the lower precision introduced additional overhead:

• Model weights: 55.1GB
• KV Cache: 12.8GB, unchanged
• Dequantization buffers: INT4 weights must be dequantized to FP16 before computation, requiring about 55GB of temporary storage that can be released after each compute step
• Runtime overhead: approximately 20.4GB
• Total usage: approximately 143.3GB, with peak usage occurring during dequantization

This test used a dynamic dequantization strategy, converting weights layer by layer to avoid allocating the full temporary buffer at once. Final inference speed was 31 tokens/s, lower than INT8 because dynamic dequantization increases memory read and write operations. However, memory usage dropped to the point where only two 80GB GPUs were required, significantly lowering the hardware barrier.

3.4 Measured Results and Key Conclusions

Quantization	Weight Memory	Total Memory	Inference Speed (tokens/s)	Memory Saved vs FP16
FP16	220.4GB	278.4GB	12	-
INT8	110.2GB	151.6GB	48	45.5%
INT4	55.1GB	143.3GB	31	48.5%

Three core conclusions:

1. INT8 quantization offers the best cost-performance balance: memory is cut in half, inference speed improves by 3x, and accuracy loss is negligible.
2. INT4 should be chosen only when memory is severely constrained: although it saves an additional 5.5% of memory, inference speed drops by 35%, and deployment complexity increases significantly.
3. KV Cache and runtime overhead cannot be ignored: in long-sequence or high-concurrency scenarios, KV Cache can even exceed weight memory. Planning must reserve sufficient headroom.

4. Choosing a Quantization Strategy: Balancing Accuracy, Speed, and Resources

Not every model should be quantized, and lower precision is not always better. Consider the following dimensions when selecting a quantization approach:

4.1 Task Sensitivity Analysis

• Code generation and mathematical reasoning: highly sensitive to numerical precision. Prefer INT8 quantization, and retain FP16 where necessary.
• Text summarization and conversation: more tolerant of precision loss. INT4 can be considered.
• Retrieval-augmented generation (RAG): pay attention to the embedding layer. A common practice is to keep the embedding layer in FP16 while using lower precision for other layers.

4.2 Inference Framework Operator Support

Different inference frameworks vary significantly in their support for quantized operators. Some frameworks support mixed-precision inference, with some layers running in INT8 and others in FP16, which can further optimize memory and performance. After selecting a framework, test its officially validated quantization configurations first.

4.3 Hardware Architecture Compatibility

The Ada Lovelace architecture provides dedicated acceleration for INT8 computation, while INT4 support depends on operator fusion. The Hopper architecture introduces FP8, which achieves a better balance between memory usage and precision. Developers with access to newer hardware should consider FP8 quantization first.

5. From Single-Node Deployment to Production Resource Planning

Local deployment is only the first step. Building a stable service requires additional planning:

1. Memory headroom: reserve 20% to 30% of GPU memory to handle fluctuations in input length and concurrent requests.
2. Model parallelism strategies: when a single GPU cannot hold the model, use tensor parallelism, which splits layers across GPUs, or pipeline parallelism, which assigns layers to GPUs sequentially. Combining these approaches can maximize resource utilization.
3. Dynamic batching: merging requests of similar lengths into a single batch can significantly improve GPU throughput. In our tests, increasing batch size from 1 to 4 improved throughput by 2.5x to 3x.
4. Degradation and fault tolerance: configure high availability for online services. When a GPU fails, isolate it automatically and degrade to a lower-precision mode to maintain basic service availability.

6. Summary and Action Checklist

Local deployment of large language models is a systematic engineering effort. The core methodology can be summarized in four steps:

1. Estimate memory requirements using quantization theory to determine hardware options.
2. Start with INT8 quantization for deployment testing to validate functionality and performance.
3. If memory is still insufficient, try INT4 quantization while optimizing KV Cache and dequantization strategies.
4. Adjust precision configuration based on task characteristics to achieve a dynamic balance between accuracy and resource consumption.

For most use cases, the recommended order is: FP16 when resources are sufficient, INT8 as the default choice, and INT4 for extreme deployment constraints. In practice, use automated tools to calibrate quantization parameters and avoid the uncertainty of manual tuning. Finally, always run stress tests before production launch to verify memory usage and latency under peak load, ensuring long-term stability of the deployment.