Local LLM Deployment: Model Selection, Hardware Assessment, and Practical Setup
This article explains how to deploy large language models locally. It starts with the difference between dense and MoE architectures, then shows how to estimate hardware requirements for VRAM, memory, and compute. It also covers practical steps for choosing quantization, configuring weight offloading, testing performance, and avoiding common deployment mistakes.
1. Understand the Architecture First: What Determines Whether It Can Run
Before choosing a model for local deployment, it is important to understand the model architecture. The architecture directly affects whether your hardware is sufficient. Today’s major open-source models generally fall into two categories: dense architectures and sparse Mixture-of-Experts (MoE) architectures.
A dense model activates all parameters during each forward pass. As the model grows, both VRAM usage and compute demand increase. MoE architectures take a different approach. They contain multiple parallel expert subnetworks and use a routing or gating mechanism to select only a small number of experts for each input token. As a result, the total parameter count can be large, while the number of parameters actually activated during a single inference remains small.
Models of this type are often described using an “A×B” notation. For example, a model labeled “about 35B total parameters and about 3B activated parameters” has roughly 35 billion total parameters, but each token only passes through a path of about 3 billion parameters. This “large storage, small computation” characteristic has an important practical implication:
Speed is determined by activated parameters, while VRAM usage is determined by total parameters.
Therefore, the key to running such a model quickly without exhausting VRAM is to keep frequently activated parts in VRAM and schedule the rest reasonably.
2. How to Find Reliable Information About New Models
Rather than waiting passively for recommendations, it is better to build a habit of actively tracking new open-source models. Open-source model communities are the main starting point. Most of the latest open-source models are published there, covering text, image, audio, video, multimodal, and reinforcement learning directions.
When evaluating a model, focus on three types of information:
- Model card: This describes the parameter count, architecture type (whether it is MoE), activation ratio, maximum supported context length, and recommended inference method.
- Version and weight format: Pay attention to the release date and the availability of different precision or quantized versions. Newer models generally perform better in instruction following and reasoning.
- License: Check whether the model allows commercial use or is restricted to research. This directly affects whether it can be used in production.
A practical approach is to follow community leaderboards and trending lists, and then cross-check the information with official announcements. This helps avoid being misled by a single source.
3. How to Estimate Whether Your Hardware Is Enough
Hardware evaluation is one of the easiest areas to get wrong in local deployment. The core constraint is VRAM, followed by system memory and compute.
1. Rough VRAM Estimation
A useful approximation is:
VRAM required ≈ parameter count × bytes per parameter
Under FP16, each parameter takes about 2 bytes. A 7B model therefore requires about 14GB just for the weights. To reduce memory usage, quantization is commonly used, which represents weights with fewer bits:
- 8-bit quantization: VRAM requirements are roughly half of FP16.
- 4-bit quantization: VRAM requirements are roughly one quarter of FP16, allowing larger models to run on consumer-grade GPUs.
Lower bit widths save more VRAM, but going too low may reduce accuracy. The choice depends on the trade-off between quality and available resources.
2. The Impact of Context Length
Many people only calculate the model weights and forget about the KV cache. The longer the context, the more VRAM the cache consumes. A 24GB GPU may seem sufficient for a medium-sized model, but once the context length is pushed to the maximum, VRAM can be exhausted quickly. This is a common reason why a GPU with a seemingly large amount of VRAM can still run into memory pressure.
3. System Memory and Weight Offloading
When VRAM is insufficient, system memory can help share the load. Many inference runtimes support offloading part of the layer weights to the CPU and system memory. This is especially effective for MoE models. Because only a small number of experts are activated at a time, less frequently activated layers can be placed in memory, while active parts remain in VRAM. This can significantly reduce VRAM usage.
4. Compute Determines Speed
GPU compute power and memory bandwidth together determine the actual inference speed. Bandwidth is often more important than raw compute, because token-by-token generation is a memory-access-intensive task.
4. A Practical Deployment and Tuning Workflow
After understanding the principles, deployment can follow this process:
Step 1: Choose the Right Precision
If VRAM is tight, such as around 8GB, start with lower-bit quantization. For 12GB or 16GB GPUs, begin with a moderate quantization level. If needed, try layer-offload values such as 10, 15, 20, and 25, then choose the setting that is both stable and fastest. For a 24GB GPU running with very long context, you can also start with a smaller offload value, such as 5, and fine-tune from there.
Step 2: Configure Weight Offloading
Use the parameter that controls how many MoE layers are offloaded as a kind of VRAM buffer knob:
- Sufficient VRAM: Set it to 0 and keep everything in VRAM for the fastest speed.
- Limited VRAM: Increase the value appropriately to move some MoE layers to CPU and system memory.
- Very long context: Even with 24GB of VRAM, you may need to increase this value to free up space for the cache.
Step 3: Validate and Stress-Test
After deployment, do not only check whether the model starts. Run the model and measure generation speed, such as tokens per second. At the same time, monitor VRAM and memory usage to see whether they remain stable. This helps prevent crashes caused by cache growth during long sessions.
5. Common Mistakes and Practical Takeaways
- Comparing only parameter count and ignoring architecture: Two models both labeled “35B” can behave very differently. An MoE model may run at a speed closer to a much smaller dense model, so the cost-performance tradeoff can vary significantly.
- Calculating only weights and forgetting the KV cache: In long-context scenarios, always leave enough room for cache growth.
- Chasing extremely low quantization: Too low a bit width may noticeably reduce quality. Choose a precision level that keeps accuracy acceptable.
- Treating offloading as a universal solution: Offloading can save VRAM, but it introduces data transfer overhead between system memory and the GPU. Increasing offloading blindly can actually slow down inference.
In short, deploying large language models locally is fundamentally about balancing VRAM, system memory, and compute. First, use architecture knowledge to determine whether a model can run. Then, use quantization and weight offloading to optimize resource usage. Finally, use real measurements to fine-tune the deployment. With this approach, you can quickly assess the feasibility of almost any new model and produce a practical deployment plan.
