Running 27B Open-Source LLMs Locally on Consumer GPUs: A Practical Guide
This article explains how to assess whether a 27B open-weight LLM can run locally on consumer GPUs. It covers VRAM budgeting, quantization trade-offs, bandwidth-limited decode speed, and the token cost of reasoning chains, with code snippets and practical checks for deployment planning.
1. Background: Parameter Scale and the Local Deployment Boundary
The parameter scale of an open-weight model determines whether it can run on a personal workstation or an ordinary server. Models with fewer than 10B parameters can usually be loaded directly on many consumer GPUs. Models in the 20B to 30B range sit in a special zone: after quantization and compression, their weights can fit within the VRAM of a single mid-range to high-end consumer GPU, while their capability is clearly higher than that of smaller models.
This range has drawn attention in recent years for three main reasons. First, improvements in model architecture and training methods have continued to raise capability at the same parameter scale. Second, quantization technology has matured, allowing weight size to be compressed to a fraction of the original. Third, inference software now supports consumer hardware more comprehensively. Together, these factors have turned “running a usable general-purpose model locally” from a laboratory demonstration into an executable engineering plan.
This article discusses a general technical question. Benchmark results for different models, and their relative gaps to closed-source models, should be based on their respective public evaluation results. This article does not rank models.
2. VRAM Budget: Calculate Before Deployment
Before deployment, estimate VRAM first. Weight memory is calculated directly: parameter count multiplied by bits per parameter, then divided by 8 to get bytes.
def weight_memory_gib(params_b: float, bits: float) -> float:"""Estimate VRAM occupied by model weights in GiB, excluding KV cache and runtime overhead."""return params_b * 1e9 * bits / 8 / 1024**3for bits in (16, 8, 4, 3):print(f"27B @ {bits:>2} bit: {weight_memory_gib(27, bits):.1f} GiB")
Using the function above, the weights of a 27B model are roughly: about 50.3 GiB at 16-bit, about 25.1 GiB at 8-bit, about 12.6 GiB at 4-bit, and about 9.4 GiB at 3-bit. This shows that only after quantization can such models potentially fit on a single consumer GPU.
Beyond weights, KV cache must also be reserved. KV cache grows linearly with context length, calculated as:
KV cache bytes = 2 × layers × KV heads × head dimension × context length × bytes per element
The number of layers, KV heads, and head dimension must be checked in the configuration file of the model being used; they should not be estimated from experience. Available VRAM can be understood as follows:
- Total VRAM minus the memory used by the operating system and display output
- Minus weight memory
- Minus KV cache memory
- Minus the runtime overhead of the inference framework itself
In practice, reserve some margin for the overheads above. Otherwise, in long-context scenarios the program will fail due to insufficient VRAM, or it will be forced to offload part of the computation to system memory, causing a clear drop in speed.
3. Quantization: Trade-offs Between Size, Speed, and Quality
Quantization is the process of converting weights from a high-precision representation to a lower-precision one. The lower the bit count, the smaller the file, the less VRAM is used, and the greater the precision loss.
Precision loss is not uniform across tasks. The following types of tasks are usually more sensitive to precision:
- Multi-step mathematical reasoning
- Long-chain code generation
- Structured tasks with strict output format requirements
Relatively robust tasks include general Q&A, summarization, and text rewriting. Therefore, when choosing a quantization level, do not look only at a composite score; compare results at different levels on your own task samples.
A practical principle is: choose the highest-precision quantization level that fits within available VRAM. When VRAM is clearly insufficient, lower-bit options such as 3-bit can be considered as alternatives, but they need dedicated validation for the target task; their quality should not be assumed to be usable.
4. Inference Speed: Memory Bandwidth Sets the Theoretical Upper Bound
When generating text, a dense model usually needs to read all weights once for each token generated. Therefore, single-batch decoding speed is mainly limited by memory bandwidth, not by compute capability.
def decode_upper_bound(bandwidth_gbs: float, weight_gib: float) -> float:"""Theoretical upper bound for single-batch decode speed (tokens/s), ignoring attention computation, KV reads, and scheduling overhead."""weight_gb = weight_gib * 1.073741824 # Convert GiB to GBreturn bandwidth_gbs / weight_gbprint(decode_upper_bound(400, 9.4)) # Approximately 40 tokens/s
With 3-bit weights of about 9.4 GiB and memory bandwidth of 400 GB/s, the theoretical upper bound is about 40 tokens/s. This number is only an upper bound. Actual speed is also affected by attention computation, KV cache reads, kernel scheduling, and power policy, and is usually clearly lower than the theoretical value. Before final selection, use measured data from the actual deployment environment.
In addition, the processing stage for long prompts (prefill) is mainly affected by compute capability, which is a different bottleneck from token-by-token generation. For tasks with very long inputs, the wait for the first token may be longer than expected, so these should be measured separately during evaluation.
5. Token Cost of Reasoning Chains
Some models generate a segment of intermediate reasoning content before giving the final answer. These tokens also consume time and power, and therefore directly affect task cost.
In practice, the following pattern can be observed: the clearer the task description, the shorter the intermediate reasoning usually is; when the task description is vague or the input conditions are incomplete, the model tends to make repeated attempts and self-checks, and token consumption rises clearly. In other words, the main way to control reasoning cost is prompt design, not just changing the model.
Specific measures include:
- Specify the source and format of the input
- Specify the output structure, for example output only JSON
- Provide acceptance criteria and explain what result counts as passing
- Break large tasks into sequential sub-tasks
