27B Local LLM Inference Speed: Real-World Benchmarks and Optimization Methods
How fast can a 27B-class model run on consumer hardware? This article explains the factors that determine local inference speed, including quantization, VRAM capacity, KV cache usage, context length, inference engines, speculative decoding, and memory offloading. It also presents a practical benchmarking methodology and optimization steps for achieving more stable performance on both newer and older GPU platforms.
1. What Determines the Speed of a 27B-Class Model?
When discussing local LLM performance in Tokens/s, it is not enough to look at a single final number. The same model can perform several times faster or slower depending on the task, context length, and inference engine. A more accurate approach is to break down the main factors that affect inference speed.
For local deployment, the most important factors include:
- Model size and quantization precision
- VRAM capacity and memory bandwidth
- Context length and KV cache usage
- The level of operator optimization in the inference engine
- Whether speculative decoding or multi-token prediction is enabled
- Task type, output length, and reasoning workload
- Whether the model can remain fully resident in VRAM
The number of model parameters determines the basic amount of computation required for each generation step. Quantization affects weight size and memory traffic, while context length can significantly increase KV cache usage.
On older platforms, the primary bottleneck is often not the number of CPU cores. Instead, performance is usually limited by whether the GPU has enough VRAM, whether memory bandwidth is sufficient, and whether model weights must be repeatedly transferred between VRAM and system memory.
2. Typical 27B-Class Performance on Consumer GPUs
Using an older consumer GPU with approximately 22 GB of VRAM as an example, running a low-bit quantized 27B-class model at around 30 Tokens/s is achievable under suitable conditions. However, several requirements usually need to be met:
- The model weights must use 4-bit or lower-precision quantization.
- The KV cache should use a lower-precision format.
- As much of the model as possible should remain in VRAM.
- The context may be long, but it must not cause severe VRAM overflow.
- The inference engine should support graph optimization, operator fusion, or speculative decoding.
- Other applications with high VRAM usage should not run at the same time.
If the model weights occupy approximately 14 GB, the remaining VRAM can accommodate part of the KV cache, temporary compute buffers, and runtime overhead. Under these conditions, short-context generation may exceed 30 Tokens/s. As the context grows to hundreds of thousands of tokens, however, performance will decline because of higher cache usage, more demanding attention operations, and increased memory traffic.
It is also important to distinguish between prefill speed and decode speed. The Tokens/s value commonly displayed by a user interface usually refers to token-by-token generation. When a long prompt is submitted, the system must first perform substantial prefill computation.
The actual waiting time for a long-context task should therefore include both:
- Prefill latency: The time required to process the user input and previous context.
- Decode latency: The time required to generate the final response.
Comparing decode speed alone can overestimate the real user experience for long tasks.
3. How Quantization Affects Speed and Quality
Quantization is more than a way to reduce file size. It affects VRAM usage, memory bandwidth, numerical precision, and final output quality at the same time.
3.1 4-Bit Quantization
4-bit quantization is often the standard choice for running 27B-class models on consumer GPUs. It significantly reduces model size and memory bandwidth requirements. When the model fits entirely in VRAM, it usually provides a good balance between speed and output quality.
However, different quantization methods distribute error differently. Some preserve higher precision in important layers, while others apply more aggressive and uniform compression. Two quantized models with similar file sizes can therefore produce noticeably different results.
3.2 5-Bit and 6-Bit Quantization
Higher-precision quantization can improve stability in complex reasoning, code generation, and long-context tasks. The tradeoff is higher VRAM usage.
On a device with limited VRAM, using a higher-precision format may force part of the model into system memory. In that situation, real-world performance can decline substantially. When choosing a quantization level, the first question should not be theoretical precision, but whether the complete model can remain in VRAM.
A fully GPU-resident 4-bit model is often faster than a higher-precision model that requires transfers between the GPU and system memory.
3.3 KV Cache Precision
The KV cache is a major component of long-context inference. Its size increases as the context grows. Higher-precision storage can quickly consume available VRAM, while lower-precision storage allows longer contexts but may have a small impact on some complex tasks.
In practice, model weight precision and KV cache precision should be considered separately. Model weights can use a relatively stable 4-bit format, while the KV cache can use a lower precision based on available VRAM. This provides a better balance between speed, capacity, and output quality.
4. Why Longer Contexts Reduce Inference Speed
Many local inference benchmarks highlight context windows of hundreds of thousands or even millions of tokens. However, being able to load a long context does not mean that it can be processed efficiently.
Increasing context length creates three main problems:
Continuous KV cache growth
Every additional section of context requires more intermediate states to be stored, increasing VRAM usage.Higher attention costs
When generating a new token, the model must access more historical information. Even with optimized algorithms, long contexts increase memory traffic and computational pressure.More complex memory management
When VRAM is insufficient, the runtime may move part of the cache to system memory or perform frequent data transfers. This can cause a sudden performance drop.
For this reason, testing should record at least three ranges:
- Speed with an empty or short context
- Speed with a medium context, such as tens of thousands of tokens
- Stable speed with a long context, such as hundreds of thousands of tokens
If a system reaches 40 Tokens/s with a short context but only 20 Tokens/s with a long context, there is no contradiction. This is normal behavior. The most useful number is the sustained speed that matches the intended workload.
5. The Practical Value of Speculative Decoding and Multi-Token Prediction
Some modern inference engines support speculative decoding. The basic approach is to use a smaller and faster assistant model to predict several candidate tokens, which the main model then verifies in batches. When the candidates match the main model’s predictions, the number of individual calculations required by the main model can be reduced.
This approach is often more effective for:
- General question answering
- Structured text generation
- Code completion
- Continuous output with a stable writing style
- Tasks where the main and assistant models have similar output distributions
It does not accelerate every workload. In complex reasoning, conversations that frequently change direction, highly constrained output, or tasks with long internal reasoning processes, the acceptance rate of candidate tokens may be lower and the overall benefit may be limited.
When evaluating speculative decoding, it is not enough to report a peak speed. The following metrics should also be considered:
- Average generation speed
- Candidate-token acceptance rate
- Time to first token
- Sustained speed on long tasks
- Startup and model-loading time
An engine that reaches a high peak speed on short outputs but fluctuates frequently during long tasks may provide a worse daily experience than a slightly slower but more stable implementation.
6. Why Older Platforms Can Still Offer Good Value
Many local inference users rely on older CPUs, mixed-memory configurations, and aging motherboards. Such systems may not be ideal for general-purpose computing or data processing. However, as long as most of the model remains on the GPU, the CPU does not necessarily become the main bottleneck.
The advantages of an older platform include:
- Low hardware acquisition cost
- Support for large amounts of system memory
- Suitability for data loading, context management, and assistant-model tasks
- Lower upgrade costs for users who already own an older GPU
There are also important limitations:
- Lower system memory bandwidth
- Suboptimal motherboard memory-channel configurations
- Slower storage that increases model-loading time
- Limited PCIe bandwidth, which makes offloading more expensive
- Potential stability issues with mixed memory configurations
The central principle for an older platform is therefore:
Let the GPU handle as much of the inference workload as possible, and avoid frequently placing model weights or the KV cache in system memory.
7. Use a Consistent Benchmarking Method
To make performance comparisons meaningful across different systems, use a standardized testing process.
Fix These Parameters Before Testing
- The same model file
- The same quantization format
- The same context window
- The same sampling parameters
- The same output length
- The same thread count and batch size
- The same cache precision
- The same inference engine version
Record at Least the Following Metrics
| Metric | Description |
|---|---|
| Model-loading time | Time from startup until the model is ready for input |
| Time to first token | Time from request submission until the first token appears |
| Prefill speed | Speed when processing the input context |
| Decode speed | Tokens/s during continuous generation |
| Long-task average speed | Stable performance after the task has been running for some time |
| Peak VRAM usage | Memory used by weights, caches, and temporary buffers |
| System memory usage | Indicates whether significant offloading is occurring |
| Power consumption and temperature | Helps evaluate long-term operating cost and stability |
The test set should ideally include general question answering, code generation, long-document summarization, and multi-turn conversations. A single short response can produce a high peak value, but it does not represent real-world usage.
8. How to Further Improve Inference Speed
If current performance is below expectations, investigate the system in the following order.
Step 1: Check for VRAM Overflow
Determine whether model weights, the KV cache, and runtime buffers are all located in VRAM. If some components are being offloaded, first consider lowering quantization precision, shortening the context, or reducing cache precision.
Step 2: Adjust the Context Window
Do not pursue the maximum context length unconditionally. A moderate window is usually faster for everyday question answering. Larger windows are mainly necessary for tasks such as long-document analysis and codebase retrieval.
Step 3: Use an Appropriate Batch Size
A batch size that is too small can reduce hardware utilization, while one that is too large increases VRAM pressure. Adjust it gradually according to available memory instead of immediately choosing an extreme configuration.
Step 4: Test Speculative Decoding
If the workload is suitable and a compatible assistant model is available, measure the average benefit of speculative decoding. Focus on stability during long tasks rather than only on short-lived peak speed.
Step 5: Optimize Storage
Storage usually does not determine sustained generation speed directly, but it affects initial startup, model switching, and cache loading. Placing model files on slow storage can noticeably worsen the startup experience.
9. How to Evaluate “Best Value”
The value of a local inference setup should not be defined simply as Tokens/s divided by hardware cost. A more useful evaluation should consider:
- Generation speed
- Supported context length
- Output quality
- Stability during long-running workloads
- Peak power consumption
- Hardware acquisition cost
- Model-loading time
- Whether additional hardware is required
- Suitability for the user’s actual tasks
For example, one system may be extremely fast on short questions but frequently slow down with long contexts. Another may have a slightly lower peak speed but remain stable during complex workloads. The second system may be more suitable for everyday use.
For users on a limited budget, an older GPU combined with low-bit quantization, low-precision KV caching, and an optimized inference engine can provide a strong local experience. Users who care more about coding, agent workflows, and long-document processing should focus on sustained speed, available VRAM, and context-management capabilities.
10. Conclusion
The performance of a 27B-class model on a consumer GPU cannot be represented by one fixed number. With short contexts, low-bit quantization, and an optimized inference engine, reaching 30 Tokens/s or more is possible. When the context expands to hundreds of thousands of tokens, or when the task involves complex reasoning, code execution, and extensive multi-turn history, a lower speed is expected.
The most important aspect of evaluating a local inference system is not chasing the highest value shown in a screenshot. Instead, establish a repeatable benchmark that measures short-context speed, long-context speed, time to first token, sustained generation speed, and VRAM usage. Only by comparing these metrics with real workloads can you determine whether an older platform truly offers good value.
In practice, appropriate quantization, controlled context length, avoiding VRAM overflow, and selecting a suitable inference engine are often more effective than simply replacing the hardware. Speculative decoding and other general optimization techniques can provide additional gains when the workload is compatible.
For local LLM users, a stable and controllable 30 Tokens/s that can be maintained over time is usually more valuable than a peak speed that lasts only a few seconds.
