How to Choose and Optimize Inference for Locally Deployed Open-Source LLMs
Learn how to select an open-source large language model for local deployment and evaluate inference strategies for different workloads. This guide explains speculative decoding with an external draft model, why its performance can vary with concurrency, which deployment settings matter, and how to benchmark speed, latency, output quality, and resource use.
1. Start with Your Requirements
There is no single “best” open-source LLM for every local deployment. Before choosing a model, define the workload and operating constraints:
What tasks will the model handle?
- General question answering and knowledge retrieval
- Code generation and review
- Long-document summarization
- Multi-turn conversations
- Structured information extraction
- Tool calling and AI agents
- Mathematical reasoning or complex analysis
How much concurrency do you expect?
A model for a single user, a small internal team, and a public online service will have very different requirements. Single-request workloads tend to prioritize time to first token and streaming speed. Higher-concurrency services place more emphasis on aggregate throughput, GPU memory use, queueing time, and reliability.How much context do you need?
If your workload only processes a few thousand words at a time, supporting an extremely long context window may add unnecessary memory and compute costs. Larger context windows generally require more KV cache, which can reduce the number of concurrent requests the system can handle.Do you need tool calling or advanced reasoning?
For external tool use, multi-step reasoning, or strict structured output, evaluate the model’s instruction-following ability, compatibility with the required tool-calling format, and the inference server’s support for the relevant parsers.
Model selection is a balance among workload, hardware, concurrency, and cost—not a ranking by parameter count alone.
2. What Is an External Draft Model?
In conventional autoregressive decoding, the main model generates one token at a time. Each token requires another model computation. For large models, the compute and memory-access costs can limit streaming speed.
Speculative decoding takes a different approach: a smaller, faster draft model proposes several candidate tokens, and the main model verifies them in a single pass.
The process works as follows:
- The draft model quickly generates a short sequence of candidate tokens.
- The main model verifies the candidates in parallel.
- Candidates consistent with the main model’s distribution are accepted.
- At the first mismatch, acceptance stops, and generation continues from the main model’s result.
- The process repeats until generation is complete.
The key factor is not simply how fast the draft model is, but how many of its candidate tokens the main model accepts. When both models make similar predictions for a workload, the main model can validate multiple tokens at once, potentially improving generation speed. If acceptance is low, the main model must often resume token-by-token generation, while loading and scheduling the draft model adds overhead.
A rough way to think about the potential speedup is:
Effective benefit ≈ average tokens accepted per round × verification efficiency − draft-model overhead
Speculative decoding therefore does not deliver consistent gains across every dataset or concurrency level.
3. Why It Can Help at Low Concurrency
With one or a few active requests, the GPU may not be fully utilized. In this situation, inference can be constrained by token-by-token generation, memory access, and small-batch computation.
If the draft model proposes several good candidates at a time, the main model can reduce the number of sequential decoding steps. For example, if the draft model proposes seven tokens and the main model accepts most of them during one verification pass, each round can produce more useful output than conventional decoding.
Under some test conditions, external-draft-model approaches have achieved roughly twice the continuous output speed at single-request concurrency. Treat this only as a result from a particular combination of hardware, inputs, and sampling settings—not as a general guarantee.
Low-concurrency acceleration may be useful for:
- Internal knowledge assistants
- Personal development tools
- Interactive code completion
- Question-answering applications sensitive to responsiveness and streaming fluency
- Tasks that generate long passages of text
A faster streaming rate does not necessarily mean lower time to first token. Loading and scheduling the draft model, along with its initial computation, can add startup overhead. Measure time to first token, generation speed, and end-to-end request time separately.
4. Why Higher Concurrency Can Reduce Performance
As the number of concurrent requests increases, inference servers often use batching to improve hardware utilization. The main model can then process more requests together, making conventional token-by-token decoding more efficient.
An external draft model can introduce several challenges at higher concurrency.
1. The draft model competes for memory and compute
The main model may already use most of the available GPU memory. The draft model requires additional space for its weights, cache, and intermediate tensors. If memory is limited, the server may need to reduce batch size or may encounter memory fragmentation, more scheduling overhead, or longer queues.
2. Acceptance rates vary by data
Draft-model acceptance depends heavily on the input. Fixed templates, common code, and formatted text may be easier to predict. Random text, complex reasoning, rare terminology, and high-temperature sampling can reduce acceptance.
3. Batching gains may be outweighed by overhead
For a high-concurrency service, the goal is usually to maximize aggregate throughput. Faster decoding for one request does not guarantee better system-wide throughput if the draft model adds scheduling or verification costs.
4. Scheduling becomes more complex
The server must coordinate the main model, draft model, candidate tokens, accepted results, and request queues. At higher concurrency, an inefficient scheduling policy can lower batch utilization and make latency less predictable.
Single-request results should not be used to predict high-concurrency performance. In one test, speculative decoding delivered about 171 tokens per second (TPS) at low concurrency, compared with about 72 TPS for a standard approach. At higher concurrency, however, an external-draft-model approach could reach only about 59.8 TPS, while a more mature multi-token prediction method reached about 99.2 TPS. These figures illustrate how strongly results can depend on the workload and serving conditions.
5. Deployment Settings to Review
When deploying an LLM inference service, check the following settings and compatibility requirements.
1. Inference engine version
Speculative decoding requires inference-engine support for external draft models. Before deployment, verify whether the engine:
- Supports an external draft model
- Allows you to set the number of draft tokens
- Supports the relevant verification method
- Can use the same or compatible vocabularies for the main and draft models
- Supports tool calling, chain-of-thought parsing, and streaming output as required
Do not move directly to production just because an engine version is recent. First validate compatibility with a representative test set.
2. Tensor parallelism
Multi-GPU deployments typically use tensor parallelism to distribute the main model’s weights across devices. Set the parallelism degree to match the available hardware, and account for interconnect bandwidth.
Too little parallelism may leave insufficient memory on each device. Too much can increase communication costs. For large-model inference, inter-device communication can be as important as the compute capacity of each GPU.
3. GPU memory utilization
Avoid setting memory utilization as high as possible without leaving room for other needs. In addition to model weights, reserve memory for:
- KV cache
- Draft-model weights
- Intermediate activations
- Temporary batching buffers
- Communication buffers
- Peak request load
If the service frequently runs out of memory or experiences unstable request times, first consider lowering the maximum context length, maximum batch-token count, or maximum number of concurrent sequences.
4. Number of draft tokens
More draft tokens are not always better. Too few may limit the benefit of speculative decoding; too many may increase drafting and verification costs.
Start with a modest range, such as four to eight candidate tokens, and compare:
- Acceptance rate
- Time to first token
- Continuous output speed
- End-to-end request time
- Aggregate throughput
- GPU memory use
Choose the final setting based on measurements rather than copying a configuration from another environment.
6. A Practical Deployment and Evaluation Process
A careful evaluation can help distinguish real gains from workload-specific results.
Step 1: Establish a baseline
Test the main model without speculative decoding and record:
- Time to first token
- Average output speed
- P50, P95, and P99 latency
- Single-request throughput
- Aggregate throughput at multiple concurrency levels
- GPU memory use
- Service error rate
Without a baseline, it is difficult to determine whether an optimization has actually helped.
Step 2: Prepare representative test data
Do not rely only on random text. Include at least:
- Short question-and-answer prompts
- Long-document summarization
- Code generation
- Structured output
- Tool calling
- Complex reasoning
- Different input and output lengths
Measure acceptance rate and performance changes for each category. An overall average can conceal problems with particular workloads.
Step 3: Increase concurrency gradually
Test at concurrency levels such as 1, 2, 4, 8, and 16, then continue higher if appropriate. Watch for a smooth throughput curve. If throughput stops increasing or latency suddenly worsens, the system may be limited by memory, scheduling, or communication.
Step 4: Compare decoding strategies
Compare external-draft-model decoding with conventional autoregressive decoding, the main model’s internal multi-token prediction mechanism, and different batching strategies. For a meaningful comparison, use the same hardware, inputs, and sampling settings.
Step 5: Validate output quality and consistency
Speculative decoding is intended to preserve the main model’s output distribution, but production deployments should still check that:
- Outputs are complete
- Tool-call formatting remains stable
- Structured results such as JSON are not corrupted
- Long outputs are not unexpectedly truncated
- Results remain consistent across sampling settings
Performance improvements should not come at the cost of output quality or service stability.
7. Choosing the Right Approach
For personal use, internal assistants, or other low-concurrency workloads, prioritize:
- A model size that fits the available GPU memory
- Strong instruction-following ability
- An inference setup that supports long context while allowing context length to be adjusted
- A serving framework compatible with the required tool calling and structured output
- A speculative-decoding configuration tested under low-concurrency conditions
For high-concurrency production services, focus more on aggregate throughput, tail latency, recovery from failures, and resource utilization. An external draft model can be evaluated as an optimization, but single-request speed is not enough to justify a production rollout. A safer approach is to establish a stable baseline with conventional decoding, then introduce speculative decoding gradually and test it with representative traffic.
There is no single deployment choice that suits every open-source LLM workload. Model capability sets the performance ceiling, hardware affects cost, and the inference engine affects efficiency. The data distribution and concurrency pattern determine the actual results. For external draft models and speculative decoding, the most useful measure is not a theoretical speedup claim, but whether the approach consistently improves effective throughput on your tasks and hardware while keeping latency, output quality, and resource costs within acceptable limits.
