Speculative Decoding for Faster LLM Inference: Drafting Design and Long-Context Optimization
Decode-phase latency is a common bottleneck in large language model inference: each step generates one token but must read all model weights, leaving most of the compute idle. This article explains how speculative decoding trades a single verification for multiple tokens, walks through the draft-and-verify flow and its cost model, and shows why the technique is output-lossless. It then introduces block-level parallel drafting, which produces a whole block of draft tokens in one forward pass and uses layer-wise feature injection to raise acceptance length. Drawing on benchmarks for 27-billion-parameter models, it reports decode speeds of 472 tok/s for short code requests and about 163 tok/s (roughly 2.98x autoregressive) for long-context code tasks, and covers deployment tuning across sampling parameters, draft length, draft-model matching, and bucketed load testing.
1. Background: Why the Decode Phase Becomes the Bottleneck
LLM inference generally splits into two phases. The prefill phase processes the entire input in parallel; it is compute-intensive and usually saturates the available compute. The decode phase generates only one token per step but still needs to read all model weights. Take a 27-billion-parameter model at 16-bit floating point (e.g., BF16): the weights alone are about 54 GB, so for a single request, every token requires moving that entire weight set through the device while most of the compute sits idle. This is a classic memory-bandwidth bottleneck, and the effect is more pronounced the smaller the batch.
As a result, the same model can run several times faster or slower under different quantization precisions and decode strategies. Comparing parameter counts alone is not enough to judge real-world experience; you also have to look at the precision configuration, drafting strategy, context length, and concurrency. Common inference-system optimizations fall into roughly three categories:
- Reduce weight movement: grouped-query attention, KV-cache quantization, low-bit weight quantization, and so on;
- Produce more per step: speculative decoding generates multiple tokens per step via “draft then verify”;
- Improve parallelism: continuous batching, paged KV cache, and so on.
The rest of this article focuses on the second category and puts it to the test in real serving scenarios.
2. How Speculative Decoding Works: Trading Verification for Time
2.1 Core Flow
In ordinary autoregressive decoding (AR below), the target model produces one token per forward pass. Speculative decoding splits a generation into two steps — “draft” and “verify”:
- Draft: a smaller draft model proposes k candidate tokens;
- Verify: the target model runs a single forward pass over these candidates and obtains the output distributions at k+1 positions;
- Truncate: compare from the front, stop at the first rejected position, keep only the accepted prefix, and use the verification rules to add 1 more token.
In this way, you advance as far as you are accepted. A step advances at least 1 token and at best k+1. What you save is the cost of multiple per-token target forward passes; the speedup depends on two factors: how well the draft model matches the target model, and how cheap the verification itself is.
2.2 Cost Model
Let τ be the average acceptance length, i.e., the number of tokens actually advanced per step (including the one the target model adds). The time per step can be approximated as:
T_round ≈ T_draft + T_verify + T_otherspeedup ≈ τ / (1 + T_draft/T_AR + T_other/T_AR)
Here T_AR is the time for one AR decode step without speculative decoding, and T_other covers scheduling, sampling, state maintenance, and other overhead.
If the drafting cost is negligible, the speedup approaches τ — “get the most accepted tokens with the least drafting.” If the drafting cost is roughly equal to one AR decode and τ = 3, the speedup is only 3/2 = 1.5. So “accepting more” is only a necessary condition; cheap drafting is the other half. This is the direct reason research teams keep exploring new drafting approaches.
One key observation: related work measured that on 8-billion-parameter models, verification accounts for about 72%–76% of total per-step time, with little difference across drafting schemes. This has two implications:
- Verification is already a fixed “big chunk,” so every bit of drafting cost you save is a net gain overall;
- Improving acceptance length remains the main line, but avoid making drafting heavier and heavier, which would eat back the verification time you saved.
2.3 Why Output Quality Is Not Affected
The key to using speculative decoding in production is that the verification rule is unbiased. Let the target model’s distribution at position x be p, the draft model’s distribution be q, and the drafted token be t:
- Accept t with probability min(1, p(t)/q(t));
- On rejection, resample from the normalized residual distribution max(0, p−q);
- When the entire block is accepted, use the target model’s distribution at that position to add 1 more token.
It can be proven that every output token strictly follows p and is exactly equivalent to per-token sampling. Greedy decoding (T=0) is a simplified case: accept if the target model’s top choice matches the draft, otherwise switch to the target model’s top choice. In other words, speculative decoding changes how fast the computation runs, not whether the output is correct.
2.4 Design Choices for Drafting
The capacity, size, and design of the draft model jointly determine the balance between τ and T_draft. Common approaches include:
- Using the target model’s own MTP (multi-token prediction) module for drafting;
- Independently training a small model;
- Self-speculative decoding on the target model, e.g., layer skipping or early exit;
- Training a dedicated draft model paired with a specific target model.
Whichever route you take, you face a tension: making drafts more accurate requires a larger, slower draft model; making it faster degrades draft quality.
3. Block-Level Parallel Drafting: Producing a Whole Block in One Forward Pass
3.1 Limitations of Token-by-Token Drafting
An early approach was to have the draft model autoregressively generate k drafts. The problem is that k drafts require k sequential forward passes, so drafting cost grows linearly with k. Even if each forward pass is cheap, it spends back some of the verification time you saved. In long-context scenarios, the draft model also has to read a growing KV cache, and this cumulative cost becomes more pronounced.
3.2 Block-Level Drafting and Cross-Position Dependency
A widely adopted idea is block-level parallel drafting: the draft model produces candidate tokens for k positions in parallel in a single forward pass, much like the target model processing a batch of tokens — it gets features at only k positions and then predicts each via the output head.
Drafting cost is therefore largely decoupled from k. But this introduces a new problem: the prediction at position t+1 should originally depend on the token actually generated at position t, yet during parallel prediction the later position cannot see the draft already fixed at the earlier position. A recent study offers an improvement. The authors come from a university and an internet company’s research lab, and the paper was published at the end of September 2026. Its approach can be summarized as:
Follow the idea of “drafting the whole block in one forward pass,” but at each layer of the draft model, inject the current prediction features of the later position into the earlier one, so that dependencies between tokens are built up layer by layer along network depth.
You can understand it as “trading depth for time”: cross-position dependencies that cannot be passed through a single layer are progressively completed through feature propagation across multiple layers. This is of the same origin as “trading iteration rounds for parallel prediction” in work such as parallel text editing; the difference here is that each iteration is one layer of the model, requiring no multiple forward passes.
Illustrative pseudocode (only to explain the idea, not a real implementation):
class BlockDrafter:"""Block-level parallel drafting: produce candidate tokens for k positions in one forward pass"""def draft(self, hidden, k):# hidden: the target model's hidden state at the current position, [B, 1, D]x = hiddenfor layer in self.layers:h = layer.attn(x, cache=self.cache) # one attention computation per position# Inter-layer injection: add the later position's "current" prediction features onto the earlier oneshifted = shift_right(layer.quick_head(h), k) # shift the whole block right by onegate = layer.gate(h) # gating weights determined by the current featuresh = h + gate * shiftedx = layer.ffn(h) # remaining compute; information keeps propagating layer by layer# A single output head predicts k positions at oncedraft_ids = self.head(x).argmax(dim=-1) # [B, k]self.cache.update(x) # cache for reuse in the next drafting roundreturn draft_ids
A few engineering notes:
- After
shift_right, the first position has no predecessor; fill it with a learnable vector; - The exact injection order and whether injection is bidirectional should follow the original method;
- All operations are computed in parallel across positions with no serial loop; drafting finishes in one forward pass.
3.3 Observed Performance
The work ran experiments on 4-billion- and 8-billion-parameter models: under greedy decoding, the 8-billion-parameter model’s average acceptance length is about 27.8% higher than another single-forward-pass whole-block drafting method, and throughput is about 23.3% higher in a mainstream open-source inference framework. The official open-source repository released two draft models matched to the models above.
4. Interpreting Real-World Speed: Short Requests and Long Context
4.1 Recap of Public Benchmarks
A researcher ran a long-context code agent test on a professional accelerator with around 96 GB of memory. Targeting a 27-billion-parameter model at 16-bit floating point with block-level parallel drafting, decode speed was about 4x that of disabling speculative decoding, and about 13% faster than another similar drafting scheme.
This article’s author then re-tested, in a cloud-based interactive development environment, a mixed low-bit quantized version of the 27-billion-parameter model on the same accelerator (most linear layers use 4-bit floating-point quantization, with a few sensitive layers kept at higher precision), again paired with block-level parallel drafting:
- Decode speed for short code requests reached 472 tok/s;
- For code-repair tasks with 4 contexts of about 100k tokens each, estimated from recorded and merged results, about 163 tok/s — roughly 2.98x AR.
It is important to emphasize that these are results under specific hardware, specific precision, and specific workloads. Numbers cannot be directly converted across different GPUs, quantization formats, or context lengths; each project should rely on its own measurements. Still, a few general conclusions can be drawn from them.
4.2 Short Code Requests: The Scenario Closest to the “Ideal Ceiling”
Short code completion has short inputs, highly regular output patterns, and small per-request batch, so every step reads the full set of weights — a classic memory bottleneck. Here speculative decoding is essentially trading one weight read for multiple tokens, and the gain is closest to the theoretical ceiling. 4-bit mixed quantization further cuts the weight-movement volume to about a quarter of the original; only the combination of the two produces the 472 tok/s figure.
Look at it another way: even estimating with 27 billion parameters, 4-bit weights are about 13.5 GB, and with the KV cache, each step moves on the order of tens of GB. Converting that out, 472 tok/s corresponds to an effective bandwidth of about 6.4–30 TB/s, far exceeding the memory bandwidth of any single accelerator. This shows the number can almost only come from the amplification effect of “generating multiple tokens per step,” not from a faster single forward pass.
4.3 Long-Context Code Tasks: Smaller Gains but Still Significant
Long-context scenarios differ in two ways:
- Large prefill: every request must first process about 100k tokens of input; once this cost is counted into the request’s total time, the decode-phase speed advantage is diluted;
- Diverse output forms: code-repair outputs often contain explanatory text and are not as highly regular as completion, so acceptance length drops accordingly.
Yet the measured result still reached 2.98x AR, showing the gains are preserved in real workloads. For tools like code agents, where inputs keep getting longer and outputs are not always well-formed code, long-context data is more representative than short requests.
Note also the difference between single requests and concurrent serving. Speculative decoding yields the largest gains at small batch sizes. As concurrency rises, the compute for verifying k+1 tokens per step is also amortized, and the per-token compute increment from speculative decoding may cancel out the gains, or even turn negative. This is why load testing is essential before deployment.
4.4 Under Quantized Deployment, Drafting Cost Deserves Careful Budgeting
Comparing the two data sets yields one more inference:
- At 16-bit precision, block-level parallel drafting is about 3.98x faster than AR and about 13% faster than another similar drafting scheme;
- Under mixed low-bit quantization, the same drafting scheme is about 2.98x relative to AR.
Quantization lowers overall latency but also lowers the AR baseline, so a lower speedup ratio is not surprising. A more valuable question is: after the target model gets faster from quantization, the drafting cost’s share per step rises relatively, and the gap between the two drafting schemes may be more pronounced than 13%.
The takeaway: in quantized deployments, drafting also needs to be “used sparingly” — reuse the target model’s KV cache as much as possible, avoid extra forward passes, and keep the draft model’s parameter count under control.
5. Deployment Tuning: Four Key Areas
5.1 Sampling Parameters
- Greedy decoding: drafts and verification results align easily, giving the longest acceptance length. Code generation commonly uses low temperature, which fits well;
- Random sampling: the higher the sampling temperature, the larger the divergence between the draft and target distributions, and acceptance length usually drops. For creative tasks like rewriting and brainstorming, the gains may be noticeably smaller than for code completion;
- Flexible verification: tree drafting and flexible acceptance strategies can improve acceptance length but add verification complexity. Weigh this against your business scenario.
5.2 Draft Length
Longer drafts mean a higher theoretical ceiling but also more risk: the target model usually stops at the first rejected position, so most of an overly long draft is wasted. Measure the best value for your target scenario rather than simply increasing it.
5.3 Training and Matching the Draft Model
Acceptance length depends largely on whether the draft model “understands” the target model’s output habits. Ideally:
- Same series and same vocabulary as the target model, distilled on the target model’s own output distribution;
- Training data covering the main business workloads, e.g., code continuation and repository-level repair;
- Context length covering typical lengths in real serving, e.g., the 100k-token level.
5.4 Service Integration and Evaluation
When integrating, confirm whether the inference service supports configuring speculative decoding, mounting the matching draft model, and paged attention mechanisms. The latter lets drafting and verification share the same KV cache, avoiding duplicate computation.
For evaluation, use bucketed load testing, because a single average hides too much. At minimum, cover:
| Dimension | Suggested values | What to watch |
|---|---|---|
| Concurrency | 1, 4, 8, 16 | Latency at low concurrency; throughput inflection at high concurrency |
| Context length | 1k, 8k, 32k, 100k | How much acceptance length decays with length |
| Task type | Short completion, single-turn long context, multi-turn agent | tok/s and quality consistency per scenario |
| Sampling parameters | Greedy, low temperature, high temperature | Sensitivity of acceptance length to temperature |
Core metrics include: average acceptance length τ, the ratio of drafting cost to per-step time, and time-to-first-token and per-token output latency. Also compare “speculative decoding on/off” outputs under a fixed random seed to verify whether generation results are identical. If they differ, the implementation is flawed — not a “quality difference from acceleration.”
6. Conclusion
The value of speculative decoding lies in trading one verification for multiple tokens. In memory-bound, small-batch decoding, it delivers close to a doubling in speed. Block-level parallel drafting pushes drafting cost down to be largely independent of draft length, and layer-wise feature injection improves acceptance length by “trading depth for dependency.” In tests on 27-billion-parameter models, quantization combined with this scheme pushed short-code-request decode speed to 472 tok/s, and code tasks with about 100k-token contexts stayed at around 163 tok/s.
Always remember: tok/s is a result, not a cause. What truly determines real-world gains is the combination of conditions — acceptance length, drafting cost, quantization configuration, and concurrency level. Doing bucketed load testing before integration and confirming the correctness of the lossless verification implementation are what let speculative decoding perform reliably in real workloads like long-context code agents.
