Evaluating Speculative Sampling for Accelerating LLM Inference: Mechanisms, Tradeoffs, and Deployment
\
Speculative sampling (also called speculative decoding) accelerates large language model inference by having a lightweight draft model propose candidate tokens that the full target model verifies in a single forward pass, targeting the memory-bandwidth bottleneck of autoregressive decoding rather than its compute. This article explains why decoding is slow, how speculative sampling guarantees lossless output through its accept/reject criterion, how chain and tree candidate strategies differ, and the two main draft approaches—a jointly trained multi-token prediction head versus an independently trained lightweight draft model. It closes with a six-dimension framework for evaluating these solutions, their applicability boundaries, and a practical deployment path, so you can judge whether the technique reliably delivers higher throughput without changing the output.
1. Why Decoding Is Slow: The Bottleneck Is Weight Movement, Not Compute
Mainstream large language models generate autoregressively. To produce each token, the model runs a forward pass at the current position, combining the entire historical context to predict the probability distribution of the next token. During deployment, every layer’s weights and the attention mechanism’s KV Cache of past key-values stay resident in GPU memory, growing incrementally and reused by later steps.
Decoding has two phases, and their performance bottlenecks differ:
- Prefill phase: processes the whole input at once. Here sequences are long and matrix multiplications are large, so it is a compute-intensive workload with high utilization of the GPU’s compute units.
- Incremental decode phase: each step processes just one new token and produces one token per forward pass. The dominant operation degrades from matrix-by-matrix to matrix-by-vector, so each weight read serves only a single token and arithmetic intensity is extremely low. The bottleneck here is memory bandwidth: for every token, all model weights must move from memory to the compute units at least once.
An analogy: the weights are like a thick dictionary you consult repeatedly, and every lookup requires hauling the entire dictionary from cover to cover. The ceiling on decode throughput is roughly “memory bandwidth divided by the bytes moved per forward pass,” largely independent of the GPU’s peak floating-point compute.
This property dictates the key to acceleration: produce more tokens per weight movement without increasing the per-step compute. That is precisely the premise of speculative sampling.
2. Speculative Sampling: One Forward Pass, Multiple Tokens Verified
2.1 Basic Flow
Standard speculative sampling is a collaboration between two models:
- Draft model: a lightweight model that cheaply generates several candidate tokens autoregressively;
- Target model: the full model, which runs one forward pass over those candidates to verify them in parallel and decide how many to accept.
The reason “one forward pass can verify multiple tokens” is that the target model’s attention is causal: when processing candidate token i, its attention covers only tokens 1 through i, identical to the result when the target model generated the first i-1 tokens itself. Feeding the candidate sequence into the target as a batch of positions is therefore equivalent to decoding several times over, at the cost of just one extra weight read.
A full round proceeds as follows:
- The draft model generates K candidate tokens from the current context;
- The target model verifies the K candidates in parallel and also computes the next token “after the verified position”;
- Candidates are accepted left to right: at the first rejected position, stop immediately and discard that position and everything after it;
- From the last accepted position, sample an extra “bonus token” using the target model’s own prediction.
Even if every candidate is rejected, the round still commits one token. So a round produces at least 1 and at most K+1 tokens, while the target model executes only one forward pass.
2.2 Why the Output Matches the Target Model Exactly
Speculative sampling is a form of “lossless acceleration,” and the core is the acceptance criterion:
- Let the target model’s next-token distribution at position t be p, the draft model’s distribution be q, and the candidate token be x;
- Accept x with probability min(1, p(x) / q(x));
- Once a token is rejected, resample a token from the normalized residual distribution norm(max(p − q, 0)) and end the round;
- If all K candidates are accepted, sample the bonus token directly from p.
It can be proven that every token produced by this procedure strictly follows p, and is mathematically equivalent to sampling directly from the target model. Draft quality affects only speed, not output quality. This also means the benefit of speculative sampling depends entirely on the acceptance rate: the more “in sync” the draft is with the target, the greater the gain.
Three points need care in engineering:
- Under greedy decoding, the acceptance criterion reduces to “the candidate token equals the target model’s argmax.” The logic is simpler, and you can evaluate the whole batch of candidates with a single vectorized comparison.
- Under sampling mode, parameters such as temperature, top-k, and top-p should be applied to the target distribution before comparison. The draft model should ideally use the same or a more lenient sampling strategy; otherwise the distribution gap will lower the acceptance rate.
- If the target model uses specific sampling features—such as randomly breaking argmax ties or applying random penalties—a dedicated criterion is needed, or consistency cannot be guaranteed.
The following is illustrative code for greedy mode:
def speculative_step(ctx, draft, target, K):"""One round of speculative sampling under greedy decoding.ctx: confirmed output sequence (including KV Cache state)draft: lightweight draft model, responsible for cheap guessestarget: main model, responsible for parallel verificationreturns: newly confirmed token sequence for this round"""# 1. Draft model autoregressively guesses K candidates (each step far cheaper than the target)cand, state = [], ctxfor _ in range(K):nxt = draft.argmax_next(state) # draft forward + argmaxcand.append(nxt)state = draft.append(state, nxt)# 2. Target model verifies in parallel: append the candidate sequence after ctx, one forward pass# yields the target's logits at each candidate position, shape [K + 1, vocab]q = target.forward_suffix(ctx, cand)# 3. Compare left to right, truncating at the first mismatchm = 0while m < len(cand) and cand[m] == q[m].argmax():m += 1# 4. Bonus token: the target's prediction at the last accepted position# always sampled from the target distribution regardless of rejection (argmax under greedy)bonus = q[m].argmax()return cand[:m] + [bonus]
2.3 Where the Speedup Comes From
Let the per-step acceptance rate be α and the draft length be K. Under the common approximation model for chained speculation (where the draft extends along a single path), the expected number of tokens produced per round is:
E(α, K) = (1 - α^(K+1)) / (1 - α)
The target model’s average cost per token is 1 / E, and the draft model’s average cost is about 1 / (α·K). Let the target’s single-step time be T_main and the draft’s be T_draft (with T_draft far smaller than T_main). The theoretical speedup is approximately:
Speedup ≈ E / (1 + 1 / (α · K · (T_main / T_draft)))
This equation clearly bounds the optimization space: either raise the acceptance rate α, or lower the draft model’s per-step cost. The draft cost factor is α·K·(T_main/T_draft), meaning the expected number of accepted draft tokens per round multiplied by the ratio of single-step costs.
The approximation has two limitations:
- The actual acceptance rate varies during generation. Relatively deterministic passages such as instruction following tend to have higher acceptance; open-ended creation is lower.
- Draft models usually keep a separate KV Cache, so in long contexts their per-step cost also grows with context, and real-world performance is often below the model’s estimate.
2.4 From Chain Candidates to Tree Candidates
Chain speculation verifies only one candidate path per round; a single wrong guess invalidates every subsequent guess. A more efficient drafting strategy has the draft model produce multiple candidates at each position in one forward pass, forming a candidate tree. Each node in the tree can be accepted independently, and the target model verifies the whole tree in one forward pass using a tree attention mask, finally selecting the path with the most accepted nodes.
The tree attention mask lets each node see only its ancestors, guaranteeing that each node’s computed result matches what sequential decoding would produce:
import torchdef build_tree_mask(depths, device):"""Build a causal attention mask for tree candidates.depths: integer sequence of length N; depths[i] is the depth of candidate i in the tree.Nodes at the same level are numbered in expansion order, and a node's parent is"the same-level node with the largest index smaller than the current one."N: total number of candidate nodes submitted this round"""n = len(depths)keep = torch.zeros(n, n, dtype=torch.bool, device=device)last_at_depth = {}for i, d in enumerate(depths):keep[i, :i + 1] = True # left context + self, always visibleif d > 0 and (d - 1) in last_at_depth:keep[i, i + 1:last_at_depth[d - 1] + 1] = False # mask non-ancestor same-level nodeslast_at_depth[d] = ikeep = keep | torch.eye(n, dtype=torch.bool, device=device) # the diagonal is always visiblereturn keep
The tree approach brings improvements in two respects:
- Candidate coverage: multiple paths coexist, so as long as one path works out, you gain;
- Scalable verification structure: some implementations allow verifying one extra “depth node” each round alongside the current candidate chain, adding a near-certain opportunity at almost no extra drafting cost.
The cost is more complex mask construction, verification computation, and KV Cache management, all of which must be implemented in coordination with a specific decoding kernel.
3. Two Mainstream Routes: Jointly Trained Prediction Head vs. Independent Lightweight Draft Model
Where can the draft come from? The industry has settled on two routes with similar ideas but different engineering properties.
Route 1: Introduce multi-token prediction objectives during pretraining. Make “predicting multiple future tokens” a training objective, attach a lightweight prediction module to the model body, and train it jointly; at inference this module plays the draft role. Its advantage is that the draft is natively of the same origin as the target and fully reuses the target’s representations. Its limitation is that it must be decided during pretraining: bolting such a module onto an already-frozen target usually requires additional alignment training.
Route 2: After the target model is finalized, train an independent lightweight draft model. The draft is a model decoupled from the target. During training, the target serves only as a “data labeler” and “feature provider”:
- Run the target model forward over a batch of real corpus and collect its hidden states at each position;
- Feed the context to be predicted, together with the target model’s features at the corresponding positions, to the draft model;
- Train using the target model’s outputs as the supervision signal.
After training, the target model no longer participates in inference; only the draft model and the target model each run one forward pass.
The most notable property of this route is that “the draft is decoupled from the target,” which delivers four practical benefits:
- Acceleration even after the target is finalized: for models whose weights are already frozen and whose training pipeline is inconvenient to change, you only need corpus and compute to train a companion draft offline;
- One draft serves multiple model versions: variants within the same model series usually share a vocabulary and have similar internal representations, so the same draft adapts with only minor tuning;
- Customization by scenario and dynamic combination: train multiple drafts for different business distributions and select among them by request type online; you can also disable the draft under rising load to protect throughput;
- Independent deployment and operations: the draft loads as an independent component with its own version management, memory quota, and canary switch, allowing fast rollback if problems arise.
It is worth stressing that an independent draft does not win merely by “having fewer layers.” If you simply trim a few layers off the target, the draft has systematic biases in semantics and instruction following relative to the target, and acceptance is often limited. Effective independent drafts usually follow three design principles:
- Use the target model’s features as input: the draft no longer understands the context from scratch, but predicts the next step on top of the representations the target has “already understood”;
- Draft-scale small size: the body usually has only one to a few Transformer layers, with each layer’s output of the target model fed in as extra input features, so the compute is far smaller than the target’s;
- Multi-layer feature fusion: compared with taking only the last layer’s hidden state, fusing low-, mid-, and high-depth features provides richer information, balancing local lexical and long-range semantic signals and improving the “sync” with the target.
Two boundaries also need to be clear:
- Training still requires the full target model: collecting features means running a target forward pass over the entire training corpus, and the compute cost arises mainly at this step;
- The draft and target must share a vocabulary: the independent draft must still output logits aligned to the target’s vocabulary to participate in verification.
| Dimension | Jointly trained prediction head | Independent lightweight draft model |
|---|---|---|
| Training timing | Completed together with the target’s pretraining or post-training | Trained offline after the target is finalized |
| Relationship to target | Deeply coupled, exported together | Decoupled, loaded and replaced independently |
| Feature usage | Natively shares internal representations | Target features explicitly injected as input |
| Flexibility | Changing the target requires retraining | Combinable by model version and business scenario |
| Inference overhead | Implemented uniformly by the decoding framework | Requires scheduling a two-model inference flow |
| Main cost | Extra compute during pretraining | One full-corpus feature-extraction forward pass |
4. Operator-Level Breakdown: How the Target Model Gets “Multiple Tokens From One Forward Pass”
Take the example “target model is a dense model on the order of 4B parameters, draft is a lightweight model,” and walk through what each does in a complete decoding round.
Step 1: The draft model guesses consecutively. Conditioning on the target model’s output features, the draft runs several forward steps around the current candidate sequence (chain) or candidate tree (tree). Its computational structure is likewise “Embedding + attention + feed-forward network + output projection,” just with fewer and narrower layers, so it reads only a small fraction of the target’s weights per step. The draft side’s bottleneck is also memory bandwidth, but because it moves fewer bytes, its per-step latency is markedly lower than the target’s.
Step 2: The target model verifies in one forward pass. This is the crucial step. In ordinary decoding, each request’s input sequence length per step is only 1; during verification, several candidate tokens are appended at the end at once, and one forward pass outputs the prediction distribution at each candidate position. Taking a cacheless attention implementation (ignoring KV Cache) as the example:
- During ordinary decoding, the model’s internal tensor shape is
[B, 1, H], where B is the batch size and H is the hidden dimension. Each linear layer’s computation almost degrades to “vector times matrix.” - Verifying K chained candidates at once, the input length becomes K+1 and the shape becomes
[B, K+1, H], so each linear layer becomes a noticeably larger matrix multiplication and GPU utilization rises. - The output layer produces the vocabulary distribution for all K+1 positions at once. After comparing with the draft tokens, the first mismatch and everything after it are truncated, and this round’s output is the accepted sequence plus one bonus token.
- For tree candidates, the submitted sequence is all the candidate-tree nodes expanded by level; combined with the tree mask from Section 3, one forward pass verifies all branches, and the path with the most accepted nodes is kept.
Note that in real inference engines, decoding steps usually use a paged KV Cache: the key-values for candidate positions are first written to the cache, then read by the attention kernel; the tree case must handle “locate by parent node” addressing. The simplified core verification flow is shown below; real implementations rely on fused kernels and graph capture to reduce scheduling overhead:
def verify_with_target(ctx, draft, target, tree, T_main):"""The target model verifies a candidate tree in one forward pass and truncates by acceptance rules.tree: the candidate tree, with fields:nodes -- candidate tokens expanded by level (length N)parent -- the parent node index for each node (-1 means root)depth -- the depth of each node"""# 1. Target forward: input sequence = context + all tree nodesinput_ids = torch.cat([ctx.ids, tree.nodes], dim=-1).unsqueeze(0)logits = target(input_ids) # [1, L + N, V]pred = logits[0, -N - 1:-1].argmax(-1) # "what it should be" at each candidate position# 2. Trace back along parent pointers, find the longest path consistent from the rootbest = []for i in range(len(tree.nodes)):if tree.nodes[i] != pred[i]:continuepath, j = [i], tree.parent[i]while j >= 0:if tree.nodes[j] != pred[j]:breakpath.append(j)j = tree.parent[j]path.reverse()if len(path) > len(best):best = path# 3. Bonus token: the target's prediction at the end of the longest pathlast = best[-1] if best else -1bonus = logits[0, L + last].argmax(-1)# 4. Write-back: accepted nodes are written into the KV Cache; other branches' key-values are invalidatedoutput_ids = torch.cat([tree.nodes[best], bonus.unsqueeze(0)])kv_len = ctx.kv_len + len(best) + 1return output_ids, kv_len, T_main
Parallel batching in the decoding framework makes this more intuitive. Let D be the draft model’s vocabulary size, V the target model’s vocabulary size, H the draft’s hidden dimension, and H′ the total dimension of the injected features. In ordinary decoding, each request does one [1, V] × [V, H] vector-times-matrix at the target’s output layer per step; the verification stage changes this to a [K+1, V] × [V, H] small matrix multiplication. Compute increases by a factor of K+1, but the weights are read only once.
This leads to two important engineering conclusions:
- The gain comes from memory-access reuse: weight reads are “amortized” across multiple positions, improving hardware utilization rather than the per-token computational complexity;
- The bottleneck may shift: once K or the batch size grows large enough, the verification stage may go from memory-bound to compute-bound, at which point further increasing the draft length no longer helps and can even slow things down through wasted computation. The draft length should be determined empirically, not maximized blindly.
One more implementation point on the target side: the KV Cache for rejected branches must be handled correctly. The simple approach is to keep only the accepted portion’s key-values and maintain a valid cache-length pointer, so later steps do not read leftover invalid data.
Correctness assurance. Consistency checks can be done on both the inference and training sides. For example, on a test set, decode token by token with speculative sampling disabled to obtain a baseline output, then decode the same input with speculative sampling enabled (using the same random seed). Under greedy decoding the two should match token for token; if outputs differ in sampling mode, you need to go back to the acceptance criterion to investigate implementation errors. A common practice in public evaluations is to measure baseline throughput and speculative throughput separately under fixed decoding parameters, rather than reporting only hardware utilization.
5. How to Evaluate Such Solutions: Six Key Dimensions
- Effective throughput, not theoretical peak. Use end-to-end output tokens per second as the sole standard, while also watching the CPU scheduling overhead introduced by the speculative logic. This matters especially for deep models and ultra-long contexts.
- Acceptance rate and its distribution. Measure it separately by business data type, generation stage, and context length. Looking only at averages easily overestimates real online gains.
- Draft configuration. Whether draft depth, width, tree structure, and dynamic draft-length strategy are configurable, and whether it can degrade to ordinary decoding when load rises.
- The extra-cost ledger. Beyond the draft model’s own inference, count memory usage (draft weights, draft KV Cache, candidate-tree buffers) and the extra verification compute in batched serving. When batch size is very large, speculation can lower overall throughput.
- Output consistency. Whether the criterion strictly follows the accept/reject rules and can guarantee distributional equivalence in sampling mode. Getting this wrong is unacceptable—once outputs are inconsistent, none of the other metrics mean anything.
- Engineering complexity. How intrusive the approach is to the decoding kernel (attention masks, KV Cache management), how tightly it is coupled to the framework, and operational observability: whether acceptance rate, tokens produced per round, and the draft’s share of overhead can be conveniently exported to monitoring and alerting systems for tuning and troubleshooting.
6. Applicability Boundaries and Deployment Recommendations
Speculative sampling is not a universal accelerator. It fits best the scenarios that are “memory-bound and easy for the draft to hit”:
- Single-user interactive chat: small batch, high interactivity, sensitive to per-token latency (TPOT), and acceptance is usually high;
- Code generation and editing: code has strong local patterns and clear context constraints, so the draft hits easily;
- Target already finalized: the target is inconvenient to retrain, but you still want lower decoding latency;
- An existing strong domain small model: it can serve as the draft source, saving extra training.
The following scenarios need caution:
- High-concurrency online serving: batching already pushes the GPU toward being compute-bound, so the extra draft compute and verification overhead may exceed the benefit;
- High draft cost: the draft model is too large, or its distribution differs markedly from the target’s, so acceptance is low;
- Open-ended creative tasks: generation is divergent, and acceptance is often low;
- Ultra-long context: the draft’s per-step cost grows with context, diluting the gains;
- Short-output scenarios: requests generate only a few tokens, so gains are hard to amortize against the fixed overhead.
Drawing on the analysis above, the recommended deployment path is:
- First profile the performance bottleneck: measure the time, GPU utilization, and memory-bandwidth usage of the prefill and decode phases separately, and confirm the bottleneck really is memory access on the decode side before deciding whether to introduce speculative sampling.
- First use a training-free approach to assess feasibility: for example, an n-gram retrieval-based draft over recently appeared content. It costs almost nothing to train and quickly reveals the ceiling on acceptance for your business scenario. If acceptance is too low, training a draft model’s return on investment is usually not attractive either.
- Customize the draft by scenario: train the draft on real business corpus, focusing on acceptance under long context and domain terminology; start from a minimum viable draft size and gradually deepen and widen it.
- Tuning and experiment design: draft length, tree structure, and dynamic strategy should be tuned in tandem with the concurrency level, and fixed-random-seed A/B experiments should compare throughput, TPOT, and output consistency.
- Build observability and degradation mechanisms: bring acceptance rate, tokens produced per round, and the draft’s share of overhead into monitoring and alerting, and automatically or manually disable speculation when anomalies appear, falling back to ordinary decoding.
In terms of deployment form, this kind of acceleration logic is usually embedded in the model inference engine and carried by general capabilities such as containerized inference serving, elastic compute scheduling, and monitoring and alerting; the inference services and compute products offered by cloud platforms such as Baidu AI Cloud can equally serve as the foundation for such optimizations.
7. Conclusion
The value of speculative sampling is that it uses “parallel verification” to sidestep the hardware waste in autoregressive decoding where “one forward pass yields only one token,” and it does so without sacrificing output quality. Solutions exemplified by the independent lightweight draft model decouple the draft from the target: you can still accelerate after the model is finalized, and support scenario-based combination and canary switching.
The “multiple tokens per forward pass” speedup ultimately depends on three things being done right:
- Whether verification correctly coordinates with the decoding kernel, attention masks, and KV Cache management;
- Whether the feature alignment between the draft and target is sufficient;
- Whether the draft’s cost is strictly kept down.
Back to the original question—how should such solutions be evaluated? The answer lies not in any theoretical speedup ratio, but in whether they can reliably deliver the two promises of “higher throughput, unchanged output” under a specific business distribution and deployment form. Ground these six evaluation dimensions in data, then continuously correct with monitoring data—that is the most reliable way to make an engineering judgment.
