Speculative Decoding: Two Approaches Compared — Embedded MTP Modules vs. Standalone Draft Models

    This article explains why the decode phase is the main latency battleground in large language model inference, then walks through a unified draft–verify–rollback framework and the speedup formula that governs it. It compares two competing approaches in depth: embedded Multi-Token Prediction (MTP) modules that share the main model's hidden states and are trained jointly, and external standalone lightweight draft models that use multi-layer feature fusion and a single decoder layer. For each route it covers structure, why it must be kept small, hidden costs and constraints, and expected acceptance behavior. It closes with practical guidance for online serving (batch effects, adaptive draft length, memory management, evaluation pitfalls, deployment shape) and clear selection criteria for different training and inference scenarios.

    Speculative Decoding: Two Approaches Compared — Embedded MTP vs. Standalone Draft Models

    1. Where the Problem Starts: Why Decode Is the Latency Bottleneck

    Large language model inference splits into two phases: prefill and decode. Prefill processes the entire input at once, has high computational density, and is typically compute-bound. Decode produces just one token per forward pass, with extremely low arithmetic intensity in its matrix multiplications; most of the time goes to moving weights and the KV Cache across GPU memory, making it typically memory-bound.

    This property leads to a counterintuitive conclusion: during decode, increasing the batch size or pushing up compute utilization does almost nothing for single-request latency, and can even make it worse through queuing. The genuinely effective lever is “produce more tokens per forward pass” — which is exactly the problem speculative decoding sets out to solve. The core idea is to let a cheap draft model “guess” several tokens first, then have the main model verify them all in one pass. Accepted tokens are adopted directly; on a mismatch, the sequence rolls back to the point of divergence and resumes.

    Because verification runs in parallel (a single forward pass over the draft sequence yields a distribution at every position), one draft–verify round can produce up to “draft length + 1” tokens while the main model’s forward count increases by only 1. So the speedup does not depend on how strong the main model is, but on the balance of three factors: the draft is cheap enough, the draft hit rate is high enough, and verification overhead stays under control.

    2. A Unified Framework: Draft, Verify, Rollback

    Each round of speculative decoding can be abstracted into four steps:

    1. Draft phase: conditioned on the current context, the draft model autoregressively generates $k$ candidate tokens;
    2. Verify phase: the main model runs one forward pass over “context + $k$ candidates”, producing probability distributions at $k+1$ positions;
    3. Accept/rollback phase: it compares the draft distribution against the target distribution position by position, using rejection sampling to guarantee the output distribution matches the target model exactly; once a position is rejected, all subsequent drafts are discarded and a new token is sampled at that position;
    4. State synchronization phase: accepted tokens are written into the KV Cache, and the draft model’s forward state is rolled back to the same position before the next round.

    The key property is distributional unbiasedness: as long as the rejection sampling is implemented correctly, speculative decoding’s output distribution is identical to token-by-token decoding with the main model. It is therefore a “lossless speedup” that does not change model capability. This also means all the optimization headroom lives in two places: “how many tokens are produced on average per round” and “how many extra milliseconds each round costs”.

    Let $\alpha$ be the latency of a single draft forward pass relative to one main-model forward pass, $s$ the number of serial draft steps per round, $1+\varepsilon$ the relative cost of one main-model verification, and $A$ the average number of accepted tokens per round. Then the per-request speedup is approximately:

    $\text{Speedup} \approx \frac{A}{1 + \varepsilon + s \cdot \alpha}$

    This equation explains why the two routes differ so sharply in design philosophy: $s$ is only worthwhile when $\alpha$ is small enough, while $A$ is determined by draft quality.

    3. Route One: The Embedded Multi-Token Prediction Module

    3.1 Structure: Extracting Features from the Last Layer and Additive Mixing

    The typical embedded approach attaches one or more extra prediction heads (the MTP modules) on top of the main model’s output layer. Its input is not simply the previous token, but the last-layer hidden state of the main network combined with the embedding of the next token:

    1. # Pseudocode: draft path of an embedded MTP module
    2. h = main_hidden # Output of the main network's last layer
    3. e = embed(next_token) # Embedding of the currently generated token
    4. x = h_proj(h) + e_proj(e) # Project both feature spaces, then mix additively
    5. h_mtp = mtp_block(x) # One full (heavy) decoder block
    6. logits = head(h_mtp) # Distribution over the next token

    Because the draft path’s input is the main model’s hidden state, these modules naturally have very strong modeling power — they share all of the main model’s semantic representations, so the draft and main-model distributions align closely, and single-step hit rates often beat any external small model. More importantly, they are trained jointly with the main model: during training, gradients flow back into both the main model and the MTP modules, shaping the main model’s representations to be more favorable for multi-step prediction. This synergy cannot be replicated by a bolted-on draft model.

    3.2 Why It Must Be Made Small and Shallow

    The cost shows up in latency. The $h_proj$, $e_proj$, and that “full block” above mean a single draft forward pass includes attention, normalization, fully-connected layers, and even MoE routing and expert computation. And the draft is autoregressive: to generate the 3rd draft token, you must first sample the results of the first 2, so draft inference latency grows linearly with the number of draft steps.

    Engineering can only compress along two axes: shrink the shapes (hidden dimensions, intermediate dimensions, and expert sizes all far smaller than the backbone) and reduce the depth (one block instead of many). A publicly reported reference figure puts the single-step overhead of such an embedded module at roughly 1%–2% of one main-model forward pass. That looks tiny, but if you run 5–7 steps serially, the cumulative overhead approaches or even exceeds the cost of one main-model forward pass — at which point speculative decoding actually gets slower.

    For this reason, embedded approaches are typically constrained in production to MTP-1, i.e., producing just 1 draft token per round. The gain comes from the verification step: while verifying, the main model naturally produces the next real token, so a round yields about 2 tokens in practice, roughly halving the main-model forward count. Typical speedups land in the 1.3–1.8x range (which varies widely with model size and implementation and must be measured).

    3.3 Hidden Costs and Constraints

    • High coupling cost: the module must be introduced during the main model’s pretraining or post-training; bolting on a well-aligned module after the main model is frozen is very difficult.
    • No cross-model reuse: the module is tightly bound to the hidden distribution of specific weights; changing the base model means retraining.
    • Complex inference stack: the draft path shares the same kernel scheduling as the main path, and the attention mask must build a composite “causal + draft-visible” mask for all $k$ positions simultaneously, with many implementation details.
    • Fixed ceiling on gains: MTP-1 means it can only offer “2x parallel verification” and cannot improve further by adding draft steps.

    4. Route Two: An External Standalone Lightweight Draft Model

    4.1 Structure: Multi-Layer Feature Extraction and Concat

    The typical external standalone draft model no longer uses only the last-layer hidden state. Instead, it extracts features from multiple depths of the main model (for example, three semantic layers — low, middle, and high — plus the embedding layer, 5 positions total), concatenates them with the token embedding, and projects each back into the draft model’s working dimensions:

    1. # Pseudocode: feature concatenation in an external standalone draft model
    2. feats = [main_model.layer[l].output for l in (12, 24, 36)] # Different depths
    3. x = cat([feats[-1], feats[-2], feats[-3], embed(token)]) # Preserve each feature space
    4. h = draft_proj(x) # Project back to draft dimensions
    5. h = single_decoder_layer(h) # Just one layer, ~138M params

    Compared with additive mixing, the benefit of concat is that it does not destroy the semantic space of each feature source: the embedding retains discrete symbol information, the deep hidden retains abstract semantics, the middle layer retains local syntax, and the projection layers learn how to fuse them. The cost is that the concatenated dimension grows, so the projection matrix overhead must be kept down using low-rank or blocked techniques at implementation time.

    Another decisive advantage is decoupling: because the draft model lives outside the main model, a lightweight draft can be trained additionally while the main model stays fully frozen with fixed weights. This means existing models can also get speculative-decoding speedups without rerunning pretraining; it also supports “one main model + multiple draft models” that can be switched per business scenario (a draft head each for code, math, and general conversation).

    4.2 Light Enough for Multiple Serial Steps

    Since the draft side has only one decoder layer, its single-step overhead can be compressed to around 0.1% of one main-model forward pass (a reference figure; actual values vary with parameter count and batch size). At this point the $s \cdot \alpha$ term in the formula becomes tiny: the total latency of running 7 draft steps serially is still below the cost of a single step with the embedded heavy module.

    The result is that each round can produce close to 7–8 tokens (when fully accepted) while verification still costs just one main-model forward pass. Assuming an average accepted length of 4, the main-model forward count drops to 1/4, and the theoretical speedup can exceed 3x. That is the order-of-magnitude difference between the two routes.

    This route’s costs deserve an honest note: a single shallow layer is far weaker in expressive power than a full block, and low-level features align less well than jointly trained ones, so the draft hit rate is usually lower than with the embedded approach, and the accepted length decays quickly with draft position (acceptance might be 0.8+ for the 1st–2nd token but drop to 0.2–0.3 by the 6th–7th). Whether it pays off depends on the shape of the decay curve and the ratio against $\alpha$ — it must be measured, not inferred from parameter counts.

    4.2 Comparison of the Two Routes

    Dimension Embedded MTP module External standalone lightweight draft
    Attached to the main model Yes, mounted above the last layer No, independent weights
    Feature fusion Hidden + embedding, projected then mixed additively Multi-layer features + embedding, concat then projected
    Draft-side depth One full heavy block (incl. MoE, etc.) Single DecoderLayer
    Single-step overhead ~1%–2% of main-model forward ~0.1% of main-model forward
    Serializable steps 1 (MTP-1) 5–7 steps still below 1 embedded step
    Draft quality High (shared representations + joint training) Medium (shallow; compensated by steps)
    Deployment timing Must be trained together with the main model Can be trained afterward once the main model is frozen
    Reusability Tightly bound to specific weights Reusable across versions; switchable per scenario
    Implementation complexity Medium (composite mask, shared scheduling) Medium (feature extraction must hook to specified layers)

    5. Practical Notes for Online Serving

    5.1 Batching Dilutes the Gains from Speculative Decoding

    All of the analysis above assumes batch=1. But in online serving, the batch is usually much larger than 1. As the batch grows, decode gradually shifts from memory-bound toward compute-bound: the previously “idle compute” windows get filled, and verifying $k$ extra draft positions becomes real, wasted compute consumption.

    The engineering answer is adaptive draft length:

    1. def pick_draft_len(batch_size, load_level, hist_accept):
    2. # Small batch / low load: spare compute, lengthen the draft
    3. if batch_size <= 8 and load_level < 0.5:
    4. return min(7, hist_accept + 2)
    5. # Medium batch: conservative value, avoid wasting verification compute
    6. if batch_size <= 32:
    7. return 3
    8. # Large batch / high load: disable or reduce drafting to protect throughput
    9. return 1 if load_level > 0.8 else 2

    Pairing this with runtime statistics on the actual accepted length over the last N rounds for feedback adjustment is usually much more stable than a fixed step count.

    5.2 Variable-Length Drafts and Memory Management

    Different requests have different accepted lengths, which leads to uneven sequence lengths within a batch. Implementation notes: the draft region needs its own set of KV Cache slots, and rejected portions must be safely reusable; attention masks must be built per request; for GPUs, capturing the main path’s decode phase with a graph-capture mechanism (CUDA Graph–style) yields clear benefits, but it requires static shapes, so when draft length varies you should pad up to fixed tiers rather than going fully dynamic.

    5.3 Evaluation Methods and Common Pitfalls

    • Don’t report only the acceptance rate: a high acceptance rate combined with an expensive draft model can make end-to-end performance slower; you must report end-to-end TPOT/TTFT/throughput.
    • Separate the scenarios: conclusions for interactive batch=1 scenarios and large-batch offline scenarios can be completely opposite, so report data for each separately.
    • Keep the distribution aligned: use rejection sampling during acceptance to guarantee consistency with the main model’s distribution; if you switch to “accept drafts directly” (relaxed acceptance) for performance, that changes the output distribution, so you must clearly label it as an approximation and assess the quality impact.
    • Look at long sequences: the draft-side KV read/write cost grows with context length, so long-context scenarios need separate measurement.
    • Look at the task distribution: low-entropy tasks like code generation and structured output have naturally high acceptance rates; open-ended conversation has high entropy and low acceptance.

    5.4 Deployment Shape

    In production, the main model’s and draft model’s weights are usually kept together in a single model repository and scheduled by the inference framework within the same process. In the cloud, you can build it from a general-purpose combination of “model serving + elastic GPU resources + logging, service, and monitoring/alerting”, wiring draft length, the accepted-length distribution, and TPOT percentiles into alerting as core monitoring metrics. If you use the general-purpose inference and elastic-compute capabilities from Baidu AI Cloud, you can publish the draft model as auxiliary weights alongside the main model, then validate the speedup via a canary comparison before rolling it out to all traffic.

    6. Selection Guidance

    1. Plan during the training phase: if you are doing pretraining or large-scale post-training, embedded MTP is the most cost-effective choice — the distribution alignment from joint training cannot be added afterward, and MTP-1 already delivers a solid, stable speedup.
    2. Existing frozen models: prefer an external standalone lightweight draft. Zero changes to the main model, low draft-training cost (convergence is achievable with anywhere from hundreds of thousands to a few million samples), and the ability to maintain multiple draft heads per scenario.
    3. Pursuing extreme interactive latency: an external draft + 5–7 serial steps + adaptive step count is the only path that breaks through the “MTP-1 ceiling”, but you must compress the draft side down to a single-layer level, otherwise the linearly growing latency will eat the gains.
    4. High-concurrency scenarios where throughput is the priority: the marginal benefit of speculative decoding decays as the batch grows, so invest resources first in vectorization, parallel strategies, and scheduling optimization, and treat speculative decoding as a supplementary tool for low-load periods.

    7. Summary

    The essential difference between the two approaches is a trade-off between “stronger per step” and “cheaper per step”. Embedded MTP trades main-model representation sharing and joint training for the highest single-step hit rate, but the heavy block dictates one step per round. An external standalone lightweight draft trades a single-layer network and multi-layer feature concatenation for extremely low single-step cost, unlocking multiple serial steps — but it must compensate for its per-step hit-rate loss with more steps.

    To judge which is better, just return to that simple relationship: expected accepted length / extra cost per round. Evaluating any approach shouldn’t stop at isolated metrics like “how big is the draft model” or “how high is the hit rate”; it should measure end-to-end percentile latency under the target batch and context length. The former determines the direction of the route; the latter determines the parameter configuration — both need real measured data, not inference from model size.