Cutting LLM Inference Costs: Pruning and Speculative Decoding — Conditions and Limits
Large language model inference costs come mainly from weight memory, KV Cache memory, and multi-GPU communication overhead. This article explains how to establish a cost baseline, when and how to prune low-activation experts in mixture-of-experts models, how speculative decoding works and when it pays off, the monitoring metrics to track after deployment, and the boundaries of both techniques. It includes a KV Cache sizing formula, a pruning workflow, speculative decoding pseudocode, and practical deployment safeguards.
1. First, Confirm Where the Cost Comes From
Inference costs for large models are mainly driven by three components: GPU memory occupied by model weights, memory occupied by the KV Cache during inference, and communication overhead from multi-GPU parallelism. The total parameter count determines weight size, and the KV Cache size for a single sequence can be estimated as:
KV Cache per sequence ≈ 2 × number of layers × number of KV heads × head dimension × bytes per element × sequence length
Before optimizing anything, teams should measure a baseline on their own hardware, inference framework, and real request distribution — including cost per million tokens, P99 latency, and GPU utilization. Without a baseline, any claimed cost reduction figure cannot be reproduced.
2. Expert Pruning: Look at Routing Before Deciding What to Cut
In mixture-of-experts (MoE) models, expert activation frequencies can vary widely. The first step in pruning is to analyze routing statistics and identify experts that remain rarely activated on your target workload. A recommended sequence:
- Measure: record the activation count of every expert in every layer on a representative dataset.
- Filter: set a utilization threshold and only mark experts below it as pruning candidates.
- Prune: remove only a small batch of experts at a time, and keep a rollback switch so the original model can be restored.
- Regression-test: compare accuracy on evaluation sets covering core business scenarios, and record score changes per subtask rather than only the overall score.
- Deploy: roll out gradually (canary release) and compare cost and latency against the baseline.
There is no universal pruning ratio. In one experiment, removing a subset of experts preserved high scores on a particular task set — but results can differ significantly on different business data. When reporting pruning gains, always specify the model version, task set, evaluation methodology, and measurement period.
Memory freed by pruning can be reallocated to a larger KV Cache, but this does not automatically translate into longer context capability. Supporting longer inputs requires:
- Positional encoding that can extrapolate to the target length, or corresponding training;
- Attention implementations that support long sequences, such as chunked computation or memory-efficient operators;
- An inference framework with paged or quantized KV Cache management;
- Quality evaluation at the target length to confirm that retrieval and reasoning ability have not degraded noticeably.
3. Speculative Decoding: Choosing a Draft Model and Understanding Acceptance Rate
In speculative decoding, a small draft model proposes multiple tokens at once, and the main model verifies them in parallel. The speedup depends primarily on the acceptance rate α and the draft length k. Under a simplified greedy decoding assumption, the average number of tokens produced per main-model forward pass is approximately:
(1 − α^(k+1)) / (1 − α)
The higher the acceptance rate, the greater the speedup. The closer the draft model’s output distribution is to the main model’s, the higher the acceptance rate typically is. However, the draft model’s own compute and memory costs must also be included in the total cost accounting.
The following pseudocode illustrates the flow:
def speculative_decoding(main_model, draft_model, prompt, max_new_tokens, k=4):tokens = list(prompt)target_len = len(prompt) + max_new_tokenswhile len(tokens) < target_len:# 1. Draft model generates k candidate tokensdraft = draft_model.generate(tokens, num_tokens=k)# 2. Main model verifies all candidates in one forward pass,# returning the accepted prefix and a correction tokenaccepted, correction = main_model.verify(tokens, draft)tokens.extend(accepted)# 3. On rejection, append the correction token from the main modelif correction is not None:tokens.append(correction)return tokens[:target_len]
In production you also need to handle: the draft model running out of memory, individual request timeouts, and acceptance rates staying below a threshold. In these cases, the system should automatically fall back to standard autoregressive decoding and raise an alert.
Gains on a given hardware setup must be verified empirically. When reporting results, state the GPU model, per-card memory, parallelism strategy, draft model parameter count, acceptance rate, and QPS.
4. Post-Deployment Monitoring Metrics
| Metric | Meaning | Purpose |
|---|---|---|
| GPU utilization | How busy the compute units are | Decide whether to adjust concurrency or batch size |
| Memory usage and fragmentation | Share of weights, KV Cache, and fragmentation | Determine remaining headroom for context or batch size |
| Speculative decoding acceptance rate | Fraction of draft tokens accepted | Disable speculative decoding when it drops below a threshold |
| P99 latency | Response time of tail requests | Confirm optimizations have not hurt user experience |
| Cost per million tokens | Resource cost divided by output tokens | Measure the final cost reduction |
Releasing idle instances during low-traffic periods and scaling up quickly at peak times reduces idle cost. Container platforms and elastic compute should be selected based on your team’s existing contracts, compliance requirements, and operational capabilities; this article does not recommend specific vendors.
5. Applicability Boundaries
Suitable scenarios include online services with large parameter counts, tight GPU memory, clearly under-activated experts, or long-context requirements. For low request volumes, abundant hardware, or extremely strict tolerance for accuracy fluctuations, the gains from pruning usually fail to justify the verification and maintenance costs.
The two techniques can be combined, but each step should be measured independently. If you change both the pruning configuration and the decoding configuration at once, there is no way to attribute how much benefit each one contributed.
