LLM Inference Cost Optimization: Expert Pruning and Speculative Decoding in Practice
This article explains why inference costs are high for very large language models. It covers how expert-level pruning in mixture-of-experts (MoE) models can reduce memory needs, how the freed memory can host a draft model for speculative decoding, and how GPU sizing and serving-layer optimizations add to the savings. It closes with practical steps for enterprise deployments.
1. Why Inference Costs Are So High for Very Large Models
In recent years, the parameter counts of flagship open-weight large language models have kept growing. Some mixture-of-experts (MoE) models now exceed several trillion total parameters. This growth creates two kinds of resource pressure.
First, a hard limit on GPU memory. Even with multi-bit quantization, the weights of a trillion-parameter model alone need hundreds of gigabytes of memory. Add the KV cache, activations, and framework overhead, and such models usually need several data-center GPUs connected through NVLink or a similar interconnect to run on one machine. With today’s mainstream 80 GB GPUs, deployment typically starts at eight cards, so the hardware cost of a single node can reach millions of yuan.
Second, structural waste in compute utilization. MoE architectures use a router to activate only some expert networks during inference, which lowers the compute per token. However, the weights of all experts must stay resident in memory so the router can call them at any time. In other words, memory is fully occupied while compute is sparsely used. This mismatch is the starting point for the pruning approach described below.
For enterprises that provide AI through APIs or private deployments, GPU purchases and cloud rental fees are the largest operating expenses. Any method that reduces resource needs while keeping accuracy nearly unchanged therefore has direct and significant business value.
2. Expert Pruning: Removing Parts of the Model Structure
Pruning is not a new idea, but expert-level pruning in MoE models has its own technical logic.
2.1 Basic Approach
An MoE model usually contains dozens or even hundreds of expert feed-forward networks. The router selects the top-K experts for each input token based on its features. Industry research suggests that expert usage varies widely. Some experts handle most of the traffic, while long-tail experts are rarely activated over the service lifetime, or activate only for inputs in specific domains such as certain languages or specialized content.
The core idea of expert-level pruning is to measure routing hit rates, evaluate each expert’s contribution, and selectively remove expert modules that contribute little to the task. Compared with neuron-level or layer-level pruning, expert-level pruning has several advantages:
- It works at a coarser granularity. Removing one expert reduces memory by the full weight size of that module, so the savings are certain and easy to estimate.
- It leaves the attention structure inside Transformer layers untouched, so the impact on the model’s overall representation ability is relatively controllable.
- It requires no complex weight reconstruction after pruning. The main work lies in designing the calibration dataset and evaluating contributions.
2.2 Key Engineering Challenges
The biggest risk of expert pruning is “capability collapse.” Some experts that appear low-frequency may handle critical long-tail abilities, such as advanced mathematical reasoning or rare language processing. In practice, teams usually follow these strategies:
- Build a multi-domain calibration set that covers typical task distributions, including scientific reasoning, code, multilingual tasks, and instruction following. This keeps evaluation from being biased toward high-frequency scenarios.
- Prune iteratively in small batches. Remove a few experts at a time, run regression tests on the benchmark set right after each step, and roll back once key benchmark scores drop below a set threshold.
- Keep a safety margin. In practice, after pruning roughly 30 to 40 percent of expert modules, the model’s scores on high-difficulty reasoning benchmarks can still stay above 99% of the original. This shows that expert redundancy can be substantial.
The following pseudocode illustrates the contribution evaluation workflow:
# Illustrative expert contribution statisticsexpert_hits = Counter()for batch in calibration_dataset:routed_experts = router(batch) # Expert IDs selected for each token at each layerexpert_hits.update(routed_experts)# Make batch-by-batch pruning decisions based on downstream task scoresfor candidates in sorted(expert_hits, key=expert_hits.get):if prune_experts(candidates) and eval_score(model) >= BASELINE * 0.99:commit_prune(candidates)else:rollback(candidates)
2.3 Reusing the Freed Memory
Memory freed by pruning does not have to go toward cost reduction alone. It can also fund performance work. This is a clever part of the approach: the freed memory hosts a smaller draft model, which works with speculative decoding to speed up generation.
3. Speculative Decoding: A Small Model Drafts, a Large Model Verifies
Speculative decoding is widely recognized in the industry as an effective inference acceleration technique. It works in three stages:
- Drafting: A small draft model autoregressively generates N candidate tokens quickly.
- Verification: The main model runs one parallel forward pass over these N candidate tokens and outputs the probability distribution at each position.
- Acceptance or rejection: Each token is accepted or rejected based on how well it matches the main model’s distribution. Accepted tokens are kept as they are. At rejected positions, the main model resamples the token.
Because the main model verifies tokens in parallel, a single forward pass can produce several accepted tokens, which raises token output per pass. The acceptance mechanism guarantees that the final output distribution matches what the main model would produce when decoding on its own. This makes the speedup mathematically lossless. It is not an approximation that trades quality for speed.
In this approach, the memory freed by pruning is just enough to hold the draft model. This creates a positive loop: pruning reduces memory, the memory holds the draft model, and speculative decoding increases throughput. The result is that even on eight mainstream high-end GPUs, generation speed improves noticeably, and the deployment does not need more expensive top-tier cards.
4. From Adding GPUs to Reducing GPU Count and Specs
Inference optimization typically progresses in three steps:
| Stage | Goal | Method | Cost Impact |
|---|---|---|---|
| Step 1 | Reduce GPU count | Reduce memory footprint, optimize parallelism strategy | Halves the number of GPUs of the same spec |
| Step 2 | Lower per-GPU spec | Adapt to second-tier flagship GPUs after pruning and quantization | Lowers per-card purchase cost |
| Step 3 | Optimize the serving layer | Batch scheduling, KV cache management, request routing | Raises overall utilization |
The first two steps are model-side optimizations. The third is serving-side optimization, which includes dynamic batching, prefix cache reuse, and load-aware multi-instance scheduling. Combined, these steps deliver substantial results. In this case, GPU purchase costs fell by up to about 20%, and on-demand cloud rental costs fell by up to about 22%. Rental savings matter especially to enterprises, because most teams run on elastic cloud resources, and every minute of rental is billed.
5. Practical Recommendations for Enterprises
Enterprises planning LLM inference infrastructure can follow this order:
- Measure before optimizing. Use inference profiling tools to find whether the bottleneck is memory, compute, or interconnect bandwidth, and avoid adding GPUs without evidence.
- Assess model redundancy. If you deploy an MoE model, evaluate expert-level pruning first, using key benchmarks as regression guardrails.
- Combine acceleration techniques. Pruning, quantization, speculative decoding, and continuous batching act on different dimensions, so their gains can stack.
- Make compute elastic. Use the elastic GPU rental options of cloud platforms, and scale capacity on demand alongside model optimizations. This turns fixed purchases into variable costs.
- Monitor quality continuously. Build an online quality dashboard, and require every optimization to pass a service quality gate before release. This prevents saving costs at the expense of user experience.
6. Conclusion
Optimizing LLM inference costs is shifting from an optional task to a required one. Once parameter counts reach the trillion scale, buying higher-end GPUs alone is no longer sustainable. Coordinated model-side and serving-side methods, such as pruning and speculative decoding, give enterprises a repeatable engineering approach for controlling costs while maintaining inference quality. As inference optimization extends from model compression into serving scheduling and execution, there is still considerable room to improve the return on AI infrastructure investment.
