Reducing LLM Inference Costs: Cost Modeling Before MoE Pruning and Speculative Decoding
This article breaks down LLM inference costs into weight memory, KV cache, compute, communication, speculative decoding overhead, and rental cost. It then explains when and how to apply MoE expert pruning and speculative decoding, along with validation, rollback, and cost-based launch criteria. Readers get measurement formulas, risk controls, and pre-launch checklists.
1. Start with Cost Breakdown: Where Inference Money Goes
Inference cost is mainly determined by three things: GPU memory occupied by model weights, GPU memory occupied by the KV cache during generation, and compute volume.
Weight memory is roughly proportional to parameter count. At 2 bytes per parameter (FP16 or BF16), a trillion-parameter model needs about 2 TB of memory for weights alone, far beyond the capacity of a single GPU, so it must be partitioned across multiple GPUs. Quantization can reduce bytes per parameter to 1 byte or less, but it introduces accuracy risk and requires separate evaluation.
KV cache size is proportional to batch size, context length, number of layers, and number of KV heads. More concurrent requests and longer contexts increase KV cache usage. In long-context scenarios, the memory bottleneck many teams encounter actually comes from this part.
For compute, a dense model’s compute per generated token is roughly proportional to parameter count. A mixture-of-experts (MoE) model activates only some experts per token, so per-token compute is related to the number of activated parameters. However, all expert weights still need to reside in memory or be switched across GPUs, which limits deployment scale.
Attention compute overhead grows with context length, especially during the prefill stage.
The cost breakdown is summarized in the table below:
| Cost item | Main determining factors | How to measure |
|---|---|---|
| Weight memory | Parameter count, numerical precision, parallel partitioning method | Check memory usage per GPU after loading |
| KV cache | Batch size, context length, number of layers, number of KV heads | Stress test combinations of concurrency and context length; record peak memory |
| Compute time | Activated parameter count, batch size, operator implementation | Record per-token latency and throughput |
| Communication overhead | Parallel strategy, inter-GPU bandwidth | Measure the share of multi-GPU communication in total time |
| Speculative decoding overhead | Draft model size, acceptance rate | Record draft model time, weight memory, and acceptance rate |
| Rental cost | GPU type, billing model, number of replicas | Calculate cost per million tokens using the formula below |
The cost per million tokens is calculated as follows:
Cost per million tokens = GPU unit price × number of GPUs × running hours ÷ actual generated tokens × 1,000,000
During stress testing, fix the concurrency level, input-length distribution, and output-length distribution. Record actual generated tokens, not the max_tokens limit set in requests. Exclude warm-up data. Idle replicas must also be included in running time; otherwise, the unit cost will be underestimated.
2. MoE Expert Pruning: How to Do It and What Risks Exist
An MoE model consists of multiple experts, and each token selects only a few experts through the router. When some experts are rarely used in business traffic or have overlapping functions in certain domains, removing them can reduce memory usage. However, pruning changes the model structure, so quality must be validated.
The applicable conditions are: memory has become a deployment bottleneck, business requests are concentrated in a few domains, and expert activation distribution is clearly uneven. If activation distribution is close to uniform, the quality risk of pruning is usually greater.
A typical process is as follows.
- Prepare a calibration set. The calibration set should cover the real business distribution, including main task types, languages, and text length ranges. It is recommended to first use tens of thousands to hundreds of thousands of real requests for statistics, and ensure each expert has enough activation samples—for example, each expert is activated at least hundreds of times in the calibration set (example value; adjust according to actual distribution). With too few samples, the statistics are unreliable.
- Measure expert importance. For each token in the calibration set, take the top-k experts from the router output, add the routing weight of each selected expert to that expert’s score, normalize by the total number of tokens in the calibration set, and sort scores from low to high. Record activation counts at the same time. Looking only at activation counts can be misleading; a low-frequency expert may be irreplaceable for a few critical tasks. After removing candidate experts, also measure the change in model output loss as a second signal.
- Prune progressively by ratio. Remove only a small number of experts per round—for example, no more than a small fraction of total experts per round (example: within 5%)—and record quality metrics in each round. Do not prune too many at once.
- Recalibrate or fine-tune. After removing experts, the router output distribution changes, so calibration or short-term fine-tuning is usually needed. Whether full retraining is required depends on the results of quality regression tests.
- Check routing load. After pruning, some experts may receive too many tokens, causing load skew. In severe cases, expert collapse occurs, where a few experts carry most of the traffic. Continuously monitor each expert’s token share and calculate the ratio of the maximum expert share to the average share. Alert when this ratio exceeds a set threshold (example: more than 2x; adjust according to hardware and business).
- Set fallback conditions. If quality on key evaluation sets drops beyond a threshold relative to baseline (example: more than 1 percentage point), or long-tail tasks degrade noticeably, immediately roll back to the original model or a lower pruning ratio.
Recommended monitoring frequency after pruning: check routing load and key evaluation metrics daily after launch, and run dedicated evaluations for long-tail tasks weekly. Before each release, rehearse a rollback once and record the actual time from the switch command to service recovery.
3. Speculative Decoding: Applicable Conditions and Validation
Speculative decoding uses a small draft model to propose k tokens consecutively, then the target model verifies these tokens in parallel in one forward pass. The accepted prefix is kept directly. At the first rejected position, the target model provides a corrected token, and generation continues from that position. The benefit depends on whether the draft model’s tokens are accepted by the target model.
The applicable conditions are: the target model’s decoding stage is the latency bottleneck, batch size is small or medium, and the draft model’s output distribution is close enough to the target model’s.
Draft model selection should meet the following requirements:
- The vocabulary and tokenizer must be exactly the same as the target model; otherwise, token-by-token comparison is impossible.
- Prefer a small model from the same model family with similar training data, since distribution deviation is usually smaller.
- Let c be the ratio of the time for one draft proposal round to the target model’s single-step time. Smaller c is better. If c is too large, draft overhead cancels out the benefit.
- The draft model’s weights and KV cache also consume memory; include them in deployment calculations.
The two core metrics are as follows.
Acceptance rate α: the probability that a draft token is accepted by the target model. It must be measured on business data, not replaced by generic data.
Expected number of output tokens per round: E = (1 − α^(k+1)) ÷ (1 − α)
Approximate speedup: S ≈ E ÷ (k × c + 1)
Below is an example using pure formula calculation; it does not represent any actual benefit: when α = 0.8, k = 4, and c = 0.1, E is about 3.36 and S is about 2.4. Real values depend on business data and hardware and must be measured.
Also pay attention to the following during validation:
- As batch size increases, the target model gradually becomes compute-bound, and speculative decoding benefit may decrease or even become negative. Stress test under different concurrency levels.
- Sampling parameters (such as temperature and top-p) and sampling implementation must match the baseline. Standard rejection sampling theoretically preserves the target model’s output distribution, but custom samplers or implementation errors can change outputs. Before launch, compare output distribution and quality metrics on a fixed sample set.
- Acceptance rates for structured output (such as JSON) and code generation may differ significantly from natural language; collect statistics separately by task type.
- In long-output tasks, draft bias accumulates; observe tail latency separately.
Pre-launch checklist for speculative decoding:
- The draft model and target model have exactly the same tokenizer;
- Acceptance rate is measured separately by task type and recorded on business data;
- Throughput and p95 latency under different concurrency levels are better than baseline;
- Sampling results show no significant difference from baseline on a fixed sample set;
- The draft model’s additional memory usage has been included in the deployment plan;
- There is a one-click switch to disable speculative decoding.
4. Validation and Rollback: Controls Before and After Launch
During offline validation, prepare task-specific quality evaluation sets, including main tasks, long-tail tasks, and safety-related samples. Record baseline scores for each evaluation set, and compare every change against the baseline item by item.
Pre-launch checklist:
- Quality: the drop in key evaluation set scores relative to baseline is within threshold (example: no more than 1 percentage point), and long-tail tasks show no obvious degradation;
- Latency: p50 and p95 for time to first token and per-token latency meet targets; p95 must not be higher than baseline (example);
- Memory: peak memory still has a safety margin at the target batch size and longest context (example: reserve more than 10%);
- Cost: the cost per million tokens calculated by the formula above is lower than baseline after accounting for maintenance and validation costs;
- Rollback: the rollback switch and old model instances are ready; a rollback drill has been completed and its duration recorded.
During online canary rollout, it is recommended to route 5% of traffic first and run it in parallel with the baseline for a period. Compare error rates, latency distribution, and quality sampling results between the two groups. Sampling can be manual, or automated judge models plus manual review. After confirming there are no anomalies, gradually increase the traffic share.
Rollback conditions should be written before launch. Roll back if any of the following occurs: a key quality metric exceeds threshold, error rate rises noticeably, p95 latency exceeds target, or routing load alerts persist beyond a set duration. Keep hot standby instances of the old model during rollback to avoid cold-start time extending the failure window.
5. Cost Accounting and Launch Decision
The launch decision should compare net benefit rather than looking at a single metric. It can be estimated as follows:
Monthly net benefit = baseline monthly cost − new solution monthly cost − average monthly validation and maintenance cost − converted cost of quality loss
The converted cost of quality loss needs to be defined by the team based on business impact. For example, the handling cost of incorrect answers in customer service and the repair cost of incorrect code in code generation should use different conversion methods.
After memory usage is reduced, the required number of GPUs may decrease. However, after reducing GPU count, recalculate concurrency limits and peak load to confirm that the number of replicas can still cover peak traffic.
The decision principle is simple: invest only when quality meets the threshold, latency and memory meet targets, and the cost per million tokens still drops clearly after deducting maintenance costs. Otherwise, keep the original solution and continue optimizing batching and caching strategies.
Model costs first, then choose the method. Benefits should be based on measured results on your own business data and hardware.
