Expert Pruning and Speculative Decoding to Reduce LLM Inference Costs
Large Mixture-of-Experts (MoE) models can remain expensive to serve even when only part of their parameters are activated at inference time. This article explains how selective expert pruning, speculative decoding with a lightweight draft model, context-length and memory trade-offs, accelerator right-sizing, and a closed-loop cost model can reduce inference costs while maintaining quality. It also outlines calibration, regression testing, canary rollout, and observability practices for production deployment.
1. Background: As Parameter Scale Grows, Inference Cost Is Not Just About Activated Parameters
Very large Mixture-of-Experts (MoE) models are often described as activating only some experts during inference, but that does not mean the cost is negligible. The total parameter count determines weight storage size, memory footprint, and loading overhead. Expert routing determines the actual computation path triggered by different requests. Context length determines the capacity and bandwidth pressure of the KV Cache. Batching strategy affects throughput, latency, and accelerator utilization. For trillion-parameter models, inference serving cost usually consists of four parts: accelerator procurement or rental cost, specification cost driven by memory capacity and bandwidth constraints, cache cost in long-context scenarios, and redundant capacity reserved to meet latency targets.
Therefore, cost reduction is not simply equivalent to “switching to cheaper cards.” A more reasonable path is: first reduce memory and compute requirements through model structure optimization, then improve per-unit-time throughput through inference execution optimization, and finally perform accelerator specification downgrades and cloud capacity orchestration. One type of public engineering practice shows that by selectively pruning expert modules in a trillion-parameter MoE model and using the freed memory to run a lightweight draft model, it is possible to maintain core inference quality while migrating the required accelerators from high-specification, high-cost configurations to relatively lower-cost configurations. In a typical case, accelerator procurement cost fell by up to about 20%, and cloud rental cost fell by up to about 22%. The core of such results is not a single trick, but the combination of “model lightweighting + inference execution optimization + closed-loop cost modeling.”
2. Expert Module Pruning: Removing Redundant Paths, Not Core Capabilities
A Mixture-of-Experts model consists of multiple expert modules and a routing mechanism. Different experts take on different responsibilities for different tasks and different tokens. Expert pruning is not about crudely deleting layers or reducing hidden dimensions. Instead, it uses calibration data to collect statistics on routing frequency, expert contribution, and task sensitivity, selectively removes expert modules that contribute little to core tasks or are highly redundant, and then uses lightweight recovery training, distillation, or calibration so that the remaining experts adapt to the routing distribution.
A practical workflow usually includes: first, build a calibration set covering core business tasks, long-tail tasks, long-context inputs, and boundary inputs; second, collect routing statistics, including expert hit rate, routing entropy, load balance, and sensitivity to output loss; third, score by “contribution × sensitivity × redundancy” and define a candidate pruning set; fourth, prune iteratively by ratio rather than making a large one-time cut; fifth, run quality regression and evaluate routing load; sixth, perform short-cycle recovery training or distillation when necessary. The pseudocode can be abstracted as follows:
function prune_experts(model, calibration_set, keep_ratio):stats = collect_router_stats(model, calibration_set)score = contribution(stats) * sensitivity(stats) * redundancy(stats)keep_set = top_k(score, keep_ratio)pruned_model = remove_experts_not_in(model, keep_set)pruned_model = calibrate_or_distill(pruned_model, calibration_set)return evaluate(pruned_model, core_eval_set, long_tail_eval_set)
In public cases, the number of modules was reduced by up to about 37%, while scores on a high-level scientific reasoning evaluation remained above 99% of the original model. This shows that the key to pruning is not “the more cut, the better,” but finding the inflection point between quality and cost. In engineering practice, three risks deserve attention: first, long-tail task degradation, because core evaluation sets may not cover all business scenarios; second, routing collapse, where load becomes overly concentrated on the remaining experts and throughput drops; third, changes in context capability, because freed memory after pruning may enable longer context but may also change cache management strategy. Therefore, pruning must be tied to business regression sets and cannot rely on a single benchmark score.
3. Draft Models and Speculative Decoding: Turning Freed Memory into Generation Speed
The memory freed by expert pruning does not have to be used entirely to lower card specifications. It can also be used to run a lightweight draft model. The basic idea is that the draft model quickly proposes several future tokens, and the main model verifies these proposals in parallel in one forward pass, accepts the parts consistent with the main model distribution, rejects inconsistent parts, and continues generation. This is the general logic of speculative decoding. It does not change the output distribution of the main model, but it can reduce the number of serial decoding steps in the main model at lower cost.
The pseudocode can be abstracted as follows:
function speculative_decode(main_model, draft_model, prefix, max_new_tokens):while generated < max_new_tokens:proposal = draft_model.generate(prefix, k_tokens)accepted = main_model.verify(prefix, proposal)prefix = prefix + accepted.tokensif accepted.all:continueelse:prefix = prefix + main_model.resample(accepted.next_position)return prefix
The benefit of speculative decoding depends on the consistency between the draft model and the main model, the acceptance rate, the proposal length, and the batching shape. The higher the acceptance rate and the lower the draft cost, the more obvious the speedup. If business inputs are highly diverse, a draft model that is too small will lead to a low acceptance rate, while one that is too large will erode memory and compute benefits. In engineering practice, teams should monitor acceptance rate, average accepted length, main-model verification overhead, P99 latency, and cost per token, and dynamically adjust proposal length according to business traffic. Public practice mentions that faster generation can be achieved even on 8 accelerators, which shows that the optimized memory headroom is effectively converted into execution efficiency rather than simply reducing hardware specifications.
4. Context Length and Memory Trade-offs: Pruning Ratios Must Serve Business SLAs
Long context is an important source of memory consumption. The longer the context, the larger the KV Cache, and the higher the pressure on attention computation and memory bandwidth. After expert modules are reduced, the freed memory can be used to support longer context: for example, a version with about 30% fewer modules can handle a context of about 512,000 tokens, and a version with about 37% fewer modules can handle about 1 million tokens. The key here is not to pursue the maximum context length, but to choose the pruning ratio according to business SLAs.
If the business is mainly short-text, high-concurrency workloads, memory should be prioritized for larger batch sizes and throughput. If the business is mainly long-document analysis and long-conversation memory, more memory should be reserved for the KV Cache, together with general techniques such as paged caching, cache reuse, and quantized caching. The higher the pruning ratio, the greater the potential quality risk and routing imbalance risk. Therefore, tiered configurations should be set: core business uses a lower pruning ratio, while non-core or offline tasks use a higher pruning ratio; long-context requests should be queued separately to avoid competing with short requests for memory. The ultimate goal is to make context length, latency, throughput, and quality explainable under the same cost model.
5. Accelerator Specification Downgrades and Cost Modeling: Do Not Look Only at Unit Price
Only after model memory requirements decrease and inference throughput improves is it possible to migrate the required accelerators from high-specification configurations to relatively lower-cost configurations. Previous similar optimizations reduced the high-specification accelerators required to run a very large model from 8 cards to at most 4 cards, and further optimization later lowered the specification level required for subsequent machine types. Such cost reduction should be based on a complete cost model, not just the price of a single card.
A simplified formula can be used for evaluation:
cost_per_token =(accelerator_unit_price × number_of_cards × runtime + network_and_storage_cost + operations_cost)/ effective_output_tokens
Here, “effective_output_tokens” should deduct failed requests, retries, queuing, and periods of low utilization. For procurement scenarios, look at total cost of ownership, including card count, data center power, cooling, spare parts, and utilization. For cloud rental scenarios, look at instance unit price, elasticity and scaling efficiency, idle time, and cross-specification migration cost. In public cases, procurement cost fell by up to about 20% and cloud rental cost fell by up to about 22%, but enterprises should retest with their own traffic structure. If lower-specification cards cause higher P99 latency, lower batching efficiency, or frequent scaling, the nominal unit price reduction may be offset by utilization losses.
6. Deployment Method: Quality Regression, Canary Traffic Shifting, and Observability
This type of optimization is not suitable for a one-time full switch. A four-layer validation approach is recommended: first, offline quality regression covering core tasks, long-tail tasks, long context, refusal behavior, and safety boundaries; second, routing and structure health checks, observing expert load, routing entropy, and module hit distribution after pruning; third, performance stress testing, recording throughput, P50/P99 latency, peak memory, acceptance rate, and cache hit rate; fourth, canary traffic shifting, gradually increasing traffic by business line or traffic percentage while retaining fast rollback capability.
Monitoring metrics should include at least: cost per token, throughput per card, memory utilization, KV Cache occupancy, draft acceptance rate, main-model verification overhead, error rate, timeout rate, and long-tail task scores. Only when quality regression passes, the cost model shows positive returns, and P99 latency meets the SLA should larger-scale switching begin. For cloud deployments, general-purpose elastic accelerated compute, container platforms, monitoring and alerting, and logging services—such as the general service categories offered on Baidu AI Cloud—can support this workflow; specific specifications and billing depend on actual availability. This article does not involve any specific partnership or customized service.
7. Conclusion: Cost Reduction Is a Systems Engineering Problem
Reducing inference cost for trillion-parameter models is not a single “swap the cards” action. It is the combined result of model structure, inference execution, context management, capacity planning, and observability. Expert pruning reduces memory and redundant computation, draft models and speculative decoding convert freed resources into generation speed, context strategy determines resource allocation priorities, and the cost model prevents “unit price down but total cost up.” Enterprises should first validate the quality inflection point with small pruning ratios, then gradually expand the pruning ratio and specification downgrade scope, and ultimately form an inference cost governance system that is observable, rollback-capable, and re-testable.
