LLM Inference Cost Reduction: Expert Pruning and Speculative Decoding in Practice

    This article explains how to reduce large language model inference costs by combining MoE expert pruning with speculative decoding. It covers router-statistics-based pruning, multi-dimensional regression evaluation, memory recovery, draft-model deployment, service-layer optimization, and a six-step implementation plan with risk controls.

    1. Background: Parameter Growth and Inference Cost Constraints

    The capability gains of very large models often come with simultaneous increases in memory capacity, memory bandwidth, inter-card communication, and scheduling complexity. For models using a mixture-of-experts (MoE) architecture, only some experts are activated during a single inference pass, but engineering systems still need to reserve space for loading, distributing, and communicating all expert weights. Longer contexts also make KV cache usage more significant. Ultimately, cost is reflected not only in the number and specification tier of accelerator cards, but also in hourly rental fees in cloud environments, idle resources, and operational overhead.

    Therefore, reducing inference cost cannot rely only on “switching to cheaper hardware.” It should address three issues at the same time: first, whether the model side can reduce redundant expert modules without significantly damaging core capabilities; second, whether the execution side can use the freed memory to run a draft model and improve throughput per unit time through speculative decoding; third, whether the service side can turn optimization gains into stable cost reductions through batching, cache management, elastic scheduling, and canary evaluation. Public optimization practices have reported that by combining expert pruning with inference execution optimization, accelerator card procurement costs can be reduced by about 20%, and cloud environment rental fees can be reduced by more than 20%. Such results should be understood as engineering gains from multiple optimizations combined, not as a fixed promise from any single technique.

    2. Expert Pruning: From “Deleting Modules” to “Regressible Lightweighting”

    The core of an MoE model is that the router network selects experts per token. The first step in pruning is not direct deletion, but building a calibration set and collecting router behavior statistics: which experts are called frequently, which experts appear only in a small number of tasks, and which experts contribute more to sensitive tasks. Then, scoring is performed by combining usage frequency, routing confidence, output sensitivity, and task coverage, prioritizing removal of expert modules with low contribution and low sensitivity. The pruning ratio should be applied in stages—for example, start with light pruning to validate quality, then gradually increase the ratio. After each pruning step, short recovery training or distillation calibration is needed so that the remaining experts take on part of the responsibilities of the removed modules.

    The approach mentioned in public materials is to selectively remove expert modules responsible for specific tasks, reducing the number of modules by up to more than 30%, while maintaining most of the original score on difficult scientific reasoning benchmarks. In engineering practice, “quality preservation” should be broken down into multi-dimensional regression: general question answering, code generation, mathematical reasoning, long-document reading, structured output, and safety evaluation should all be covered. Looking only at a single average score is not enough. After pruning, it is also necessary to observe whether router load becomes imbalanced, avoiding latency jitter caused by overloading a small number of experts. The freed memory can be used for higher context tiers or draft models, but context expansion must also evaluate attention computation and cache bandwidth overhead.

    An illustrative pruning flow is as follows:

    1. for layer in moe_layers:
    2. stats = collect_router_stats(calib_set, layer)
    3. for expert in layer.experts:
    4. score = usage_score(stats, expert) * sensitivity_score(calib_set, expert)
    5. candidates = select_low_score_experts(layer, ratio=target_ratio)
    6. layer.remove_experts(candidates)
    7. calibrate_and_evaluate(model, regression_suite)

    3. Memory Recovery and Speculative Decoding: Using Saved Memory to Improve Throughput

    Memory freed by expert pruning has two high-value uses: supporting longer contexts and running an auxiliary draft model. The basic idea of speculative decoding is that a draft model first quickly proposes several future tokens, and the main model performs a parallel verification of these candidates, accepting the prefix consistent with the main model distribution, and falling back to main-model generation when inconsistency occurs. The benefit depends on draft model quality, proposal length, and verification cost; if the acceptance rate is low, verification and fallback overhead may offset the gains. Therefore, the draft model should be close in style to the main model, proposal length should be tuned according to business workload, and acceptance rate, average generation length, and tail latency should be monitored.

    The pseudocode can be expressed as:

    1. def speculative_decode(main_model, draft_model, prompt, max_new_tokens, gamma):
    2. output = prompt
    3. while len(output) < max_new_tokens:
    4. candidates = draft_model.propose(output, gamma)
    5. accepted, correction = main_model.verify(output, candidates)
    6. output += accepted
    7. if correction is not None:
    8. output += correction
    9. return output

    In deployment, a version with “about 30% fewer modules” can be used for a context tier of around 500K tokens, and a version with a higher pruning ratio can be used for a tier of around 1M tokens, provided that long-context evaluation and cache management pass. At the service layer, common methods such as continuous batching, paged KV cache management, quantization, and expert parallelism can be used, together with monitoring and alerting to observe memory watermarks, throughput, P99 latency, and error rates.

    4. Implementation Path and Risk Control

    It is recommended to proceed in six steps: 1) establish a baseline, recording quality, throughput, latency, memory, and cost per thousand tokens; 2) profile bottlenecks, distinguishing weight memory, cache memory, communication, and scheduling overhead; 3) prune in stages and perform sensitivity analysis; 4) introduce the draft model and speculative decoding, and tune proposal length; 5) conduct canary releases, gradually increasing traffic by business volume; 6) perform continuous regression and cost accounting. Risks include damage to long-tail tasks, router drift, declining draft acceptance rate, long-context cache bloat, and evaluation sets that are too narrow. Only by treating model lightweighting, inference execution, and service optimization as a combined solution can accelerator card procurement and cloud rental costs be steadily reduced while maintaining core inference quality.