Reducing LLM Inference Costs: Practices in Expert Pruning and Speculative Decoding

    As trillion-parameter Mixture of Experts (MoE) models become more prevalent, managing inference costs has emerged as a critical challenge. This article explores a practical optimization strategy that combines MoE expert pruning, draft-model speculative decoding, and long-context memory reuse. By optimizing model structure, inference execution, and service orchestration simultaneously, this approach can reduce both GPU procurement and cloud rental costs by approximately 20% while maintaining core inference quality. Readers will gain insights into routing-based pruning techniques, the integration of speculative decoding, and comprehensive cost metrics for deploying large-scale models efficiently.

    1. Background: Balancing Parameter Scale and Inference Costs

    Ultra-large-scale Mixture of Experts (MoE) models continue to break parameter size records among open-weight models. While trillions of parameters imply a vast number of expert modules, the inference phase only activates a subset of them at a time. However, VRAM usage, expert routing overhead, KV Cache management, and long-context processing still significantly drive up GPU resource requirements. For enterprises, directly running a full-scale model on high-end GPUs often leads to high procurement costs, expensive cloud rental fees, and deployment constraints.

    Recent public practices have demonstrated that for trillion-parameter MoE models, a combined optimization approach of “expert pruning + draft model speculative decoding + long-context memory reuse” can reduce GPU procurement costs by about 20% and cloud rental fees by around 20%, all while maintaining core inference quality. This allows the model to run on fewer GPUs or lower-spec hardware. The value of this approach lies not in simply swapping hardware, but in systematically reducing costs across three layers: model structure, inference execution, and service orchestration.

    2. Overall Strategy: Reduce Memory, Boost Throughput, Lower Specs

    This solution can be broken down into three layers:

    1. Model Lightweighting: Analyze MoE expert routing behavior and selectively remove expert modules that contribute little to specific tasks, thereby reducing VRAM usage and expert parallelization communication overhead.
    2. Inference Execution Optimization: Utilize the VRAM freed by pruning to run a smaller draft model. Through speculative decoding, the draft model proposes subsequent tokens in advance, and the main model verifies them in a single pass, effectively boosting decoding throughput.
    3. Service Spec Optimization: Once quality regression meets the standards, reduce the required GPU specs or card count per instance. Simultaneously, use the freed VRAM to expand the context window, enhancing usability for long documents, codebases, and knowledge bases.

    It is important to emphasize that pruning is not merely deleting experts. It is based on calibrating data to measure routing frequency, expert co-occurrence, and inter-layer sensitivity, followed by calibration and regression evaluation to ensure that core inference capabilities do not noticeably degrade. Public practices indicate that such models can still retain over 99% of the original score on high-level scientific reasoning benchmarks. This proves that “task-importance-based pruning” is far more controllable than “uniform parameter-based trimming.”

    3. Expert Pruning: From Routing Statistics to Layered Trimming

    In the forward pass of an MoE model, a router typically selects several experts for each token. The activation frequency of different experts varies greatly depending on the task. If certain experts exhibit low activation over a long period in a target business distribution, or are highly redundant with other experts, they can be removed while ensuring routing load balance.

    An actionable pruning process is as follows:

    1. # Pseudocode: Expert importance statistics and pruning mask generation
    2. expert_stats = init_stats()
    3. for batch in calibration_set:
    4. routes = model.route(batch) # Get token-to-expert routing for each layer
    5. expert_stats.update(routes)
    6. importance = score_experts(
    7. expert_stats,
    8. metrics=["activation_freq", "task_gain", "co_activation"]
    9. )
    10. keep_ratio = 0.65 # Example: retain about 65% of experts
    11. mask = select_experts(importance, keep_ratio, per_layer=True)
    12. pruned_model = apply_mask(model, mask)
    13. calibrate(pruned_model, calibration_set)
    14. evaluate(pruned_model, eval_suites)

    Key considerations include:

    • Layered Retention Rates: Shallow layer routing affects general representations, while deeper experts are more task-specific. A uniform trimming ratio is not advisable.
    • Task Importance Scoring: Do not rely solely on activation frequency; also consider the impact on target benchmarks after removal.
    • Router Recalibration: After pruning, the output dimensions of the router change, requiring short-phase calibration to restore load balance.
    • Fallback Mechanism: Retain the original model or a version with a higher retention rate for long-tail tasks to avoid “cutting costs at the expense of quality.”

    According to public practices, a version with a 30% reduction in modules can handle context lengths of hundreds of thousands of tokens, while a version with a nearly 40% reduction can process millions of tokens. The logic is straightforward: fewer expert modules free up VRAM, which can then be used to expand the KV Cache or increase concurrent batches, thereby extending context capabilities.

    4. Draft Model and Speculative Decoding: Turning Saved Memory into Speed

    The basic idea of speculative decoding is “small model proposes, large model verifies.” The draft model quickly generates several candidate tokens, and the main model verifies them in parallel during a single forward pass. It accepts the matching parts and falls back upon rejection. For large models whose decoding is limited by memory bandwidth, speculative decoding can significantly increase the number of output tokens per unit of time.

    1. # Pseudocode: Speculative decoding loop
    2. prefix = input_tokens
    3. while not finished(prefix):
    4. draft_tokens = draft_model.generate(prefix, k=4)
    5. accepted = main_model.verify(prefix, draft_tokens)
    6. prefix.extend(accepted)

    The VRAM freed by pruning can perfectly host the draft model without requiring additional GPUs. From an engineering perspective, attention should be paid to:

    • Distribution Match between Draft and Main Models: A higher acceptance rate leads to more obvious acceleration; a low acceptance rate increases verification overhead.
    • Candidate Length k: If k is too large, it leads to invalid verification; if too small, the acceleration is limited. It should be determined based on business load stress testing.
    • Batching and Concurrency: Speculative decoding alters the VRAM-to-compute ratio, requiring batch size retuning.
    • Quality Assurance: The final output is still verified by the main model, so the main model’s distribution theoretically remains unchanged. However, an improperly chosen draft model can affect latency stability.

    Public practices show that faster response generation is achievable even on a limited number of GPUs. This indicates that cost reduction and speed enhancement are not mutually exclusive: pruning reduces VRAM pressure, while speculative decoding improves decoding efficiency. Together, they optimize the cost per token.

    5. Cost Metrics: Look Beyond the GPU Unit Price

    When enterprises evaluate such solutions, they should establish unified metrics:

    1. Cost per million tokens =
    2. GPU hourly cost / Tokens generated per hour
    3. + Scheduling and storage costs / Total tokens

    A 20% reduction in procurement and rental costs is only a surface-level metric. More attention should be paid to:

    • Whether single-instance throughput has decreased;
    • Whether Time-to-First-Token (TTFT) and decoding latency meet SLAs;
    • Whether VRAM remains stable in long-context scenarios;
    • The quality loss of the pruned version on target business evaluation sets;
    • Whether running parallel dual versions increases operational complexity.

    6. Implementation Suggestions and Risk Control

    It is recommended to proceed in the following order: “Build evaluation sets -> Analyze routing -> Small-scale pruning -> Calibration and regression -> Speculative decoding stress testing -> Spec downgrade validation.” Risks primarily stem from long-tail task degradation, routing imbalance, draft model acceptance rate fluctuations, and long-context VRAM fragmentation. A fallback path to the original model should be maintained, with continuous regression testing on core scenarios such as scientific reasoning, code generation, and long-document QA.

    Overall, cost reduction for inference with ultra-large-scale MoE models is shifting from single-model compression to a systematic engineering approach encompassing “model lightweighting + inference execution optimization + service spec optimization.” If enterprises can combine expert pruning and speculative decoding under the premise of measurable quality and reversible risks, they have the opportunity to utilize existing AI infrastructure more cost-effectively.