Cut LLM Inference Costs by 20%: Expert Pruning and Speculative Decoding in Practice

    This article walks through a real case in which a team optimized inference for a sparse ~2.8-trillion-parameter MoE model, cutting GPU procurement costs by up to about 20% and cloud GPU rental costs by up to about 22% while keeping the model above 99% of its original score on a hard scientific-reasoning benchmark. Rather than promising universal numbers, it explains the general method behind the result: a cost model built around memory, the prefill/decode phases, and per-token cost; two core techniques — progressive expert pruning and migrating to lower-cost, adequate hardware; and turning freed memory into speed and long-context capability through speculative decoding and memory rearrangement. It also covers the five risks of moving to cheaper hardware, a five-stage enterprise deployment roadmap, a pre-launch checklist, and common pitfalls to avoid.

    1. Starting from a Cost-Reduction Case

    In November 2025, a technical team focused on AI inference optimization publicly disclosed that it had completed inference optimization for a sparsified large model with roughly 2.8 trillion total parameters. After optimization, the model could run on accelerator cards that were both cheaper and a tier lower in specification. Public information showed that GPU procurement costs dropped by up to about 20%, and GPU rental costs in cloud environments dropped by up to about 22%. At the same time, the model still scored above 99% of the original on a hard scientific-reasoning benchmark. Earlier, the team had compressed the number of next-generation accelerator cards needed to run models in the same series from 8 down to at most 4; this time it went further and reduced the hardware specification tier.

    The reason this result is worth enterprise developers’ attention is not the size of the numbers, but the cost-reduction path: the drop in hardware specification is traded for model-side lightweighting, while acceleration and long-context capability reuse the hardware that was freed up. This is fundamentally different from the “just pile on more, newer, more expensive cards” approach.

    One caveat first: the 20% and 22% figures are public results under a specific model, specific hardware, and specific optimization combination — not a universal promise. Different model sparse structures and different input/output distributions can change the gains significantly. The goal of this article is to explain the general method behind the case: how to compress memory without sacrificing quality, how to quantify cost reduction, and what verification is required before going live.

    2. The Cost Model: What Determines the Inference Bill

    1. Memory Determines “How Many Cards You Need at Minimum”

    When deploying a large model, a single accelerator card’s memory must hold four kinds of data:

    Occupancy item Scale Notes
    Model weights parameters × bytes per parameter FP16/BF16 = 2 bytes, FP8 = 1 byte, INT4 ≈ 0.5 byte
    KV cache proportional to batch size × context length stores the Key/Value of historical tokens; the main overhead for long context and multi-turn dialogue
    Runtime overhead usually part of the reserved memory framework, communication buffers, activations, temporary tensors
    Draft model depends on implementation if using speculative decoding, requires extra model weights and KV space

    Key formula:

    1. N (required cards) ≥ weight memory ÷ (per-card memory × target utilization)

    Take a 1-trillion-parameter model as an example: at FP8 the weights are about 1 TB, at INT4 about 0.5 TB. With roughly 96 GB per card and a target utilization of 85%, usable space is about 82 GB. Weights alone require at least 13 cards (INT4) or 7 cards (FP8), and the KV cache must be added on top of that.

    This also explains why parameter count is the primary cost constraint: the more parameters, the larger the weights, the more cards you need — and the less room is left for the KV cache.

    2. Two Phases Determine “How Fast”

    • Prefill: processes all input tokens in one pass, billed by matrix-multiply compute. It is compute-intensive and directly determines time-to-first-token.
    • Decode: generates output tokens one at a time; each step must read the currently active parameters from memory. It is bandwidth-intensive and determines output tokens per second.

    In the Decode phase, sparse models activate only a few experts per token, so the compute is small, and the bottleneck is often memory access rather than compute. Optimization therefore usually focuses on: reducing weight size, speeding up weight reads, and merging steps.

    3. Unit Cost Determines “Whether It’s Worth It”

    1. Per-token cost = (number of cards × price per card-hour) ÷ (effective output tokens per second × 3600)

    The ultimate goal of every optimization is to lower this ratio. The approach in this case maps neatly onto three levers:

    1. Reduce cards and specification: prune weights to lower both the card count and the per-card tier;
    2. Swap hardware: move from “the newest generation” to a generation that is “adequate in performance, lower in unit price”;
    3. Raise throughput: use the freed memory to deploy a draft model and increase per-card output.

    3. Core Method 1: Expert Pruning, Cutting Low-Contribution Experts

    1. Why Sparse Models “Can Be Pruned”

    A sparsified large model (i.e., MoE, Mixture-of-Experts) is made up of many parallel expert feedforward modules and a router. Each token is routed to only a few experts (e.g., top-2); the rest take no part in that forward pass. This design makes “large total parameter count but small per-token compute” possible. But it also brings two problems:

    • Expert utilization is uneven over the long run: some experts are called frequently while others sit nearly idle, so redundancy exists.
    • Weights must still stay resident in memory: whether or not an expert is currently called, its parameters must be loaded onto the cards, otherwise routing cannot complete in milliseconds.

    So pruning is especially effective on MoE: it reduces both “the weights that must be loaded” and “the card slots that must be occupied.”

    2. How to Measure Importance

    Pruning decisions cannot rely on intuition; you first need to quantify each expert’s real contribution to the model output. Three metrics are commonly used:

    1. Routing frequency: how many times an expert is assigned tokens;
    2. Sum of routing weights: the total weight the router assigns to that expert;
    3. Task contribution: on a business validation set, evaluate the change in output quality after removing the expert.

    The first two only require a single forward pass over a calibration dataset to collect. The third is the most accurate but requires repeated inference, so it is more expensive. In practice, routing statistics are usually used to filter out candidates first, then the business set is used to validate.

    1. # Pseudocode: a framework-agnostic flow for expert pruning; specific tensor APIs vary by inference framework
    2. def prune_experts(model, calibration_data, prune_ratio, granularity="group"):
    3. """
    4. model: an MoE model containing a router
    5. calibration_data: calibration samples covering the real business distribution
    6. prune_ratio: target pruning ratio
    7. granularity: pruning granularity — remove by "expert group" or "intra-layer expert"
    8. """
    9. counts = [defaultdict(int) for _ in model.layers] # per layer: number of times an expert is assigned
    10. for batch in calibration_data:
    11. for layer_idx, layer in enumerate(model.layers):
    12. # run forward up to this layer and take the router's assignment
    13. assign = layer.router(batch) # shape is roughly [tokens, experts_in_this_layer]
    14. for expert_idx, n in batch_experts(assign):
    15. counts[layer_idx][expert_idx] += n
    16. total = sum(sum(c.values()) for c in counts)
    17. scores = []
    18. for layer_idx, c in enumerate(counts):
    19. layer_total = sum(c.values())
    20. for e, n in c.items():
    21. # combine the global share with the intra-layer share, so that local experts aren't emptied out and the layer's structure isn't broken
    22. scores.append((n / total + n / layer_total, layer_idx, e))
    23. scores.sort(reverse=True)
    24. survivors = [[] for _ in model.layers]
    25. keep_per_layer = int(round((1 - prune_ratio) * model.experts_per_layer))
    26. for _, layer_idx, e in scores:
    27. if len(survivors[layer_idx]) < keep_per_layer:
    28. survivors[layer_idx].append(e)
    29. # rebuild a new model from the surviving expert submodules
    30. return rebuild_model(model, survivors)

    3. Pruning Must Be “Progressive”

    Pruning away a large number of experts at once changes the routing distribution abruptly, and model output quality drops noticeably. A safer approach is progressive pruning:

    1. First prune a small ratio of 5%–10% and observe the change in quality;
    2. Fine-tune with retained training or distillation data so the model re-adapts to the new expert topology;
    3. Gradually increase the ratio until the quality curve shows a clear “inflection point”;
    4. Use the ratio just before the inflection point as the launch configuration, leaving a safety margin.

    In the public case, the pruning ratio reached up to 37%, which shows that MoE models do carry considerable redundancy and confirms that “the relationship between expert count and effective capacity is not linear.”

    4. After Pruning, Be Sure to Align the Structure

    • Align the count with the parallelism degree: the number of experts should be divisible by the parallelism degree. If you deploy with 8-way expert parallelism, the number of experts at each position should ideally be a multiple of 8; otherwise some cards carry a few extra experts (load skew), and overall throughput is dragged down by the slowest card.
    • Keep the router learnable: pruning is a hard structural constraint. It is recommended to keep training the router during fine-tuning so it learns to assign tokens under the new topology.
    • Use a unified ratio across layers: allocate the pruning quota per layer, and avoid emptying out any single layer. Otherwise every token in that layer can only go through the remaining experts, and model capacity drops sharply in that spot.

    5. After Pruning, Do “Restorative” Fine-Tuning

    • Continue training or knowledge distillation on the pruned model with high-quality data;
    • Use the original model as the teacher and have the student model (the pruned one) align its outputs on the business distribution;
    • Fine-tuning objectives should include not only the loss value but also “output-distribution consistency,” to prevent degenerate behavior on extreme inputs.

    4. Core Method 2: Switching Hardware from “Most Powerful” to “Good Enough”

    1. The Core Metric for Choosing a Hardware Generation: Price-Performance

    Pruning reduces weight size, which in turn lowers the requirements on per-card memory and compute. This means the model has a chance to migrate from a new-generation accelerator to a previous-generation card. The question to answer here is: after swapping cards, does per-token cost go up or down?

    The example table below illustrates the decision logic (all values are hypothetical, used only to demonstrate the calculation method):

    Scheme Accelerator Prune ratio Spec decoding Per-card effective output (token/s, assumed) Price per card-hour (relative, assumed) Cards Relative unit cost
    A Baseline new-gen 0% off 1000 1.00 8 100
    B Card swap only previous-gen 0% off 800 0.80 8 100
    C Pruning only new-gen 30% off 1300 1.00 8 77
    D Spec decoding only new-gen 0% on 1600 1.00 8 63
    E Combined previous-gen 37% on 1600 0.80 8 50

    Reading the results:

    • Scheme B: card price drops 20%, but per-card throughput also drops 20%, so unit cost stays flat — swapping cards accomplishes nothing;
    • Scheme C: after pruning, each step reads fewer weights. Decode is bandwidth-intensive, so per-card throughput rises significantly;
    • Scheme D: speculative decoding lets the main model verify multiple tokens at once, raising output speed;
    • Scheme E: with the three methods stacked, unit cost is only half the baseline.

    This also explains the difference between “procurement down ~20%” and “cloud rent down ~22%.” Cloud GPU pricing usually bundles compute, memory, and bandwidth, and further stacks on data-center, power, and operations costs, so the rental discount on an older generation may be slightly larger than the pure hardware procurement discount. But whether you are buying or renting, you should look not at the unit price alone, but at price-performance = effective throughput ÷ card-hour cost.

    2. Five Risk Points When Migrating to Cheaper Hardware

    1. Numerical precision and operator support: new-generation cards support low-precision formats such as FP8/FP4 more completely. A previous-generation card may support some formats poorly or lack efficient kernel implementations, so actual throughput falls below the theoretical value.
    2. Memory and bandwidth gap: with smaller per-card memory, the same weights occupy more cards; inter-card communication overhead rises, and the parallel splitting strategy needs to be redesigned.
    3. Chain costs of more cards: the failure domain grows, scheduling gets more complex, and the interconnect topology has a bigger impact on performance. Once the card count exceeds single-node scale, cross-node communication slows down Decode significantly.
    4. Energy efficiency and cooling: older-generation cards usually have lower energy efficiency. Procurement cost drops, but data-center power and cooling overhead may rise, so TCO may not fall year over year.
    5. Long-tail load: after the swap, P99 latency may worsen. It is recommended to use “P50 / P95 / P99 all meeting targets” as the acceptance standard, rather than looking at the average only.

    3. Do Compatibility Verification in Advance

    Before going live, it is recommended to run a “scaled-down” stress test in this order:

    1. Run real business inputs on the target hardware and confirm the output matches the reference hardware within an acceptable error;
    2. Use a stress-testing tool to measure throughput and latency under a fixed input length;
    3. Gradually increase concurrency and observe whether throughput scales linearly and whether memory overflows;
    4. Confirm that the quantization configuration, operator version, and driver are all available on the target machine.

    If the business load has clear peaks and valleys, you can also combine “reserved (subscription) + on-demand” instances: cover baseline traffic with reserved capacity and elastic on-demand capacity for the peaks, which further lowers overall cost.

    5. Turning Freed Memory into Speed: Speculative Decoding

    1. Principle: Draft First, Then Verify

    The memory freed up by pruning should not sit idle; it can be put to two uses:

    1. Expand the KV cache: to support longer context and higher concurrency;
    2. Load a draft model: to raise output speed together with speculative decoding.

    The speculative decoding flow is as follows:

    1. The draft model generates γ candidate tokens in a row;
    2. The main model does one parallel forward pass over these γ+1 positions, obtaining the output distribution at each position;
    3. Compare one by one: keep the accepted ones; at the first rejected token, resample from the main model’s distribution and discard all subsequent candidates;
    4. Repeat the steps above.
    1. # Pseudocode: the core loop of speculative decoding
    2. def speculative_decode(prompt, main_model, draft_model, gamma, max_new_tokens):
    3. ctx = main_model.prefill(prompt) # main model prefill
    4. dctx = draft_model.prefill(prompt) # draft model prefill, kept in sync
    5. out = []
    6. while len(out) < max_new_tokens:
    7. draft = [draft_model.sample(dctx) for _ in range(gamma)] # draft model guesses gamma tokens in a row
    8. logits = main_model.forward(ctx, draft) # main model verifies gamma+1 positions in parallel
    9. accepted = 0
    10. for i, d in enumerate(draft):
    11. if logits[i].argmax() == d: # strict match (greedy decoding)
    12. out.append(d)
    13. accepted += 1
    14. else:
    15. out.append(logits[i].sample()) # at the first mismatch, resample from the main model distribution
    16. break
    17. else:
    18. out.append(logits[gamma].sample()) # all accepted, then get one extra token for free
    19. ctx = main_model.append(out[-accepted:]) # update the main model KV
    20. dctx = draft_model.append(out[-accepted:]) # keep the draft model KV in sync
    21. return out

    2. Key Property: The Output Distribution Is Unchanged

    Under a standard implementation, the output distribution of speculative decoding is exactly the same as decoding with the main model alone (identical results under greedy decoding). This means it is a lossless speedup — unlike quantization and pruning, it does not require re-running quality evaluations.

    3. Benefit Formula and Quantification

    Let α be the acceptance probability of draft tokens (the acceptance rate), and let the draft model’s cost of generating one token be c times one forward pass of the main model. The expected number of tokens produced per round is:

    1. E(γ) = (1 − α^(γ+1)) ÷ (1 − α)
    • When α = 1, E(γ) = γ + 1;
    • The theoretical speedup relative to standard autoregressive decoding is about E(γ) ÷ (1 + γ·c), and the actual value must further deduct verification, scheduling, and other overheads.
    1. def speculative_speedup(alpha, gamma, c=0.15):
    2. # example theoretical speedup: alpha is the acceptance rate, gamma is the draft length, c is the draft overhead ratio
    3. if alpha >= 1.0:
    4. return gamma + 1
    5. expected = (1 - alpha ** (gamma + 1)) / (1 - alpha)
    6. return expected / (1 + gamma * c)
    7. for a in (0.2, 0.4, 0.6, 0.8):
    8. print(f"alpha={a}: {speculative_speedup(a, 4):.2f}x")

    Output (draft length γ = 4):

    1. alpha=0.2: 0.78x
    2. alpha=0.4: 1.03x
    3. alpha=0.6: 1.44x
    4. alpha=0.8: 2.10x

    You can see that at α = 0.2, speculative decoding actually slows generation down; at α = 0.8, the theoretical speedup can exceed 2x. So the quality of the draft model is the core variable of this whole approach.

    4. Four Factors That Determine the Acceptance Rate

    1. Capability gap between the draft model and the main model: the draft model should be small enough to “guess fast” yet large enough to “guess accurately,” usually one-tenth to one-few-dozenths of the main model’s size;
    2. Task type: highly templated tasks such as summarization, continuation, and formatted output have higher acceptance rates; open-ended reasoning and long-chain math problems lower it;
    3. Draft length: the larger γ is, the higher the theoretical benefit, but draft overhead also grows linearly, so there is an optimal value;
    4. Batch concurrency: under high concurrency, one forward pass of the main model already serves many requests at once, so the extra compute of verifying γ tokens may be amortized away, and the benefit drops or even turns negative. In practice, speculative decoding is often treated as a switch for “low-concurrency, latency-sensitive” scenarios and turned off under high concurrency.

    5. Engineering Implementation Points

    • Unified vocabulary: the draft model must share the tokenizer and vocabulary with the main model, otherwise token-by-token comparison is impossible;
    • Dynamic switch: automatically decide on/off and the value of γ based on queue length, input length, and the measured acceptance rate, to avoid consuming resources when the benefit is negative;
    • Metric monitoring: treat the acceptance rate, average tokens produced per round, and end-to-end output speed as core monitoring items;
    • Leave enough memory: the draft model’s KV cache grows with context and must be included in the memory budget to avoid OOM under long context.

    6. Where Long-Context Capability Comes From: Memory Rearrangement

    1. Context Length Is Constrained by the KV Cache

    The relationship between context length and memory usage can be approximated as:

    1. KV memory (bytes) ≈ 2 (K/V) × layers × KV heads × head dim × sequence length × batch size × bytes per element

    Taking “64 layers, 8 KV heads, head dimension 128, FP16” as an example, each token’s KV is about 256 KB; the KV for 100,000 tokens is about 25 GB, and for 1,000,000 tokens about 250 GB — and that is without counting weights and runtime overhead. So the memory freed up by pruning directly determines “how long a context you can serve.”

    In the public case, two versions with different pruning ratios supported roughly 512K and 1M token contexts respectively. This shows that context length is an engineering parameter that can be “tuned” according to pruning intensity, not a fixed property of the model. The two are not simply positively correlated; it also depends on how memory is allocated among weights, the KV cache, and the draft model.

    2. Long Context Requires Supporting Technologies

    Memory headroom alone is not enough; long-context serving also needs the following general techniques working together:

    • Paged KV cache management: split the KV into fixed-size blocks and allocate them dynamically per request to reduce fragmentation and improve memory utilization;
    • Prefix caching: requests with identical multi-turn dialogue or system prompts can reuse existing KV, avoiding repeated computation;
    • KV quantization: store the KV in FP8/INT8, cutting memory usage by nearly half;
    • Chunked prefill: split ultra-long inputs into multiple blocks processed in batches, avoiding one request monopolizing compute resources for a long time;
    • Context parallelism: distribute one sequence’s KV across multiple cards for computation.

    3. The Cost of Long Context

    Long context is not free:

    • Input tokens are billed by usage, so long-document processing costs rise linearly;
    • The longer the context, the more the model’s recall of information in the “middle position” may decline (the industry’s so-called “lost in the middle” phenomenon);
    • Long context squeezes the KV cache and lowers concurrency.

    So for knowledge-intensive businesses, you should weigh long context against retrieval-augmented generation (RAG). The former is simpler to implement and saves a retrieval step; the latter is cheaper and more scalable. A common combination in practice is to first retrieve to narrow the scope, then handle the result with long context.

    7. Enterprise Deployment Roadmap

    It is recommended to proceed through the following five stages, setting a clear “pass standard” for each to avoid launching on guesswork.

    Stage Main work Pass criteria (example)
    1. Baseline quantification Map the real business input/output distribution; measure throughput, latency, memory, and per-card cost; establish a quality baseline Key business-metric baselines are reproducible
    2. Expert pruning Collect routing distributions; do progressive pruning + fine-tuning recovery Quality drop versus baseline is within an acceptable threshold (e.g., within 1%)
    3. Hardware selection and quantization Target-hardware compatibility verification, quantization configuration tuning, stress testing Price-performance beats baseline; P95/P99 latency targets met
    4. Memory rearrangement Expand the KV cache / deploy a draft model; dynamically toggle speculative decoding by concurrency Throughput improves for long-context and low-concurrency scenarios with no regression at high concurrency
    5. Cloud servitization Containerized deployment, elastic scaling, tiered routing by SLO Per-token cost drops; SLO met

    Regarding stage 5, public information shows the company’s CEO stated that it would extend the technology from “model lightweighting” further into “inference execution and service optimization.” This is consistent with the industry trend: the gains from single-point model optimization will gradually top out, and the ultimate cost-reduction space comes from the coordination of model, hardware, and scheduling. For users, this means cloud inference services will keep getting cheaper and more flexible.

    On cloud platforms such as Baidu AI Cloud, the above capabilities can typically be achieved through a combination of general cloud products:

    • GPU cloud servers: host the inference instances;
    • Container service: handle elastic scaling and canary release;
    • Load balancing: route by tier according to input length and SLO;
    • Object storage: store model weights and archive logs;
    • Monitoring and alerting: track metrics such as GPU utilization, memory, latency, and acceptance rate.

    8. Pre-Launch Checklist and Common Pitfalls

    1. Pre-Launch Checklist

    • Has the pruning ratio been validated progressively, and does quality on the business evaluation set meet the target?
    • Is the number of experts divisible by the parallelism degree, and is the per-card load balanced?
    • Are the quantization configuration and operator version verified as usable on the target hardware?
    • Does speculative decoding meet the acceptance-rate target, and does the draft model share the vocabulary?
    • Is there switch protection for speculative decoding under high concurrency?
    • Does the long-context KV memory budget leave enough safety margin?
    • Are P95/P99 latency and per-token cost established as a baseline for comparison?

    2. Common Pitfalls

    1. Looking only at unit price, not price-performance: if cheaper hardware causes an even bigger drop in throughput, it is actually more expensive;
    2. Treating 99% as “lossless”: a single point on a hard reasoning evaluation set may correspond to a notable degradation on a different class of problems in business scenarios, so it must be validated with the business set;
    3. One-shot pruning: cutting a large number of experts at once may leave the model unable to “learn it back,” and the fine-tuning cost can even exceed the saved hardware cost;
    4. Blindly enabling speculative decoding: when the acceptance rate is below the threshold, it increases both latency and cost;
    5. Ignoring the scheduling layer: no matter how good the model optimization is, if request scheduling is unreasonable (e.g., long requests blocking short ones), the actual gains will be wasted.

    9. Summary

    The takeaway of this case for developers can be summed up in one sentence: the essence of inference cost optimization is reallocating resources within the “quality–speed–cost” triangle.

    1. Pruning compresses the weights, which buys fewer and cheaper hardware;
    2. Hardware that is adequate in performance and lower in price reduces the per-unit compute cost;
    3. The freed memory is reused through speculative decoding and long context, turning cost reduction into real capability.

    For enterprises, the deployment path is not mysterious: quantify first, then prune, then select hardware, then schedule. Every step should let data do the talking, with the business evaluation set as the backstop. As inference optimization moves from “model lightweighting” toward “servitization and platformization,” the business scale enterprises can support under the same budget will keep expanding, and the per-token cost will keep falling.