LLM Inference Cost Reduction in Practice: Pruning, Speculative Decoding, and GPU Selection

    This article presents a practical, measurement-driven workflow for reducing LLM inference costs after deployment. It covers expert pruning for MoE models, speculative decoding, KV-cache-based long-context memory planning, GPU downgrade economics, and serving-layer scheduling, with validation steps, formulas, pseudocode, and a launch checklist.

    LLM Inference Cost Reduction in Practice: Pruning, Speculative Decoding, and GPU Selection

    After an LLM goes live, inference cost often becomes more persistent than training cost. A mixture-of-experts (MoE) architecture lets each token activate only a small number of experts, so per-token compute is controllable, but all expert weights still need to remain resident in GPU memory. As a result, memory is often the first barrier in inference deployment. This article follows an implementation order: expert pruning, speculative decoding, long-context memory planning, GPU downgrades, and serving-layer scheduling, with validation at each step. The numbers below are given as formulas or examples; actual values must be re-measured with your own model and real traffic.

    1. Where Costs Come From

    Weight memory can be roughly estimated as “number of parameters multiplied by bytes per parameter.” With BF16, each parameter takes 2 bytes. A model with 2.8 trillion total parameters needs about 5.6 TB of memory for weights alone. This does not include KV cache, activations, or framework overhead.

    INT8 or INT4 quantization can reduce weight size to half or one quarter of the original, but quantization introduces accuracy loss and must be evaluated separately on the target task.

    Sparse activation in MoE models reduces per-token compute, but it does not reduce weight storage unless experts are offloaded to memory or disk. Therefore, reducing memory usage usually requires pruning, quantization, or more GPUs, and these approaches often need to be combined.

    2. Expert Pruning: Measure First, Then Cut

    The core of expert pruning is removing experts that contribute less to the target business. The recommended implementation steps are as follows.

    1. Collect calibration data: Calibration data should be sampled from real business traffic, not a generic corpus. The data must cover all typical request types; otherwise, the measured expert importance will deviate from actual usage.
    2. Measure expert contribution: Record each expert’s activation frequency, sum of routing weights, and output norm. Looking only at activation frequency can lead to deleting low-frequency but critical experts, such as experts that handle a specific language or code syntax.
    3. Remove experts and fix routing: After experts are deleted, the router output dimension changes. Top-k weights need to be renormalized, and load balance across experts must be checked. Load concentration on a few experts reduces throughput and may cause out-of-memory errors on individual GPUs.
    4. Evaluate across scenarios: Use the same decoding parameters before and after pruning, and compare both your own business metrics and public benchmarks. The acceptable quality-drop threshold should be set before pruning, not explained after results are available.

    There is no universal safe pruning ratio. A more reliable approach is to sweep ratios, for example 10%, 20%, and 30%, and choose the largest ratio that meets the quality threshold.

    The following pseudocode contains the main logic for these steps.

    1. def prune_experts(model, calib_data, prune_ratio, eval_fn, max_drop):
    2. stats = collect_expert_stats(model, calib_data) # activation frequency, routing weight, and output contribution
    3. num_remove = int(model.num_experts * prune_ratio)
    4. # Rank by frequency and contribution together to avoid deleting critical low-frequency experts
    5. ranked = sorted(stats, key=lambda s: s.frequency * s.avg_weight * s.output_norm)
    6. removed = [s.expert_id for s in ranked[:num_remove]]
    7. pruned = remove_experts(model, removed)
    8. pruned = recalibrate_router(pruned, calib_data) # fix top-k normalization and check load balance
    9. baseline_score = eval_fn(model)
    10. pruned_score = eval_fn(pruned)
    11. if baseline_score - pruned_score > max_drop:
    12. raise ValueError("Quality drop exceeds threshold; reduce pruning ratio")
    13. return pruned

    3. Speculative Decoding: Use Memory to Reduce Serial Steps

    Memory freed by pruning can be used to load a small draft model. The draft model first generates K candidate tokens consecutively, and the main model then uses one forward pass to produce predictions for those K positions at the same time.

    The verification rule is as follows. Under greedy decoding, starting from the first position, accept draft tokens as long as they match the main model’s predictions. Stop at the first mismatch and use the main model’s prediction at that position. If all K tokens are accepted, the main model also provides the prediction for the (K+1)-th position. Therefore, each round produces at least one new token, and the output is identical to decoding with the main model alone. When sampling is used, rejection sampling must be applied to preserve the distribution.

    Speedup depends on two factors. The first is the acceptance rate α, the probability that a draft token is accepted. The second is the relative overhead of the draft model. Under the simplifying assumption that acceptance events are independent, the expected number of tokens produced per round is approximately (1−α^(K+1))/(1−α). For example, when α is 0.8 and K is 4, the expected value is about 3.4 tokens. Actual acceptance rates usually vary by text type; code and templated text tend to have higher acceptance rates, while open-ended writing tends to have lower rates.

    Speculative decoding is not free. Each verification round processes K+1 positions, which increases per-round compute. It reduces serial steps, but decoding speed is often limited by memory bandwidth, so the benefit must be measured. The draft model and the main model must use the same vocabulary.

    1. def speculative_decode(main_model, draft_model, input_ids, max_new_tokens, K=4):
    2. tokens = list(input_ids)
    3. target_len = len(tokens) + max_new_tokens
    4. while len(tokens) < target_len:
    5. draft = draft_model.generate(tokens, num_tokens=K) # draft model generates K candidates
    6. # main model computes predictions for K+1 positions in one forward pass
    7. main_preds = main_model.predict_next_tokens(tokens, draft)
    8. n_accept = 0
    9. for i in range(K):
    10. if draft[i] == main_preds[i]: # greedy decoding match check
    11. n_accept += 1
    12. else:
    13. break
    14. tokens += draft[:n_accept]
    15. tokens.append(main_preds[n_accept]) # main model adds one token whether or not all are accepted
    16. return tokens[:target_len]

    4. Long Context: Calculate Memory with Formulas, Not Marketing Numbers

    Whether context length can be extended depends on whether the KV cache fits in memory. The KV cache size for a single sequence can be estimated as:

    2 × layers × KV heads × head dimension × sequence length × bytes per element × concurrency

    Take a hypothetical configuration: 60 layers, 8 KV heads, head dimension 128, and BF16 precision. The KV cache per token is about 2×60×8×128×2 bytes, or 245,760 bytes. For a sequence length of 512K tokens, the KV cache for a single request is about 126 GB, before accounting for paging fragmentation and framework overhead. This number shows that even when weights fit, long context can still become a bottleneck.

    Therefore, when evaluating how long a context a pruned version can support, state the following conditions together: base model and weight precision, KV cache precision, total GPU memory, number of concurrent requests, and per-request length distribution. Conclusions such as “supports one million tokens” without these conditions cannot be used for capacity planning. Before launch, run stress tests at maximum concurrency and maximum length to confirm there is no OOM.

    5. GPU Downgrades: Use Cost per Successful Request as the Metric

    After model lightweighting, consider migrating deployment to lower-cost GPUs. Before selecting hardware, calculate three things.

    First, the memory budget per GPU. The memory budget equals total memory multiplied by the usable ratio, minus weight shards, peak KV cache, draft model, and framework overhead. If the budget is insufficient, deployment will fail even with the same number of GPUs.

    Second, effective throughput per GPU. Throughput must be measured under the target batch size and target latency constraint, not inferred from peak compute. Sub-flagship GPUs usually have lower memory bandwidth, which has a greater impact on decoding throughput.

    Third, cost per successful request. The formula is: instance hourly price divided by the number of successfully completed requests per hour. Comparing GPU prices alone does not reflect reduced card count, changes in communication overhead, or throughput loss.

    Some proposals report percentage reductions in purchase or rental cost. These percentages depend on hardware quotes, cloud provider pricing, lease term, and load curve, and cannot be applied directly. Recalculate them with your own quotes.

    6. Serving-Layer Coordination: Scheduling Determines Resource Utilization

    Once the model and hardware are fixed, serving-layer scheduling has a major impact on utilization. Common practices include the following.

    • Continuous batching: Dynamically add requests to a batch as they arrive to reduce GPU idle time.
    • KV cache paging: Allocate memory in blocks to reduce fragmentation and support prefix reuse.
    • Request routing: Put long-context requests and short requests into different queues so long requests do not occupy batches for long periods and slow down short requests.
    • Priority and timeouts: Set latency limits for different businesses and release resources promptly when requests time out.

    The following is an illustrative configuration. Replace the values with stress-test results.

    1. service:
    2. model:
    3. path: /models/pruned-moe-model
    4. prune_ratio: 0.2 # use the maximum ratio that passes evaluation
    5. max_context_length: 131072 # match the length validated by stress testing
    6. kv_cache_dtype: fp8 # evaluate accuracy loss separately
    7. draft_model:
    8. enabled: true
    9. path: /models/draft-model-small
    10. num_speculative_tokens: 4
    11. scheduling:
    12. continuous_batching: true
    13. max_batch_size: 32 # use the largest value that meets the p99 latency target in stress testing
    14. kv_cache_paging: true
    15. long_request_queue: separate
    16. hardware:
    17. gpu_type: "sub-flagship GPU"
    18. gpu_count: 8

    7. Launch Checklist

    Before replacing the online model, confirm each item below.

    1. Calibration data comes from recent real traffic and covers the main request types.
    2. Quality metrics before and after pruning are compared under the same decoding parameters, and the gap is within the pre-agreed threshold.
    3. Routing load is balanced, and no single expert handles an excessively high share of requests over time.
    4. Speculative decoding acceptance rate is measured separately by business type, and the draft model can be disabled automatically when acceptance is too low.
    5. At peak batch size and maximum context length, peak memory has headroom, and OOM alerts are configured.
    6. Gray release is staged, for example switching 5% of traffic first and then expanding to 100%.
    7. The original model remains online for fast rollback. Delete old weights only after the new version is confirmed stable.
    8. Monitoring includes p95 and p99 latency, successful requests per second, cost per successful request, and error rate.

    Conclusion

    Reducing LLM inference cost requires measurement before deciding on pruning ratios, draft models, and hardware specifications. A recommended order is: first, collect one week of real traffic as a calibration set; second, sweep pruning ratios to find the largest ratio within the quality threshold; third, measure draft-model acceptance by business type to decide whether speculative decoding is worth keeping; fourth, compare GPU options using cost per successful request. Each step should be reproducible so that future model updates and hardware changes have a reliable basis.