Cutting LLM Inference Costs: From Model Pruning to GPU Spec Selection
This article explains how to reduce large language model inference costs through four practical approaches: MoE expert pruning, speculative decoding with draft models, context length and memory budgeting, and GPU specification selection. It covers the relationship between parameter scale and hardware cost, memory estimation formulas, acceleration ratio analysis, and the trade-offs to consider when combining techniques—helping engineers make cost-effective deployment decisions without sacrificing acceptable output quality.
1. Background: How Parameter Scale Translates into Hardware Cost
The cost of LLM inference mainly comes from memory and compute. Model weights must stay resident in GPU memory, and their footprint can be estimated with a rough formula:
Weight memory ≈ parameter count × bytes per parameter
Take a 100-billion-parameter model stored in 16-bit floating point format, where each parameter occupies 2 bytes: the weights alone require roughly 200 GB of memory. This does not include intermediate activations or the KV Cache. When a single GPU cannot hold the model, it must be sharded across multiple GPUs, and the GPU count directly determines the baseline cost of purchasing or renting hardware.
Mixture-of-Experts (MoE) architectures deserve special attention. They activate only a small subset of experts for each token, so the per-token compute does not scale with total parameter count. However, all expert weights must still reside in memory, meaning total parameter count sets the lower bound on memory requirements. Even when compute is modest, a large total parameter count demands substantial memory resources.
Inference cost can be broken down into three factors:
- Hardware unit price: procurement or rental cost per GPU-hour
- Footprint: the number of GPUs required for deployment
- Effective throughput: the number of tokens stably produced per unit of time
Cost reduction, at its core, means lowering the first two factors or improving the third—while keeping quality acceptable.
2. Path One: Expert Module Pruning
Pruning removes structural components that contribute little to the target task. In MoE models, the most direct targets are expert modules.
2.1 Basic Idea
The router selects a set of experts for each input token. If certain experts are rarely selected on your target business data, or their removal causes minimal output change, they become candidates for pruning.
2.2 Implementation Steps
- Build a calibration dataset that covers typical inputs from the target business.
- Measure each expert’s activation frequency and its contribution to the output.
- Rank experts by contribution from lowest to highest and remove them in batches.
- After each batch, run regression evaluations on both a general-capability benchmark set and a business-capability benchmark set.
- Determine the pruning ratio, keeping the most compressed version that still meets performance requirements.
Illustrative pseudocode:
def score_experts(model, calib_data):freq = [0] * num_expertscontrib = [0.0] * num_expertsfor batch in calib_data:routing = model.router(batch) # expert indices selected per tokenfor expert_id in routing.flatten():freq[expert_id] += 1for expert_id in range(num_experts):contrib[expert_id] += measure_output_change(model, batch, expert_id)return rank_ascending(freq, contrib) # sorted from lowest to highest contribution
2.3 Risks
Pruning is most sensitive to long-tail capabilities. Stable performance on a particular general benchmark does not guarantee stability in low-frequency domains (such as low-resource languages or terminology-dense text). Evaluation sets should therefore be split by domain, and a held-out set that does not participate in calibration should be reserved for final acceptance testing.
3. Path Two: Draft Models and Speculative Decoding
Autoregressive generation produces only one token per step. During decoding, GPUs are often bottlenecked by memory bandwidth, leaving compute units underutilized. Speculative decoding exploits this: a small draft model first proposes k candidate tokens at once, and the large model verifies them in parallel.
3.1 Workflow
- The draft model autoregressively generates k candidate tokens.
- The large model runs a single forward pass over these k positions to obtain a probability distribution at each one.
- Using rejection sampling, each candidate token is accepted or rejected; the sequence is truncated at the first rejection.
- The accepted tokens are emitted, the large model adds one new token, and the next round begins.
3.2 Estimating the Speedup
Assume each draft token is accepted with probability α, positions are independent and identically distributed, and each round proposes k tokens. The expected number of tokens produced per round is:
E = (1 − α^(k+1)) / (1 − α)
For example, with α = 0.8 and k = 4, each round produces about 3.36 tokens on average. The acceptance rate α is the key variable determining the payoff. When acceptance is low, the draft model’s overhead cancels out the gains—or even slows the system down overall. When choosing a draft model, weigh both its acceptance rate and its per-step latency, rather than parameter count alone.
3.3 Memory Cost
The draft model itself consumes memory and competes with the main model for compute resources. Stacking the two techniques only pays off when the memory freed by pruning is enough to accommodate the draft model and its cache.
4. Path Three: Context Length and Memory Budgeting
KV Cache size grows linearly with sequence length and is the dominant memory cost in long-context scenarios. It can be estimated as:
KV Cache memory ≈ 2 × layers × KV heads × head dimension × sequence length × batch size × bytes per element
Consider an illustrative configuration: 60 layers, 8 KV heads, head dimension 128, 2 bytes per element, batch size 1. At a sequence length of 128K tokens, the KV Cache takes roughly 30 GB; extending to 1 million tokens brings it to roughly 240 GB. These figures are for illustrating orders of magnitude only and do not correspond to any specific production model.
The takeaway: every doubling of context length roughly doubles memory usage. Memory saved through pruning and weight compression can be redirected toward longer contexts, larger batch sizes, or fewer GPUs. Which direction to choose depends on your workload’s concurrency and input length distribution.
