A Systematic Methodology for LLM Inference Performance Optimization
This article provides a systematic framework for LLM inference optimization. It explains why the Prefill stage is compute-bound while the Decode stage is memory-bandwidth-bound, and organizes mainstream optimization techniques into two routes: reducing the amount of data that must be moved (weight quantization, KV Cache compression and management, pruning) and making data movement and computation more efficient (kernel fusion, continuous batching, speculative decoding, tensor parallelism). It concludes with a practical, measurement-first optimization workflow for engineering teams.
A Systematic Methodology for LLM Inference Performance Optimization
1. Why Inference Optimization Is Easy to Get Lost In
In LLM engineering practice, inference optimization is a field with an extremely high density of jargon. KV Cache, FlashAttention, Speculative Decoding, Continuous Batching — behind every term lies an entire body of technical detail. Beginners can easily drown in terminology and lose sight of the big picture.
This article is written for developers who understand the basic concepts of large models, roughly know how the Transformer is structured, and have written or run model inference code — but lack a systematic understanding of “how to make inference faster.” Instead of piling up terminology, we will draw the map of inference optimization along one clear logical thread.
That thread is actually very simple: first figure out where the slowness comes from, then look at what problem every optimization technique is essentially solving.
2. Step One: Where Is the Inference Bottleneck?
2.1 Inference and Training Have Different Bottlenecks
Before discussing optimization, one premise must be clear: the performance bottlenecks of LLM inference and training are not the same thing.
The bottleneck in training is mainly compute. Backpropagation requires computing gradients for every layer, involving massive matrix multiplications. GPU compute units stay under heavy load for long periods — a classic compute-bound scenario.
Inference is more complicated and needs to be analyzed in two separate stages.
2.2 The Prefill Stage: Compute-Bound
The Prefill stage processes the user’s input prompt. All tokens of the prompt are fed into the model at once, and the hidden states and attention results for every position are computed in parallel.
Because all input tokens can be processed in parallel, the matrix multiplications are large enough to fully “feed” the GPU’s compute units. Therefore, the Prefill stage is compute-bound. Its latency is strongly correlated with prompt length and directly determines the user-perceived Time To First Token (TTFT).
2.3 The Decode Stage: Memory-Bandwidth-Bound
The Decode stage generates output token by token. For each new token, the model must run a complete forward pass based on the context of all previous tokens — and under the autoregressive mechanism, this step cannot be parallelized: the next token can only be computed after the previous one has been generated.
This means the matrix multiplications in each forward pass are very small (both the batch and sequence dimensions are close to 1), leaving most of the GPU’s compute power idle. The bottleneck shifts to memory bandwidth.
2.4 A Counterintuitive Fact: The Bottleneck Is Moving Data, Not Computing It
A simplified calculation illustrates this. Suppose an accelerator offers FP16 compute on the order of hundreds of TFLOPS, while its memory bandwidth is on the order of 2 TB/s.
A 7B-parameter model stored in FP16 occupies about 14 GB of memory. In the Decode stage, generating each token theoretically requires moving all 14 GB of weights from memory to the compute units. At 2 TB/s bandwidth, just moving the weights takes about 7 milliseconds — in other words, even with infinite compute power, the act of “reading the parameters” alone limits single-request generation speed to roughly 140 tokens per second.
In real scenarios, reading the KV Cache, kernel launch overhead, and other costs make the actual speed even lower.
This is the core tension of inference optimization: the GPU spends most of its time waiting for data, not computing on it. Once you understand this, nearly all inference optimization techniques fall into two categories:
- Reduce the amount of data that needs to be moved — quantization, pruning, KV Cache compression, etc.;
- Improve the efficiency of data movement and computation — kernel fusion, parallelism and scheduling strategies, speculative decoding, etc.
Let’s look at each route.
3. Route One: Reduce the Amount of Data to Move
3.1 Weight Quantization: The Most Direct Way to “Lose Weight”
Since the Decode bottleneck is moving weights from memory to compute units, making the weights smaller is the most direct and effective optimization.
Quantization compresses FP16 weights (2 bytes per parameter) into INT8 (1 byte) or even INT4 (0.5 byte). For a 7B model:
- FP16: about 14 GB
- INT8: about 7 GB, halving the theoretical bandwidth cost
- INT4: about 3.5 GB, reducing the theoretical bandwidth cost to one quarter
In the bandwidth-bound Decode stage, weight-read time is approximately proportional to data volume, so low-bit quantization often yields near-linear throughput gains. The trade-off is precision loss. In practice, techniques such as group-wise quantization and scale-factor calibration keep the loss within acceptable bounds. A common practical conclusion: INT8 quantization has minimal accuracy impact on most tasks, while INT4 requires more careful evaluation.
3.2 KV Cache Compression and Management
Beyond weights, the other growing memory cost in the Decode stage is the KV Cache — the Key and Value vectors cached for every historical token in every attention layer. The longer the sequence and the higher the concurrency, the more significant the KV Cache footprint becomes, sometimes exceeding the model weights themselves.
Optimization directions for the KV Cache include:
- KV Cache quantization: compressing the cached K/V to INT8 or lower precision, directly reducing the data read at each step;
- Attention architecture improvements: structures such as multi-query attention and grouped-query attention let multiple attention heads share K/V, reducing cache size at the architectural root;
- Paged memory management: borrowing the paging idea from OS virtual memory, the KV Cache is allocated and reclaimed in blocks, significantly reducing memory fragmentation. This allows larger batch sizes within the same memory budget — and the larger the batch, the more requests amortize each weight read, lowering the bandwidth cost per token.
3.3 Pruning and Sparsity
Pruning directly shrinks the model by removing weights or structures (such as attention heads or FFN channels) that contribute little to the output. Structured pruning produces a genuinely smaller dense model with clear benefits, but it usually requires retraining or fine-tuning to recover accuracy. The engineering cost is high, so it is less widely adopted in practice than quantization.
4. Route Two: Make Computation and Data Movement More Efficient
4.1 Kernel Fusion: Reducing Round Trips of Intermediate Results
A single Transformer forward pass consists of many fine-grained operators (normalization, activation, matrix multiplication, attention, etc.). If every operator writes its intermediate result back to memory and the next one reads it out, memory bandwidth is wasted on the round trips of “intermediate data.”
Kernel fusion merges multiple adjacent operators into a single GPU kernel, keeping intermediate results in on-chip cache and avoiding trips to memory. Attention computation is a typical example: a naive implementation must write the full attention score matrix to memory, whereas a fused, block-wise implementation can complete the entire softmax and weighted-sum process on chip — saving both memory and bandwidth, with especially significant gains for long sequences.
4.2 Batching and Scheduling: Amortizing the Weight-Read Cost
Weight reading in the Decode stage is a fixed cost, so the natural strategy is to make each weight read serve as many tokens as possible. Key techniques include:
- Continuous batching: traditional static batching must wait for the longest request in a batch to finish before releasing resources, leading to low utilization. Continuous batching schedules at token granularity — as soon as one request finishes, a new one takes its place — keeping the GPU highly utilized and improving throughput severalfold;
- Disaggregated Prefill/Decode scheduling: the two stages have different resource profiles (one is compute-hungry, the other bandwidth-hungry). Splitting them onto separate compute resources that scale independently is an engineering trend for improving overall resource utilization.
4.3 Speculative Decoding: Breaking Autoregressive Serialization with “Parallel Verification”
The idea behind speculative decoding is clever: use a small draft model to quickly guess the next several tokens, then have the large model verify all candidates in parallel in a single pass — verification can be parallelized, while generation cannot. Accept the tokens that agree with the draft, and regenerate from the first point of divergence.
Since one parallel verification pass by the large model costs about the same as generating a single token, as long as the draft’s acceptance rate is high enough, you get multi-fold speedups without changing the output distribution. This is a classic case of “trading idle compute for serial latency,” which happens to exploit exactly the idle compute characteristic of the Decode stage.
4.4 Parallelism: Aggregating Bandwidth Across Multiple GPUs
When a single GPU lacks sufficient memory or bandwidth, tensor parallelism can shard the model across multiple GPUs. Each GPU only moves its own share of the weights, and the aggregate bandwidth increases accordingly. The cost is inter-GPU communication overhead, so the sharding granularity must balance “bandwidth gains” against “communication costs.”
5. Engineering Practice: An Actionable Optimization Path
To turn the map above into concrete action, the following order is recommended:
- Measure first, then optimize. Use profiling tools to determine whether time is spent in Prefill or Decode, and whether the bottleneck is compute or bandwidth. For slow first tokens, focus on Prefill (operator efficiency, sequence length); for slow generation, focus on Decode (quantization level, batch size, KV Cache management).
- Start with quantization. For the vast majority of scenarios, INT8 weight quantization is the highest-ROI first step — low deployment cost, clear benefits.
- Make full use of the serving framework’s scheduling capabilities. Continuous batching and paged KV Cache management are already mature. Properly configuring the maximum batch size and memory watermark often yields more than modifying the model itself.
- Introduce advanced techniques as needed. For long-context scenarios, prioritize KV Cache quantization and fused attention implementations; for latency-sensitive scenarios, evaluate speculative decoding; when a single GPU is no longer sufficient, plan multi-GPU parallelism.
- Build a continuous observability system. Combine monitoring, alerting, and logging to continuously track TTFT, per-token latency, throughput, and memory usage, so that every optimization is backed by data.
6. Conclusion
LLM inference optimization comes with a lot of jargon, but there is only one logical thread: the Decode-stage bottleneck is memory bandwidth, not compute — so every optimization either reduces the data to be moved, or makes the movement more worthwhile and faster.
Hold on to the basic judgment that “Prefill is compute-bound, Decode is bandwidth-bound,” and then look at quantization, KV Cache optimization, kernel fusion, continuous batching, and speculative decoding — you will find they are no longer isolated terms, but different coordinates on the same map. Measure first to locate the bottleneck, then act step by step in order of ROI — that is the most reliable engineering methodology for inference optimization.
