LLM Inference Optimization Guide: Breaking the GPU Memory Bandwidth Bottleneck

When optimizing LLM inference, developers can easily get lost in a sea of terminology: KV Cache, operator fusion, speculative decoding, and more. These individual techniques are important, but without a systematic, big-picture view, optimization easily degenerates into blind trial and error—fixing one symptom only to create another. This article strips away the jargon and rebuilds the technical map of LLM inference optimization with clear logic, helping developers who have some foundation but lack systematic understanding find their way.

1. Where Does the Inference Bottleneck Actually Lie?

Before discussing optimization, you must first identify the performance bottleneck. The bottleneck characteristics of LLM inference and training are fundamentally different. Training is primarily compute-intensive, requiring massive matrix multiplications to compute gradients. Inference is more complex and must be analyzed phase by phase.

LLM inference is typically divided into two core phases:

1. Prefill phase: The entire user prompt is fed into the model at once, and the hidden states for all input tokens are computed in parallel. Because all tokens can be processed simultaneously, this phase is compute-intensive, and GPU compute capacity is often fully utilized.
2. Decode phase: The model generates tokens one at a time. Each new token requires information from all previous tokens for attention computation. This phase is memory-bandwidth-intensive.

Here is a counterintuitive fact: during the decode phase, most of the time is not spent on “computing” but on “moving data.”

Consider a typical high-end GPU with 80GB of memory. Its FP16 compute throughput can reach hundreds of TFLOPS per second, but its memory bandwidth is only about 2TB/s. Suppose you are running a 7-billion-parameter (7B) model, which at FP16 precision has about 14GB of parameters. During the decode phase, generating each token theoretically requires moving all 14GB of parameters from memory through the compute units.

At 2TB/s, moving 14GB takes roughly 7 milliseconds. This means that even if the GPU had infinite compute, the act of “reading model parameters” alone would cap generation at about 140 tokens per second. Add the growing KV Cache reads and other system overhead, and actual generation speed drops further.

This is the core contradiction of LLM inference optimization: GPU compute units spend most of their time waiting for data rather than processing it. With this in mind, all subsequent optimization techniques fall into two broad categories: reducing the amount of data that must be moved and improving the efficiency of data movement and computation.

2. Reducing the Amount of Data That Must Be Moved

The direct goal of this class of optimizations is to reduce the scale of data transfer from memory to compute units. The main technical paths include:

1. Model Quantization

Quantization converts high-precision floating-point model weights (such as FP16) into lower-precision representations (such as INT8 or INT4). With INT8 quantization, for example, parameter size is cut in half—a 14GB model drops to 7GB, and read time is halved. Although lower precision introduces minor accuracy loss, combined with the redundancy inherent in large models, a reasonable quantization strategy can achieve an excellent balance between accuracy and speed.

2. KV Cache Compression and Management

In the decode phase, beyond model parameters, the Key and Value of historical tokens (the KV Cache) must also be read continuously. As generation length grows, the KV Cache grows linearly and can easily become a new memory killer. Optimization approaches include:

• PagedAttention: Inspired by the virtual memory paging management of operating systems, this divides the KV Cache into fixed-size blocks, avoiding memory fragmentation and improving memory utilization.
• KV Cache quantization: Quantizing the FP16 KV Cache to INT8 or even INT4, significantly reducing the amount of data read during decoding.

3. Structured Pruning

By removing redundant attention heads or feed-forward network layers from the model, the parameter count is directly reduced. A pruned model not only uses less memory but also reduces the computation per inference pass—though it requires a rigorous fine-tuning process to recover model performance.

3. Improving the Efficiency of Data Movement and Computation

Without changing the parameter count, software-hardware co-design and computation graph restructuring can squeeze out the hardware’s maximum performance.

1. Operator Fusion

Large models consist of a large number of basic operators such as matrix multiplication, Softmax, and activation functions. If executed one by one, each operator must write intermediate results to memory and read them back, generating substantial unnecessary I/O. Operator fusion compiles multiple operators into a single composite kernel at a low level, allowing intermediate results to flow directly through GPU registers or SRAM and greatly reducing memory read/write operations.

2. FlashAttention

This is a classic example of operator fusion. Traditional attention computation requires writing an $N \times N$ attention matrix to memory. FlashAttention uses tiled computation, dividing data into blocks and completing all Softmax and scaling operations in SRAM, avoiding memory reads and writes for the large intermediate matrix and achieving a perfect balance between computation and I/O.

3. Continuous Batching

Traditional batching with variable-length sequences causes GPU utilization to drop sharply when short sequences finish generation while long sequences are still running. Continuous Batching manages the batch dynamically at iteration granularity: as soon as a sequence finishes generating, a new request is immediately inserted into the freed slot. This mechanism ensures the GPU processes a full load of tokens in every iteration, maximizing throughput.

4. Advanced Strategy: Trading Time for Space

Speculative Decoding

In standard autoregressive decoding, generating each token requires loading the full model weights once, which is extremely time-consuming. Speculative decoding introduces a small “draft model” that first quickly generates several candidate tokens at very low cost, then feeds these candidates into the large model at once for parallel verification.

If the large model agrees with the draft model’s predictions, multiple tokens are accepted at once; if not, they are rejected and regeneration starts from the point of error. This approach trades a small amount of extra computation for multiple parallel weight reads, effectively offsetting the memory wall problem and substantially reducing time to first token and overall generation time.

5. Engineering Practice Recommendations and Summary

In real-world engineering deployments, developers should not blindly stack optimization techniques but should make trade-offs based on the business scenario:

• Conversational scenarios (low latency first): Focus on optimizing the prefill phase and speculative decoding to reduce time to first token.
• Batch processing scenarios (high throughput first): Focus on implementing continuous batching and PagedAttention to ensure memory does not overflow and GPU utilization stays high under high concurrency.

LLM inference optimization is a game against memory bandwidth. Whether on a managed LLM platform such as Baidu AI Cloud’s Qianfan or a self-built inference cluster, everything ultimately comes down to maximizing the efficiency of “parameter movement” and “compute utilization.” Only by clarifying the relationship between computation and memory and building a systematic optimization map can you find the optimal solution between resources and performance.