LLM Inference Performance Optimization: From Compute-Bound to Memory-Bound Bottlenecks

1. Why LLM Inference “Has Enough Compute but Still Runs Slow”

Many teams deploying large language models run into the same puzzle: GPU compute specs look plentiful, yet actual inference speed falls far below expectations. To understand this, you first need to break the inference process into two stages.

1.1 The Prefill Stage: A One-Time Compute-Intensive Process

When a user submits a complete prompt, the model feeds the entire text through the network at once, computing the hidden states for all input tokens in parallel. Because there are no temporal dependencies among these tokens, this step can fully utilize the GPU’s matrix computation units—it is a classic compute-bound task.

Characteristics of this stage:

• The longer the input, the larger the computation, but parallelism is high;
• GPU compute utilization is relatively high;
• Latency typically scales approximately linearly with prompt length.

1.2 The Decode Stage: Token-by-Token Generation Bottlenecked by Memory Access

After prefill, the model enters the autoregressive generation phase: it produces one token at a time, appends the new token to the existing sequence, then computes the next token. Each generated token requires attention computation over the information of all preceding tokens.

Key characteristics of this stage:

• Only one token is processed at a time, so parallelism is extremely low;
• Model parameters and the KV cache must be read repeatedly;
• It is a classic memory-bandwidth-bound task.

In other words, most of the time in the decode stage is not spent on “computing” but on “moving data.” This is the core contradiction at the heart of inference optimization.

2. Understanding the Cost of “Moving Data” with a Simple Calculation

To make this contradiction more intuitive, we can work through a simplified estimate.

Assume a model with 7 billion parameters. At FP16 precision, the parameters occupy roughly 14 GB of GPU memory. Each time a token is generated, all 14 GB of parameters must theoretically be moved from memory to the compute units.

If the GPU’s memory bandwidth is 2 TB/s, moving 14 GB of data takes:

14GB / 2TB/s ≈ 7ms

This means that even if GPU compute were infinite, the act of “reading parameters” alone would cap generation speed at about 140 tokens per second. In real inference, the KV cache must also be read, attention computed, sampling performed, and so on—so actual speeds are often even lower.

Here is a highly counterintuitive conclusion:

For LLM inference, the key factor limiting throughput is often not compute power but memory bandwidth.

This also explains why many inference optimization techniques are essentially doing the same thing: reducing the amount of data that must be moved to generate each token.

3. Core Approaches to Inference Performance Optimization

Around the central theme of “reducing memory access pressure and improving data reuse,” the industry has developed several general optimization paths. Each is covered below.

3.1 KV Cache: Avoiding Recomputation, at the Cost of New Memory Access Overhead

In autoregressive generation, recomputing the Key and Value of all preceding tokens for every new token would cause computation to grow rapidly with sequence length. The KV cache approach stores already-computed Keys and Values so they can be reused directly in subsequent generation.

Benefits:

• Significantly reduces redundant attention computation;
• Converts compute pressure into storage and read pressure.

But the KV cache also introduces new problems:

• The longer the sequence, the more GPU memory the KV cache occupies;
• Every generated token requires reading the entire KV cache;
• With many concurrent requests, the KV cache can become a memory bottleneck.

As a result, KV cache management strategies—such as paged storage, on-demand eviction, and shared prefix reuse—become critical components of inference system design.

3.2 Quantization: Moving the Same Information with Fewer Bits

If the bottleneck is bandwidth, directly reducing the amount of data to move is the most straightforward approach. Quantization achieves this by lowering the numerical precision of parameters and activations.

Common quantization schemes include:

• Weight quantization: converting FP16 weights to INT8 or INT4, reducing memory footprint and bandwidth demand proportionally;
• Activation quantization: representing intermediate activations at lower precision;
• Mixed precision: keeping higher precision for precision-sensitive parts while using lower precision elsewhere.

With INT8, for example, the volume of parameter data moved can theoretically be halved, potentially doubling the upper bound of decode-stage speed. INT4 goes further, but its impact on model accuracy requires more careful evaluation.

The core trade-off in quantization is whether the accuracy loss is acceptable. In practice, the appropriate quantization level depends on task type, model scale, and user tolerance.

3.3 Batching: Amortizing Memory Access Costs Through Concurrency

During decode, a single request generates only one token at a time, severely underutilizing the GPU’s parallelism. Batching processes multiple requests simultaneously, merging the generation tasks of multiple tokens into a single matrix operation.

Benefits:

• The same model parameters can be shared across multiple requests;
• A single memory access can serve more computation;
• Overall throughput improves significantly.

But batching has costs:

• Requests have different sequence lengths, requiring padding or variable-length handling;
• Larger batches can increase per-request latency;
• KV cache memory usage grows linearly with batch size.

As a result, real systems typically use continuous batching or dynamic batching strategies to balance throughput and latency.

3.4 Attention Mechanism Optimization: Reducing KV Cache Read Pressure

Attention computation is one of the main sources of memory access in the decode stage. Attention mechanisms can be optimized in several directions:

• Multi-head attention merging: reducing the overhead of reading the KV cache separately for multiple heads;
• Grouped-query attention: letting multiple query heads share the same set of Key/Value heads, directly reducing KV cache storage and reads;
• Sliding window attention: attending only to a recent window of context, limiting KV cache growth;
• Sparse attention: computing only a subset of important attention connections, reducing overall computation and memory access.

The common goal of these methods is to shrink the size and read frequency of the KV cache while minimizing impact on model quality.

3.5 Speculative Decoding: “Probing” with a Small Model, “Verifying” with a Large Model

Speculative decoding is a technique that uses a small model to accelerate generation by a large model. The basic flow is:

1. Use a lightweight model to quickly generate multiple candidate tokens;
2. Use the target large model to verify in parallel whether these candidates are correct;
3. Accept the correct portions, reject the incorrect ones, and continue generating.

Because verification can process multiple tokens in parallel, it effectively completes multiple generation steps in a single forward pass, improving overall speed.

The key factors in speculative decoding:

• The higher the small model’s prediction accuracy, the greater the speedup;
• The verification process itself carries compute and memory access overhead;
• Candidate length must be controlled to avoid wasted computation.

3.6 Operator Fusion and Memory Layout Optimization

At the engineering implementation level, memory accesses can also be reduced through:

• Operator fusion: merging multiple consecutive small operators into one larger operator to reduce intermediate reads and writes;
• Memory layout optimization: arranging data in GPU memory to better match access patterns and improve cache hit rates;
• Asynchronous transfers: prefetching the next batch of data while computing, hiding some memory access latency.

These optimizations usually require a deep understanding of hardware architecture and the internal implementation of inference frameworks, but their impact on final performance is often significant.

4. Selecting and Combining Optimization Strategies

With so many optimization techniques available, practical deployment requires trade-offs based on the specific scenario. Here is a simplified decision reference:

Optimization Goal	Preferred Techniques
Reduce per-generation latency	Quantization, operator fusion, speculative decoding
Improve overall throughput	Batching, continuous batching, KV cache optimization
Support very long contexts	Sliding window attention, sparse attention, KV cache compression
Reduce memory footprint	Quantization, grouped-query attention, paged KV cache management

Note that these techniques are not mutually exclusive and can be combined. For example, a high-concurrency online service might simultaneously use INT8 quantization, continuous batching, and grouped-query attention to reduce memory access demands along multiple dimensions.

5. Conclusion

LLM inference performance optimization is fundamentally an efficiency battle centered on memory bandwidth. The prefill stage is relatively compute-bound, while the decode stage is memory-bandwidth-bound—and the latter is often the main bottleneck in real deployments.

With this understanding, the optimization directions become clear:

• Reduce the amount of data to move (quantization, KV cache compression);
• Improve utilization of each data transfer (batching, continuous batching);
• Lower memory access frequency (attention optimization, operator fusion);
• Trade parallelism for time (speculative decoding).

In real engineering, no single silver bullet solves every problem. Only by combining multiple techniques based on business scenarios, hardware conditions, and model characteristics can inference performance be pushed higher within limited memory bandwidth.