LLM Inference Explained: Prefill and Decode Phases in Depth
LLM inference runs in two very different phases, and each one stresses your hardware in a different way. This article breaks down Prefill, the one-shot parallel pass over the input prompt that builds the KV cache, and Decode, the token-by-token autoregressive loop that is memory-bandwidth bound. It explains why the two phases are scheduled separately, and reviews the engineering techniques that help them work together: chunked prefill, mixing both in a single batch, KV cache paging and reuse, continuous batching, disaggregated deployment, and quantization.
LLM Inference: The Two Core Phases, Prefill and Decode, Explained in Depth
Start With What You Can Observe
Watch someone use a conversational AI tool and you will see a familiar rhythm: they paste in a large block of text, hit Enter, the screen pauses for a second or two, the first character slowly appears, and then the rest of the answer comes pouring out in one continuous stream.
That contrast between the wait and the rush is not a matter of network latency or interface rendering. Two fundamentally different modes of computation are taking turns behind the scenes.
Understanding both modes is the starting point for understanding why large model inference consumes so much GPU power. It is also key to understanding why some teams can bring inference costs down to a fraction of everyone else’s. This article breaks down the two phases and walks through the engineering optimizations that come with them.
1. Prefill: Digesting the Entire Input in One Parallel Pass
What it actually computes
Prefill literally translates as “pre-fill,” but a more precise description is this: the model sends all of the user’s input tokens through the network in parallel, in a single pass. They travel through dozens of Transformer layers, and the model computes the hidden state at every position. Along the way, it produces the key data structure that the later decoding phase depends on: the KV Cache.
What the KV Cache is
KV Cache stands for Key-Value Cache. To process each token, the Transformer’s attention mechanism needs the Key and Value vectors associated with that position. During prefill, the model computes and stores the Key and Value for every input token at every layer.
You can think of it as the model’s “memory” of the input. Every time it generates a new token, it consults this memory to decide what the next output should be.
Compute characteristic: compute-bound
Suppose the input is 1,000 tokens. The model needs to:
- Convert all 1,000 tokens into word embeddings
- Run them through N Transformer layers, with a complete self-attention computation in each layer
- Pay for self-attention’s roughly O(n²) complexity: about a million basic operations for 1,000 tokens
- Produce and cache the KV Cache for all 1,000 positions
This is a highly parallel, pure compute workload. The tens of thousands of compute cores on a GPU can work in concert and finish the computation for all 1,000 tokens almost simultaneously.
As a result, GPU utilization is typically high during prefill. Compute is fully engaged, and memory bandwidth is not the bottleneck. This is often the part of inference where the hardware gets the most out of its capacity.
# Illustration: prefill processes all input tokens in paralleldef prefill(model, input_ids):# input_ids: [seq_len], all input tokens passed in at oncehidden = model.embedding(input_ids) # word embeddingskv_cache = []for layer in model.layers: # go through each Transformer layerhidden, k, v = layer.attention(hidden) # self-attention, outputs this layer's K/Vkv_cache.append((k, v)) # cache this layer's Key/Valuehidden = layer.ffn(hidden) # feed-forward networkreturn hidden[:, -1, :], kv_cache # last-position output + full KV cache
2. Decode: Pulling Out the Output Token by Token
What it actually computes
Decode is where the model starts emitting one token at a time. For each new token, it draws on the “memory” built during prefill (the KV Cache) and on the content generated so far to work out what comes next. Once a token is generated, its Key and Value are appended to the cache so the next step can reuse them.
Compute characteristic: memory-bound
Decode is the opposite of prefill. Each step processes just one token, so the compute is small. But each step must read the entire set of model weights, plus a KV Cache that keeps growing. That means:
- Low compute per step, leaving large numbers of GPU compute cores idle
- The bottleneck shifts from “how fast can it compute” to “how fast can it move data”
- As generation gets longer, the KV Cache keeps expanding and memory-access pressure keeps rising
This memory-bound nature is the root cause of relatively high per-token latency and comparatively low GPU utilization during decode. It works like a patient reader who takes in the preceding text word by word and writes a reply word by word. At every step, all existing content has to be read again.
# Illustration: decode generates autoregressively, one token at a timedef decode(model, kv_cache, max_new_tokens):tokens = []for _ in range(max_new_tokens):logits, kv_cache = model.step(last_token, kv_cache) # single forward steplast_token = sample(logits) # sample the next tokentokens.append(last_token) # append to outputreturn tokens
3. Why Split Into Two Phases? Can’t They Be One?
From a computation-graph perspective, prefill and decode are the same forward computation running on different input shapes. In engineering terms, however, clearly separating and optimizing each one is fully justified, because their resource profiles are almost exact opposites:
- Prefill: long-sequence input computed in a single pass; compute-bound, so it mainly consumes compute
- Decode: a single-token loop with repeated memory access; memory-bound, so it mainly consumes bandwidth
One scheduling strategy cannot serve two such different workloads. A single policy would inevitably compromise one for the other. Splitting them lets the system tune each against its own bottleneck, and that is the foundation for understanding the engineering techniques that follow.
4. Engineering Optimizations: Making the Two Phases Work Together
Head-of-line blocking
In production, many requests arrive concurrently. While a long-input prefill task is running, a short request that arrives later has to queue up even though its decoding is quick. This is classic head-of-line blocking. What users notice is that time to first token gets stretched out noticeably.
Chunked prefill
A common solution is chunked prefill: split the prefill of a long input into several smaller chunks and spread it across multiple scheduling steps, rather than letting it monopolize a full compute cycle at once.
The benefit is that decode requests can be scheduled in between prefill chunks. Requests that need to stream their output token by token no longer have to wait a long time, which noticeably improves the interactive experience.
Mixing prefill and decode in the same batch
A further step is to schedule prefill and decode in the same batch. With a unified batching mechanism, the scheduler can hold both requests in the prefill phase and requests in the decode phase, and dynamically balance the mix between the two workloads. That raises overall hardware utilization and serves both throughput and latency at once.
time steps →step 1: [Prefill chunk A] [Decode request X]step 2: [Prefill chunk B] [Decode request X] [Decode request Y]step 3: [Decode request X] [Decode request Y] [Decode request Z]
Other supporting optimizations
The industry has developed a range of general-purpose engineering techniques around the KV Cache and batching:
- Paged management and reuse of the KV Cache to reduce memory fragmentation and redundant computation
- Continuous batching, which schedules dynamically at the request level rather than the batch level
- Disaggregated deployment of prefill and decode, so each workload runs on the hardware that fits it best
- Quantization and operator optimizations to reduce the memory-access overhead of weights and cache
Conclusion
Prefill and decode form the two main lines of large language model inference. One is a compute-bound, one-shot parallel digestion of the input; the other is memory-bound autoregressive generation. Understanding their resource profiles and how they interact explains why inference “burns through GPUs.” It is also the foundation for tuning inference performance and cutting costs.
Techniques such as chunked prefill, continuous batching, cache management, and even disaggregated deployment are all essentially about revisiting these two modes of computation and rescheduling them, so you can squeeze more throughput and lower latency out of the hardware you have.
