High-Throughput LLM Inference: Core Techniques and Deployment Practices

    This article explains how to raise throughput in large language model serving. It covers why inference is harder to optimize than training, how continuous batching keeps compute units busy, how paged KV cache management reduces memory waste, how compute graphs and kernel fusion cut scheduling and memory overhead, and how quantization trades precision for capacity and speed, along with tuning guidance and risks for each technique.

    1. Why Inference Services Are Hard to Optimize

    Large model inference has different resource characteristics than training. Training focuses on throughput and gradient synchronization. Inference must serve many concurrent requests, keep time to first token (TTFT) under control, and maintain a sufficient number of generated tokens per unit of time. Autoregressive decoding depends on the full preceding context for every token it generates. Reading model weights and reading and writing the KV cache put heavy pressure on memory access, and compute units often sit idle while they wait for data.

    Online traffic adds further challenges: request lengths vary, arrival times are random, and concurrency fluctuates. A naive strategy that gives each request its own batch leaves compute units idle once short requests finish, while long requests keep other requests waiting. A mature serving system must coordinate optimization across four layers: scheduling, memory, compute, and communication.

    2. Continuous Batching: Keeping Compute Units Busy

    Continuous batching uses the decoding iteration, rather than the request, as its scheduling unit. After each decoding round, the scheduler removes completed sequences and frees their memory, then pulls new requests from the waiting queue. This keeps batch fill rates high.

    The following pseudocode illustrates the scheduling logic only. It does not correspond to any specific interface.

    1. while service is running:
    2. remove completed sequences and release their KV cache blocks
    3. take requests from the waiting queue by priority until the token budget or concurrency limit is reached
    4. run one forward step for the current batch, generating one new token per sequence
    5. stream each new token back to the corresponding client

    The scheduler must balance three goals. Larger batches raise throughput, but each step takes longer. Prefill processes the full prompt and is computationally heavy. If left unchecked, it blocks requests that are already decoding. A common approach is to split prefill into chunks and interleave them with decoding steps. For fairness, long requests must not hold resources indefinitely.

    Key parameters and suggested starting points:

    Parameter Role Tuning direction
    Per-batch token budget Limits the total number of tokens processed per step Increasing it improves throughput but raises time per output token
    Maximum concurrent sequences Limits the number of requests in the batch at once Constrained by memory. Beyond the limit, queuing time grows.
    Prefill chunk size Number of prompt tokens processed per chunk Smaller chunks reduce interference with decoding but increase the number of scheduling rounds
    Fairness policy For example, raising priority based on waiting time, or capping generation steps per request Emphasizing fairness sacrifices some peak throughput

    Tune these parameters against your latency targets. First fix the concurrency level. Then load test TTFT and throughput curves under different token budgets, and choose the largest value that still meets the latency constraint.

    How continuous batching compares with static batching:

    Dimension Static batching Continuous batching
    Throughput The longest request in a batch determines when the whole batch finishes, which limits throughput Short requests free their slots immediately, so throughput is higher
    Tail latency Long requests slow down the entire batch Dynamic insertion shortens queuing, but long requests must be kept from monopolizing resources
    Idle time for short requests Significant Substantially reduced
    Fairness Depends on batch composition Requires additional policies

    3. Paged KV Cache: The Core of Memory Management

    The KV cache stores the key and value tensors for every processed token so that later steps can reuse them. In long-context and high-concurrency workloads, the KV cache often exhausts GPU memory before the model weights do. The traditional approach reserves a contiguous region of maximum length for each request. This causes heavy internal fragmentation and wasted reserved space.

    Paged management borrows the virtual memory idea from operating systems. It splits the KV cache into fixed-size blocks, allocates them on demand, and uses a block table to map logical positions to physical blocks. The benefits, costs, and applicable conditions are as follows:

    Benefit Cost Applicable condition
    Reduces fragmentation: a request occupies only the blocks it actually needs Maintaining and indexing the block table adds overhead Request lengths vary widely
    Supports dynamic growth: blocks are added as generation proceeds Blocks that are too small increase management overhead. Blocks that are too large waste space in the final block. Output length cannot be predicted in advance
    Enables sharing: sequences with the same prefix can share physical blocks Shared blocks need reference counting to manage their lifetime Many shared system prompts or multi-turn conversations

    Common block sizes are 16 or 32 tokens. Choose a size based on average sequence length and measured fragmentation rate.

    Keep the following points in mind when using paged management:

    • Block size: Blocks that are too small increase management overhead and memory access fragmentation. Blocks that are too large waste significant space in the final block.
    • Shared block lifetime: A block referenced by multiple sequences can be freed only after its reference count drops to zero. Freeing it too early causes incorrect reads. Freeing it too late wastes GPU memory.
    • Memory utilization ceiling: If set too high, bursty load can cause out-of-memory (OOM) errors. If set too low, concurrency is limited. When adjusting it, monitor peak memory usage and queue wait times together.
    • OOM fallback strategy: When memory runs short, pause or preempt low-priority requests. Swap their KV blocks out to host memory, or drop them and recompute later. Also rate-limit new requests and return retryable error codes. The specific strategy depends on what the framework in use supports, so refer to the relevant documentation.

    4. Compute Graphs and Kernel Optimization

    Each decoding step has a highly similar computation structure. However, scheduling operators one at a time adds substantial kernel launch overhead. Capturing a fixed-shape sequence of computation as a compute graph, then replaying it as a whole, can significantly reduce CPU-side scheduling overhead. This approach is now widely used on mainstream GPU computing platforms and other accelerator platforms.

    Compute graphs require fixed input shapes, but batch size and sequence length keep changing. A common approach is to capture several discrete batch sizes in advance. At runtime, the actual batch is rounded up to the nearest captured size, and the extra positions are masked.

    This method has clear limits:

    • Number of captured sizes: More sizes reduce padding waste but increase capture time and memory use. Typically, a handful of sizes that cover common batch sizes is sufficient.
    • Masking cost: When the actual batch is much smaller than the captured size, the extra computation is wasted. This is especially noticeable during low-traffic periods.
    • Dynamic shapes and long-tail requests: Very long prompts or requests with unusual shapes can fall back to operator-by-operator execution. This avoids capturing a dedicated graph for them.

    Custom kernel fusion is also commonly applied to stages such as attention, normalization, and sampling. Fusion reduces how often intermediate results are written back to global memory, which lowers memory bandwidth pressure. Kernel implementations depend heavily on hardware architecture, so proven implementations are preferred. The final test should be end-to-end: compare time per output token and throughput under a fixed workload, rather than looking only at the latency of a single operator. When measuring, warm up first, fix the input distribution, and take the median of multiple runs.

    5. Quantization: Trading Precision for Capacity and Speed

    Quantization maps weights and some activations from high precision (such as FP16 or BF16) to low-bit representations (such as INT8, INT4, or FP8). Weight quantization substantially reduces memory footprint and read bandwidth, which benefits the memory-bound decoding stage. Activation quantization can further speed up computation, but it is more sensitive to outliers and more complex to implement.

    Accuracy loss from quantization should not be judged by average metrics alone. In practice:

    • Calibration: Estimate quantization parameters using representative data. The calibration data should cover the languages and length distribution of real traffic.
    • Evaluation: Compare accuracy, factual consistency, and output format compliance before and after quantization on your own evaluation set. Pay particular attention to sensitive tasks such as long-form text and code.
    • Fallback: Keep high-precision versions for critical paths, or keep sensitive layers (such as some attention layers or the output layer) in high precision.
    • Gradual rollout: Test on a small share of traffic first and compare user-facing quality metrics. Increase the share only after confirming there is no significant degradation.

    6. Parallel Inference: Splitting the Model Across Devices

    When a model does not fit on a single accelerator, or when one device cannot deliver enough throughput, parallelism is needed.