High-Throughput LLM Inference Serving: Key Optimization Practices
This article explains why large language model inference behaves differently from training and walks through the core engineering techniques that high-throughput inference services rely on. It covers paged KV Cache memory management, continuous batching, computation graph capture and operator-level optimizations, quantization, multi-GPU parallelism, decoding strategies, and production serving features such as streaming output, prefix caching, and dynamic multi-LoRA loading. It also addresses hardware backend abstraction and typical deployment scenarios, helping developers understand the tradeoffs behind each mechanism rather than just memorizing framework APIs.
1. Why LLM Inference and Training Are Two Different Problems
Before discussing how to build an inference service, it is worth clearing up a common misconception: the resource profile of the inference stage is fundamentally different from that of the training stage. Training is compute-intensive—it focuses on backpropagating gradients onto parameters as efficiently as possible. Autoregressive inference, by contrast, is typically memory-bandwidth-bound during the decode phase: each step generates a single token and requires reading the entire set of model weights from GPU memory, resulting in very low arithmetic intensity and often leaving compute units idle.
This gives rise to two core tensions:
- Memory bottleneck: The attention mechanism needs to cache the keys and values of past tokens—the KV Cache. As context length and the number of concurrent requests grow, the memory footprint of the KV Cache expands rapidly, becoming the first constraint on concurrency.
- Throughput–latency tradeoff: Batching significantly improves throughput, but it couples multiple requests together. Without careful scheduling, a single long sequence can slow down an entire batch, causing head-of-line blocking.
At its core, a mature inference library solves the engineering problem of “how to serve as many concurrent requests as possible with the highest possible throughput under limited memory.” The rest of this article builds on that theme.
2. Memory Management: From Contiguous Allocation to Paged Management
Traditional inference frameworks reserve a contiguous block of memory per sequence for the KV Cache, usually pre-allocated to the maximum possible length. This leads to two kinds of waste:
- Internal fragmentation: When the actual sequence length is far shorter than the reserved length, the extra memory cannot be reused.
- External fragmentation: Sequences vary in length, and after repeated allocation and deallocation the memory develops many gaps, so even when free space exists it is hard to satisfy a new allocation.
The industry has gradually adopted a paged memory management approach: the KV Cache is split into fixed-size blocks, each holding the keys and values of a few tokens, and each sequence’s logical tokens are mapped to physical memory through a “logical block → physical block” mapping table. The benefits are direct:
- With a fixed block size, both internal and external fragmentation are nearly eliminated, significantly improving memory utilization.
- Requests that share an identical prefix (for example, the same system prompt) can share physical blocks, achieving zero-copy reuse and further saving memory.
- Operations such as copying, migration, and swapping out are performed per block, making the management logic clearer.
A schematic of the mapping logic looks like this:
# Mapping from logical blocks to physical blocks (illustrative pseudocode)class BlockTable:def __init__(self, block_size):self.block_size = block_sizeself.mapping = {} # seq_id -> [phys_block_id, ...]def append_token(self, seq_id, allocator):# Only allocate a new physical block when the logical length crosses a block boundarylogical_len = len(self.mapping[seq_id]) * self.block_sizeif logical_len % self.block_size == 0:phys_id = allocator.allocate()self.mapping[seq_id].append(phys_id)return phys_iddef share_prefix(self, src_seq, dst_seq, n_blocks):# Sequences with the same prefix share the first n_blocks physical blocksself.mapping[dst_seq] = self.mapping[src_seq][:n_blocks].copy()
3. Continuous Batching: Breaking the Head-of-Line Blocking of Static Batching
Static batching requires an entire batch to finish generation before it can be released and admit new requests. In autoregressive scenarios, output lengths vary widely across requests, so short requests are forced to wait for the longest one, and GPU utilization drops sharply at the tail of the batch.
Continuous batching (also called in-flight batching) lowers the scheduling granularity from the “batch” level down to the “iteration step”:
- Immediately after each decode step, reclaim the memory blocks held by sequences that have finished.
- Within the same scheduling cycle, admit new requests from the waiting queue into the freed batch slots.
- All active sequences in the batch share one forward pass, generating the next token for every sequence at once.
The scheduling loop can be abstracted as:
# Scheduling loop for continuous batching (illustrative pseudocode)while engine.is_running():# 1. Admit new requests to fill available slotsready = scheduler.pop_waiting(max_new=free_slot_count)running_batch.extend(ready)# 2. Run a single forward pass over the whole batch (generate one token)logits = model.forward(running_batch)# 3. Sample the token for this steptokens = sampler(logits, decoding_config)# 4. Append tokens, update KV Cache blocks, and recycle finished sequencesfor seq in running_batch:seq.append(tokens[seq.id])if seq.is_finished():scheduler.recycle(seq)running_batch = [s for s in running_batch if not s.is_finished()]
Continuous batching and paged memory management complement each other: the former requires frequently adding and removing batch members dynamically, while the latter provides fast, block-level allocation and reclamation. Only together can they sustain high throughput while keeping latency low.
4. Computation Graph Execution and Operator-Level Optimization
Per-operator dispatch at the Python layer introduces substantial kernel launch overhead, which is especially damaging in decode scenarios characterized by “small operators, high frequency.” In practice, inference engines rely on computation graph capture to freeze a sequence of operators into a single replayable execution unit, reducing dispatch overhead and stabilizing the latency distribution.
At the operator level for compute-heavy paths, mature inference libraries provide custom kernels for hotspots such as attention, normalization, and activation functions, and they fuse several adjacent operators (for example, combining a matrix multiply, bias addition, and activation into a single kernel). These optimizations do not change the numerical semantics, but they significantly reduce memory accesses and improve effective compute utilization.
5. Quantization: Trading Precision for Throughput and Memory
Quantization is an important way to ease memory pressure and improve throughput. Common approaches include:
- Weight quantization: Compressing FP16/BF16 weights into INT8, FP8, or even lower bit widths, directly reducing the memory needed to load the model.
- Activation quantization and quantized operator fusion: Using low-bit inputs within matrix multiplications and leveraging the hardware’s low-precision compute units to accelerate computation.
- KV Cache quantization: Compressing the key-value cache to lower bit widths, multiplying the context length and concurrency that can fit in memory.
Quantization inevitably introduces some loss of precision, so in practice teams calibrate quantization on an offline evaluation set and apply mixed-precision protection to sensitive layers (such as the output layer), striking a balance among throughput, memory, and generation quality.
6. Parallelism Strategies: Putting Large Models Across Multiple GPUs
When a single GPU cannot hold the model or higher throughput is needed, parallelism becomes essential:
- Tensor parallelism: Splitting a single matrix multiplication across rows/columns over multiple GPUs for parallel computation, combining results through collective communication (such as all-reduce). Suited to multi-GPU setups within a single machine.
- Pipeline parallelism: Partitioning the model by layer, with different GPUs responsible for different layers, forming a pipeline. Suited to cross-machine deployment.
- Data parallelism: Each GPU holds a full replica of the model and handles a different share of requests, used to scale throughput horizontally.
Real systems often combine tensor parallelism with data parallelism, striking a compromise between communication overhead and memory constraints.
7. Decoding Algorithms and Controllable Generation
Inference libraries typically expose several decoding strategies for different business needs:
- Greedy decoding: Picking the highest-probability token at each step—fast but prone to repetition.
- Beam search: Keeping several candidate paths, suited to generation tasks with high demands on completeness and fluency.
- Parallel sampling / nucleus sampling: Generating multiple candidate sequences at once, or sampling with probability truncation, balancing diversity and quality.
Sampling logic is usually abstracted behind a unified strategy interface, making hot-swapping easy:
def sample(logits, strategy):if strategy == "greedy":return argmax(logits)elif strategy == "beam_search":return beam_search(logits, beam_width=strategy.width)elif strategy == "nucleus":return nucleus_sampling(logits, p=strategy.top_p)...
8. Serving Capabilities: From “Can Infer” to “Serves Well”
An inference service fit for production also needs to be complete on the functional side:
- Streaming output: Returning generated tokens incrementally so callers can render as generation proceeds, markedly improving the interactive experience.
- Interface compatibility: Exposing service endpoints compatible with the industry-standard RESTful interface specification, lowering the migration cost for clients.
- Prefix caching: Caching the KV of shared, invariant prefixes (such as a fixed system prompt) and reusing it directly on a cache hit, cutting redundant computation.
- Dynamic multi-LoRA loading: LoRA is a general-purpose fine-tuning method based on low-rank adaptation. A service can dynamically attach multiple LoRA weights onto the same base model and switch between them on demand, serving multiple customized model variants for different business directions on a single machine and avoiding the need to deploy a separate full model for each adapter.
9. Hardware Backend Adaptation: Operator Abstraction Is Key
Sustained improvements in inference performance increasingly depend on exploiting low-level hardware features, and the growing variety of AI accelerator hardware raises new adaptation challenges. Portable inference frameworks typically abstract the hardware-specific parts into an operator and execution backend layer, decoupling the upper-level scheduling logic from the hardware. With the help of automated migration/adaptation tooling or vendor-provided operator libraries, developers can shorten the adaptation cycle and bring performance on new hardware up to a usable level.
A reasonable expectation for production-grade adaptation is this: it is not finished just because the code “runs.” Attention kernels, quantization operators, and communication primitives must be tuned one by one, and results should be compared and validated against unified performance benchmarks (throughput, time to first token, per-token latency) before an inference service is ready for production.
10. Typical Deployment Scenarios and Cloud Practices
- Text generation: Tasks such as conversation, summarization, and translation are the highest-volume scenario for inference services, with a primary focus on throughput and streaming latency.
- OCR and multimodal: Chaining a vision encoder model with a language generation model, turning recognition results into structured text or natural-language descriptions.
- Cloud deployment: Relying on container platforms for orchestration, object storage for distributing model weights, and logging services plus monitoring and alerting for observability. Baidu AI Cloud, for example, can complete rapid Day-0 adaptation based on general-purpose inference frameworks when publishing a model, and complete validation on mainstream open-source inference frameworks, so that a new model gains usable inference service capability as early as possible.
Conclusion
The competitiveness of an LLM inference service ultimately comes down to three measurable dimensions: how many concurrent requests it can serve on limited hardware, how low its latency is, and how strong its stability is. Paged memory management and continuous batching lay the foundation for throughput; computation graphs and operator optimizations lower per-step latency; quantization and parallelism expand the scale boundaries; and capabilities such as streaming output, prefix caching, and dynamic multi-LoRA loading determine how usable the service is and how broad its business coverage can be. For developers, understanding the tradeoffs behind these mechanisms matters far more than memorizing a particular framework’s API—because regardless of the underlying implementation, the engineering goal is always the same: through system-level, coordinated design, convert every unit of compute into stable, efficient, and scalable service capability.
