High-Concurrency LLM Inference: Key Techniques and Engineering Practices

    As large language models move into production, the real determinant of service quality is often not the concepts themselves but a plain engineering question: when hundreds or thousands of requests hit an inference service at once, can the system stay low-latency while sustaining high throughput? This article sets aside conceptual hype and focuses on one hard-core topic, high-concurrency inference efficiency. Starting from first principles, the two-phase nature of autoregressive inference and a Roofline view of the bottleneck, it works through the mainstream optimization techniques: continuous batching, paged KV cache management, quantization, speculative decoding, prefix caching, chunked prefill, and prefill/decode disaggregation. It also covers scheduling and parallelism strategies, plus a "locate first, then optimize" methodology built on metrics such as TTFT, TPOT/ITL, throughput, and goodput.

    Over the past year, discussions around “long context” and “AI Agents” have dominated the conversation about large models. But what ultimately shapes the service experience is often not these concepts themselves, but a plain engineering question: when hundreds or thousands of requests hit an inference service at the same time, can the system keep latency low while sustaining high throughput? This article sets aside conceptual packaging and focuses on one hard-core topic, high-concurrency inference efficiency, breaking down the mainstream optimization techniques layer by layer starting from first principles.

    1. First, Understand the Two-Phase Nature of Inference

    Autoregressive inference for large models naturally splits into two phases whose performance characteristics are completely different. This is the starting point for all optimization.

    • Prefill: Encodes the entire user prompt in one parallel pass and computes the Key/Value for every token. This process involves large matrices and high parallelism, making it a classic compute-bound scenario.
    • Decode: Generates tokens one at a time based on the sequence produced so far. Each step processes only a single new token but must repeatedly read the model weights and all historical KV. It does little computation per step but touches a large amount of memory, making it a memory-bandwidth-bound scenario.

    Understanding this is critical: the Prefill phase determines time to first token (TTFT), while the Decode phase determines output speed (TPOT/ITL). Many services look like they “run,” yet they are slow on the first token and stutter afterward. The root cause is that these two phases are handled in a simple serial fashion and lack fine-grained scheduling.

    2. Locating the Bottleneck with a Roofline View

    To determine whether inference is bottlenecked on compute or on bandwidth, you can use the Roofline model, whose core metric is arithmetic intensity (the ratio of computation to memory access).

    • When arithmetic intensity is above the hardware’s “ridge point,” performance is limited by peak compute.
    • When it is below the ridge point, performance is limited by memory bandwidth.

    The batch size of the Decode phase directly determines arithmetic intensity: when the batch is too small, the GPU’s compute capacity sits largely idle; as the batch grows, the weights need to be read only once but are shared across many requests, so bandwidth pressure is amortized and arithmetic intensity rises. This leads to the single most important theme in high-concurrency inference: batch as many requests together for processing as possible.

    3. KV Cache: The Core Memory Conflict

    To avoid recomputing historical tokens, inference engines cache the Key/Value of attention at each layer, known as the KV Cache. It grows linearly with sequence length and concurrency, making it the number one memory killer in long-context, high-concurrency scenarios. When the KV Cache fills up memory, the number of requests that can run concurrently drops and throughput is directly limited. In other words, whether you can manage the KV Cache efficiently is almost equivalent to whether you can support high concurrency.

    4. Breaking Down the Core Techniques

    1. Continuous Batching

    Traditional static batching requires all requests in a batch to start and finish at the same time; short requests must wait for the long ones to finish, leaving the GPU largely idle. Continuous batching dynamically adjusts the batch on every decode iteration: finished requests exit immediately, and the freed slots let new requests join right away. Sketch:

    1. while not all_finished:
    2. step_ready = scheduler.next_ready_requests() # combine new requests + in-progress requests
    3. logits = model.forward(step_ready) # one forward pass mixing requests from different sources
    4. for req in step_ready:
    5. req.append_sample(logits[req]) # each samples independently; whoever finishes exits
    6. scheduler.evict_finished_and_admit_pending() # swap in/out on every iteration

    This significantly raises GPU utilization and is the cornerstone of modern high-performance inference engines.

    2. Paged KV Cache Management

    If the KV Cache uses large contiguous memory allocation, the varying lengths of requests cause severe memory fragmentation. Borrowing the “paging” idea from operating system virtual memory, you can split the KV Cache into fixed-size blocks that are logically contiguous but physically scattered, allocated on demand and mapped via a page table. This both reduces fragmentation waste and lets different requests that share a prefix reuse the same physical blocks (see prefix caching below).

    3. Quantization

    Quantization trades lower numerical precision for memory and bandwidth gains, and it benefits both Prefill and Decode:

    • Weight quantization (e.g., INT8/FP8/INT4): reduces weight size and directly eases the Decode bandwidth bottleneck;
    • Activation/computation quantization: combined with the right kernels, it can improve compute utilization;
    • KV Cache quantization: lowering cache precision to 8-bit or even lower multiplies the number of concurrent requests.

    In practice, watch for precision regression, and it is advisable to use an offline evaluation set to continuously monitor quality differences before and after quantization.

    4. Speculative Decoding

    Since Decode works “token by token” and is bandwidth-bound, you can use a lightweight “draft model” to guess several candidate tokens at once, then have the large model verify these candidates in parallel. If a candidate is accepted, you advance multiple steps at once. It does not change the final output distribution (with a correct implementation it is equivalent to the original sampling), yet it can significantly improve output speed, and it is especially well suited to the bandwidth-bound Decode phase.

    5. Prefix Caching

    Many requests share the same prefix, for example a common system prompt, few-shot examples, or the fixed context assembled by RAG. Computing and caching the KV for these shared prefixes only once lets later requests reuse it directly, greatly reducing repeated Prefill overhead and significantly improving TTFT. It is also an important optimization for Agent-type applications (multi-turn, long templates).

    6. Chunked Prefill

    The Prefill of a very long prompt monopolizes compute resources for a long time, causing requests that are decoding to stutter and ITL to jitter. Splitting the long Prefill into small chunks and scheduling them alternately with Decode lets the two types of requests share compute more smoothly, balancing time to first token against output stability.

    7. Prefill/Decode Disaggregation

    Because Prefill and Decode have markedly different resource characteristics (one is compute-bound, the other bandwidth-bound), you can deploy them to different instance pools, each configured with its own parallelism strategy and batching approach per its characteristics, and pass the intermediate KV over a high-speed link. This decoupling makes independent scaling easier and lets the resource allocation better match the actual load.

    5. Scheduling and Parallelism Strategies

    • Scheduling strategies: First-come-first-served, priority queues, shortest-job-first, and others each have trade-offs. Production environments care more about “throughput that meets the SLA (goodput),” which requires balancing fairness, throughput, and latency, and setting queue limits and timeout protection for requests.
    • Parallelism: When a single GPU cannot hold the model, use tensor parallelism to split the computation within a layer, and use pipeline parallelism to split the depth across layers when there are many layers; for long-context scenarios, you can also apply sequence parallelism to attention. Raising the degree of parallelism can expand capacity, but communication overhead erodes the gains, so it must be weighed against the network topology.

    6. Evaluation Metrics and Tuning Methodology

    Don’t just look at whether it “runs”; keep an eye on this set of metrics:

    • TTFT (time to first token): Mainly determined by queuing and Prefill;
    • TPOT/ITL (time per output token / inter-token latency): Mainly determined by Decode bandwidth and batch size;
    • Throughput (tokens/s): Overall output per unit of time;
    • Goodput: Effective throughput under the premise of meeting SLOs.

    For tuning, follow “locate first, then optimize”: first use small tools to run load tests, fixing the input/output length distribution and concurrency levels, and observe memory usage, GPU utilization, and arithmetic intensity to determine whether the bottleneck lies in memory, bandwidth, or compute; then apply continuous batching, paged management, quantization, and speculative decoding as appropriate; finally, use offline evaluation to hold the quality line. An example load-testing loop:

    1. for concurrency in [1, 8, 32, 128]:
    2. metrics = benchmark.run(concurrency=concurrency,
    3. input_len=2048, output_len=256)
    4. print(concurrency, metrics.ttft, metrics.tpot,
    5. metrics.throughput, metrics.goodput)

    7. Summary

    There is no single silver bullet for high-concurrency inference efficiency; it is a systems engineering effort. Starting from the “two-phase characteristics” as the cognitive foundation, and with “maximize batching, minimize KV overhead” as the main thread, you combine continuous batching, paged memory management, quantization, speculative decoding, prefix caching, chunked prefill, and prefill/decode disaggregation, along with sound scheduling and parallelism strategies. For teams that want to host large-model services on Baidu AI Cloud and related cloud products, mastering this methodology lets you, supported by general capabilities such as object storage, message queues, container platforms, and logging, monitoring, and alerting, push the latency, throughput, and cost of the inference service into a reasonable range at the same time, truly achieving “handling concurrency while controlling cost.”