LLM Inference Acceleration and Deployment: A Practical Guide to General GPU Software Stacks

    This article presents a practical methodology for accelerating and deploying large language model inference on general-purpose GPU software stacks. It explains how to quantify inference goals, track throughput, first-token latency, per-token latency, memory usage, and tail latency, and identify bottlenecks across prefill and decode phases. It then covers operator fusion, kernel tuning, attention optimization, static graph processing, numerical precision and quantization, and batching and request scheduling, with emphasis on trade-offs and implementation order.

    1. Background and Problem Definition

    The cost of large language model inference comes mainly from two areas: the time hardware resources are occupied and the quality of service response under high concurrency. Training focuses on convergence speed, while inference is more concerned with how many requests can be processed per unit time, whether first-token latency is acceptable, and whether GPU memory can hold model parameters and intermediate states. Therefore, inference optimization should first quantify the target, then select the appropriate methods.

    In real projects, the same model can perform several times differently under different deployment methods. The differences come from a combination of compute efficiency, memory bandwidth, scheduling strategy, and numerical precision. Without systematic analysis, teams can easily spend effort on isolated parameter tuning with limited returns.

    2. Core Performance Metrics

    1. Throughput: the number of generated tokens completed per unit time, used to measure the overall capacity of offline batch jobs or high-concurrency services.
    2. Time To First Token (TTFT): the time from request arrival to the output of the first token, mainly affected by prompt length and prefill-phase efficiency.
    3. Time Per Output Token (TPOT): the average time per token in the generation phase, which determines whether streaming output is smooth.
    4. GPU memory usage: includes model weights, activations, and KV cache.
    5. Tail latency (such as P99): reflects service stability under extreme traffic.

    These metrics involve trade-offs. Increasing batch size usually improves throughput but lengthens the waiting time of individual requests. Before optimization, the business priority should be clarified. Interactive dialogue places more emphasis on TTFT and TPOT, while offline summarization tasks place more emphasis on throughput.

    3. Main Bottlenecks in the Inference Pipeline

    Large language model inference is usually divided into two phases: prefill and decode. The prefill phase processes the input prompt at once. It is compute-intensive and relatively easy to fully utilize hardware compute capacity. The decode phase generates tokens one by one. Each step involves little computation but repeatedly reads weights and the KV cache, so it is often limited by memory bandwidth. The two phases have different bottlenecks, and optimization strategies should treat them differently.

    Common bottlenecks include:

    • Operators are too fine-grained, and frequent small operator calls introduce scheduling overhead.
    • Intermediate results are repeatedly written back to GPU memory, making memory access a major time cost.
    • Batching strategy is rigid, and short requests must wait for long requests to finish.
    • KV cache fragmentation reduces the number of available concurrent requests.
    • Data is frequently moved between the CPU and the accelerator.
    • Tokenization, sampling, post-processing, and other stages run serially, slowing down the overall pipeline.

    4. Optimization at the Operator and Computation Graph Level

    4.1 Operator Fusion

    Merge adjacent elementwise or reduction operations into a single compute kernel to reduce GPU memory reads and writes. Typical combinations include normalization with residual connections, linear layers with activation functions, and softmax with scaling in attention computation. After fusion, intermediate tensors do not need to be written back to GPU memory, and memory traffic drops noticeably.

    4.2 Kernel Parameter Tuning

    For specific shapes (sequence length, hidden dimension, number of heads), choose appropriate tile sizes and thread organization. The optimal configuration for the same operator can differ across input sizes, so parameters need to be determined through automatic search or empirical mapping tables.

    4.3 Attention Computation Optimization

    Use a tiled attention implementation to avoid explicitly materializing the full attention matrix. This reduces peak memory usage and improves data reuse in on-chip cache.

    4.4 Static Graph Processing

    Before deployment, perform static analysis on the computation graph, fold constant computations, remove dead nodes, and define bucketing rules for dynamic shapes. Runtime graph construction overhead is then reduced.

    Illustrative code (pseudo-code, only to show the flow):

    1. model = load_model(path)
    2. model = fuse_ops(model, patterns=["norm_residual", "linear_act"])
    3. model = compile_graph(model, shapes=[(1, 512), (1, 2048)])

    5. Numerical Precision and Quantization

    Precision selection is the part of inference optimization with the largest potential benefit and the highest risk. Common data formats include single-precision floating point, half-precision floating point, BF16, and INT8. Low-precision formats reduce storage and bandwidth consumption and may improve compute throughput, but they introduce rounding error and quantization error.

    When implementing quantization, proceed in the following steps:

    1. Establish an accuracy baseline in FP16 or BF16 and record evaluation metrics for key tasks.
    2. Select calibration data that covers the input distribution of real business traffic, rather than using only generic corpora.
    3. Collect activation ranges per layer or per channel to determine quantization scaling factors.
    4. Keep higher precision for sensitive layers, such as the output layer and some normalization layers.
    5. Compare metrics before and after quantization, and move to stress testing only after the error is acceptable.

    Weight quantization usually reduces GPU memory usage significantly, but activation quantization is more sensitive to outliers. If accuracy drops noticeably, try outlier smoothing, mixed precision, or layer-wise fallback.

    6. Batching and Request Scheduling

    Static batching requires all requests to have the same length or to be padded to the same length, and the padding wastes compute resources. Dynamic batching allows requests to join or leave a batch during runtime, which can noticeably improve resource utilization.

    The core mechanism of continuous batching is to check, after each decoding step, whether any requests have finished.