LLM Inference Optimization: Latency, Throughput, and Production Deployment

    This article explains why inference optimization is critical for moving large language models from experiments into production. It defines TTFT, TPOT, generation throughput, and RPS, then examines how batching improves throughput while increasing per-request latency due to GPU compute and memory-bandwidth constraints.

    1. Why Inference Optimization Determines Launch Success

    After large model training is complete, inference services are what actually consume resources and directly affect user experience. For the same model, response speed and unit cost can differ by several times under different scheduling methods, memory management strategies, and deployment topologies. Therefore, inference optimization is not additional work after training; it is a required step as a model moves from experiments to production.

    From an engineering perspective, inference optimization needs to answer three questions: how long users wait before seeing the first token, whether generation speed is smooth enough, and how many users the system can serve concurrently. These correspond to the three core metric categories below.

    2. Core Metrics: Definitions and Calculations

    2.1 Time to First Token (TTFT)

    Time to First Token (TTFT) is the interval from when the client sends a request to when the first output token is received. It consists of three parts:

    • Queueing wait time: the time a request waits in the scheduling queue for available compute resources;
    • Prefill time: the time the model takes to process all input tokens at once and generate the KV cache;
    • Network and gateway overhead: fixed costs such as request forwarding, authentication, and serialization.

    For conversational applications, TTFT directly determines whether users feel that the system is already responding. The longer the input, the larger the prefill computation; TTFT usually grows approximately linearly with input length.

    2.2 Generation Throughput and Time Per Output Token (TPOT)

    During the decode stage, each step produces only one token. The inverse of Time Per Output Token (TPOT) is the generation speed of a single request, measured in tokens/s.

    End-to-end latency can be approximated as:

    End-to-end latency ≈ TTFT + (number of output tokens − 1) × TPOT

    System-level generation throughput is the total number of tokens all requests produce per second. The two must be distinguished clearly: single-request speed determines user experience, while system throughput determines unit cost.

    2.3 Concurrent Processing Capacity (RPS)

    Concurrent processing capacity is the number of requests the system can reliably complete per unit time (Requests/Second). It has a fixed relationship with average latency and the number of in-flight requests, known as Little’s Law:

    In-flight requests ≈ request rate (RPS) × average request duration

    Example: if a service sustains 20 requests per second with an average request duration of 3 seconds, the system must handle about 60 in-flight requests at the same time. This number directly sets the lower bound for planning GPU memory, batch slots, and connection counts.

    3. Latency-Throughput Trade-off

    Latency and throughput usually trade off against each other, rooted in GPU compute and memory-access characteristics.

    3.1 Throughput Gains from Batching

    During decoding, the amount of computation is small; the main bottleneck is the memory bandwidth for reading model weights and the KV cache. When multiple requests are merged into a batch, the weights only need to be read once to serve multiple requests, increasing output per unit of computation and raising system throughput.

    3.2 Latency Cost of Batching

    As the batch grows, each decoding step processes more sequences, increasing per-step time. New requests also have to wait for the current batch’s scheduling window, increasing queueing time. Therefore, on the same machine, while throughput increases, per-request TPOT and TTFT usually get worse.