Optimizing LLM Inference: Throughput, Latency, and Model Selection

    A practical guide to evaluating and optimizing large language model inference. It explains key metrics such as TTFT, TPOT, percentile latency, KV cache management, kernel fusion, continuous batching, cloud-native deployment, MoE trade-offs, AI-assisted optimization, and a practical checklist for selecting and troubleshooting an inference stack.

    1. Establish Consistent Evaluation Criteria

    Before discussing inference speed, define the metrics and test conditions clearly.

    Time to first token (TTFT) measures the time from request submission to the arrival of the first output token. It is primarily affected by input length, prefill computation, and queueing time. TTFT directly determines how long users wait before seeing the first response content.

    Generation throughput is commonly measured in output tokens per second. Another widely used metric is time per output token (TPOT), or the average time required to generate one output token. For a single request, these metrics can be converted into one another. Under concurrent workloads, however, they are no longer simple reciprocals because batching affects both the speed of individual requests and the total system throughput.

    Median latency and percentile metrics such as P95 and P99 describe the distribution of user experience. The median represents a typical request, while P99 reflects the slowest portion of requests. Looking only at the average can hide serious tail-latency problems.

    A single performance number is meaningless without context. A reproducible benchmark report should specify at least:

    • Model name and parameter scale
    • Numerical precision, such as FP16, BF16, or INT8
    • GPU model and number of GPUs
    • Inference engine and version
    • Batching strategy
    • Input and output length distributions
    • Concurrency
    • Warm-up procedure
    • Statistical methodology
    • Sampling duration

    Claims such as “thousands of tokens per second” cannot be compared or reproduced when these conditions are missing.

    2. Three Core Optimization Techniques

    2.1 KV Cache Memory Management

    During autoregressive generation, the model stores the Key and Value vectors from each layer for every generated token so they can be reused in subsequent steps. This storage is known as the KV cache. Its size grows with both sequence length and the number of concurrent requests, making long-context and high-concurrency workloads particularly demanding on GPU memory.

    If a contiguous memory region is preallocated for each request based on its maximum sequence length, much of that space may remain unused. Requests with different lengths are also difficult to process efficiently together.

    PagedAttention applies the concept of virtual-memory paging to KV cache management. It divides the cache into fixed-size blocks, allocates them on demand, and uses a block table to map a logically contiguous sequence to physically noncontiguous blocks. This can significantly reduce memory fragmentation and increase the number of requests that can run concurrently on the same GPU.

    The approach also introduces trade-offs. The block table requires additional management overhead, and block size must be tuned for the hardware and workload. Blocks that are too small increase management costs, while blocks that are too large waste more memory in the final partially used block.

    2.2 Operator Fusion and Attention Kernels

    Each small GPU operator typically reads its inputs from memory and writes its results back to memory. Combining adjacent operators into a single kernel avoids repeated transfers of intermediate results and reduces memory-access overhead.

    Common fusion targets include residual addition with normalization, as well as adjacent bias and activation operations. Fusing an entire MLP, attention module, and normalization layer into one kernel is considerably more difficult, so practical implementations generally target specific model structures and tensor shapes.

    FlashAttention is another I/O-oriented optimization. It divides attention computation into blocks and performs more of the work in on-chip storage, reducing repeated reads and writes to GPU memory. Mathematically, it computes standard attention, with differences limited to floating-point rounding. Its benefits are generally more visible with longer sequences and may be limited for short sequences.

    2.3 Continuous Batching

    Static batching requires requests in the same batch to start and finish together. Shorter requests must wait for the longest request to complete, leaving the GPU partially idle during that time.

    Continuous batching schedules work at the level of decoding iterations. After each iteration, the scheduler checks which requests have finished and immediately adds new requests. Systems such as Orca introduced this scheduling approach systematically.

    Continuous batching can substantially improve throughput, but several boundary conditions require attention.

    First is fairness. If the scheduler always prioritizes short requests, long requests may be delayed indefinitely and suffer starvation.

    Second is prefill interference. When a new request arrives, its prefill computation can interrupt ongoing decoding and increase the TPOT of other requests. Common approaches include limiting the number of prefill tokens processed in each step or using chunked prefill.

    Third is the GPU memory limit. Higher concurrency increases KV cache usage. As memory approaches capacity, the system may experience queueing, preemption, or swapping, causing latency to increase sharply.

    3. Cloud-Native Deployment and Elastic Scheduling

    For production deployments, inference services can run on public-cloud GPU instances or on self-managed GPU clusters. Once containerized, they can use orchestration platforms such as Kubernetes for replica management and rolling updates.

    Autoscaling should be driven by real workload indicators. Common signals include queue length, P95 TTFT, GPU memory utilization, and requests per second. CPU utilization alone often fails to reflect the actual pressure on GPU inference workloads.

    Scaling also has a cold-start delay. Loading large model weights can take significant time, which limits how quickly additional capacity becomes available. For workloads with sudden traffic spikes, it may be useful to maintain a number of warm standby replicas or fall back to a smaller model during peak periods to preserve availability.

    Monitoring should cover:

    • TTFT
    • TPOT
    • Throughput
    • Latency percentiles
    • GPU memory usage
    • Queueing time
    • Error rate
    • Timeout rate

    Logs should avoid recording sensitive user inputs. When necessary, sensitive data should be redacted, and retention periods should be defined.

    Cost requires separate analysis. GPU instances are billed over time, so low utilization increases the cost per token. Higher batching and concurrency generally reduce unit cost, but they may also worsen latency. The right balance should be determined according to the service-level objectives (SLOs).

    4. Mixture of Experts and AI-Assisted Engineering

    Mixture-of-Experts (MoE) is a common sparse architecture. Each token is routed through only a subset of expert networks, so the computation required per token is much lower than that of a dense model with a comparable total parameter count.

    This design introduces additional engineering challenges. The parameters of all experts still need to reside in GPU memory or system memory, so sparse activation does not eliminate the overall memory requirement. Uneven routing can overload certain experts, requiring auxiliary losses or capacity limits to control the imbalance. In multi-GPU deployments, communication between experts can also become a bottleneck.

    As for AI-assisted engineering, some teams are already using automated search and code-generation tools to assist with operator implementation, hyperparameter tuning, and selected kernel optimizations. These methods have produced measurable gains for specific hardware and operators.

    Other claims, such as the idea that a model architecture was designed almost entirely by AI, currently lack sufficient public and verifiable evidence. Such claims should be treated as potential directions rather than mature, general-purpose workflows. AI-generated operators and optimization code still require correctness testing, numerical validation, and performance regression testing.

    5. Model and Inference Selection Checklist

    When selecting a model and inference solution, use the following sequence:

    1. Define the business SLOs, including TTFT, P95 latency, target throughput, and acceptable error rate.
    2. Benchmark using the actual input and output length distributions instead of relying directly on public leaderboard numbers.
    3. Verify the inference engine version, supported precisions and quantization methods, and compatibility with the target GPU.
    4. Estimate KV cache usage at peak concurrency and confirm that sufficient GPU memory remains available.
    5. Check whether the scheduling strategy limits prefill interference and whether long requests are at risk of starvation.
    6. Confirm the autoscaling triggers, cold-start time, and fallback strategy.
    7. Calculate the cost per token and evaluate it together with the latency objectives.
    8. Confirm that logging, data storage, and data transmission meet security and compliance requirements.

    6. Common Problems and Troubleshooting Order

    When performance fails to meet expectations, troubleshoot systematically instead of immediately replacing the hardware.

    Step 1: Verify that the test conditions are consistent.
    Different model versions, precisions, input lengths, or concurrency levels produce results that are not directly comparable.

    Step 2: Check GPU memory.
    If memory usage is close to the limit, first review the maximum KV cache length and concurrency limits. Only then consider quantization or adding more replicas.

    Step 3: Examine queueing time.
    If TTFT is high while TPOT is normal, the issue is likely related to queueing or prefill. Review the scheduling parameters and chunked-prefill configuration.

    Step 4: Examine TPOT variation.
    If TPOT increases significantly with concurrency, the batch may be too large or prefill interference may be affecting decoding. Adjust the token budget for each step.

    Step 5: Check for operator fallbacks.
    Some fused execution paths work only for specific shapes, precisions, and hardware. If performance drops sharply when input shapes change, verify whether execution has fallen back to an unoptimized path.

    Step 6: Validate numerical consistency.
    After quantization or operator fusion, compare outputs using fixed inputs to evaluate quality. Speed should not be optimized at the cost of unacceptable accuracy degradation.

    7. Conclusion

    First, inference speed must be measured using reproducible criteria. Performance numbers without test conditions should not be used as the basis for selecting a solution.

    Second, KV cache management, operator fusion, and continuous batching are three major directions for inference optimization. Each has its own limits and should be evaluated against the target workload.

    Finally, the key to model and deployment selection is matching the solution to the organization’s SLOs and cost objectives. Public leaderboards and other teams’ performance figures can provide useful references, but final decisions should be based on benchmarks using the intended workload.