LLM Inference Optimization: A Three-Layer Approach to High Throughput and Low Latency
Why output speed has become the central metric for LLM serving, and how to reach roughly 400 tokens/s. This article explains the memory-bandwidth bottleneck that governs the decode phase, then walks through a three-layer optimization methodology: rewriting and statically scheduling the inference engine's core path to maximize single-GPU throughput, using dynamic (continuous) batching in the scheduler to protect latency under high concurrency, and tuning the underlying infrastructure for compute/communication overlap, hardware-specific operators, and production stability. It closes with practical guidance on setting differentiated latency budgets for AI coding, real-time chat, and real-time voice.
Introduction: Why Output Speed Has Become the Core Metric
When a large language model (LLM) is put into production, what users actually feel is usually not the time to first token (TTFT), but how smoothly the model keeps streaming text — the number of tokens emitted per unit of time. At around 400 tokens/s, generated text appears almost at reading pace, which makes AI coding, real-time conversation, and voice interaction feel close to natural. For this reason, 400 tokens/s has become a converging target for inference systems engineering.
Reaching that target reliably is not a matter of simply stacking more compute. It requires teams to work on three layers at the same time — the inference engine, the scheduling system, and the underlying infrastructure — and to co-optimize them against the real performance characteristics of LLM inference. This article lays out that methodology from a neutral, general technical perspective.
1. Understanding Token Generation: The Decode Phase and the Memory Bandwidth Bottleneck
Optimizing output speed starts with understanding how LLM inference actually performs. Autoregressive generation is usually split into two phases:
- Prefill: all prompt tokens are processed in parallel. This phase is computationally dense and mainly bound by compute (FLOPS).
- Decode: tokens are generated one at a time, autoregressively. Each step handles only a single new token, but has to read the entire KV cache of all previous tokens.
The defining characteristic of decode is that every token requires moving all of the model’s weights and its full KV cache from GPU memory to the compute units. The arithmetic involved is small, but the memory traffic is enormous, which makes decode a classic memory-bandwidth-bound workload.
This explains a counterintuitive result: adding compute alone does not improve output speed linearly. What actually determines decode throughput is memory bandwidth, cache hit rate, and batching efficiency. In practice, engineering for high output speed comes down to two main threads: memory access efficiency and concurrency utilization.
2. The Inference Engine Layer: Rewriting the Core Path and Static Scheduling
The inference engine is the main battlefield for performance work, and the goal here is higher effective throughput per GPU.
1. Rewrite the core inference path. General-purpose frameworks often support dynamic control flow for compatibility, which introduces a large amount of runtime branching and scheduling overhead. Rewriting the core inference path for a specific model architecture removes redundant branches and fixes the shape of the compute graph, so every decode step follows the shortest execution path. The scarce GPU memory bandwidth then goes entirely to reading the weights that genuinely matter.
2. Static scheduling. Compiling the compute graph ahead of time into a static schedule means operator fusion, memory reuse, and access layout are all decided before the service starts, rather than being re-decided at runtime. Operator fusion — merging adjacent operations such as LayerNorm, matrix multiplication, activation, and quantization — significantly cuts the number of round trips intermediate results make through GPU memory. This is one of the most effective levers for throughput in memory-bound scenarios.
3. Efficient KV cache management. Since decode is largely held back by KV cache access, paged memory management, reduced fragmentation, and better cache hit rates let a fixed amount of GPU memory hold more concurrent sequences, which multiplies overall throughput indirectly.
3. The Scheduling System Layer: Dynamic Batching and Low Latency at High Concurrency
Once single-request speed improves, the real production challenge is concurrency. The core idea at this layer is dynamic batching.
Traditional static batching waits until a full batch of requests has been assembled before inference begins, so short requests get held hostage by long ones and latency spikes. Continuous batching instead assembles the batch dynamically at every decode step: new requests can join at any time, and finished sequences leave immediately and release their KV cache. A sketch of the scheduling logic:
# Sketch: continuous batching scheduler logicwhile engine.is_running():# Assemble the batch dynamically at each step based on# current GPU memory and request statesbatch = scheduler.form_batch(max_batch_size=MAX_BATCH,token_budget=TOKEN_BUDGET)# Advance one decode step; returns sequences finished in this stepfinished = batch.advance_one_step()scheduler.evict(finished) # free the KV cache held by finished sequencesscheduler.admit(pending_queue) # admit newly arrived requests into the batch
This kind of scheduling decouples throughput from latency: even under heavy overall load, an individual request does not have to wait for unrelated requests to complete. Teams still need careful tuning between batch size (which drives throughput) and step token budget (which drives latency) in order to hold both low latency and high throughput at once under high concurrency.
4. The Infrastructure Layer: Co-Optimization and Stability
Optimizations in the inference engine and the scheduler ultimately have to land on hardware and infrastructure before they translate into stable production gains.
1. Compute and communication co-design. Large models often require tensor or pipeline parallelism across multiple GPUs, which makes inter-GPU communication a new bottleneck. High-speed interconnect, overlapping communication with computation, and topology-aware parallel partitioning can hide communication costs underneath computation.
2. Hardware and operator co-tuning. Adapting operators to the memory bandwidth, cache hierarchy, and instruction set of the specific accelerator extracts more of the hardware’s potential and avoids the performance loss left behind by generic implementations.
3. Elasticity and stability. Production services need elastic scaling, request queuing, and degradation strategies, together with proper monitoring and alerting, logging, and metrics collection. Only then does a target like 400 tokens/s become a dependably available capability rather than an occasional peak. Teams building such services in the cloud can rely on general-purpose cloud product categories such as container platforms, object storage, message queues, and monitoring and alerting to lower the cost of building and operating the infrastructure.
5. Latency Tiering and Trade-offs by Use Case
Different workloads are sensitive to latency in very different ways, and optimization should serve the specific scenario:
- AI coding and code completion: demand very high sustained streaming speed, so they need stable high tokens/s;
- Real-time interaction and chat: care about both time to first token and generation smoothness;
- Real-time voice: end-to-end latency directly determines how natural the conversation feels, and requires end-to-end co-optimization.
When designing a solution, teams should set differentiated latency budgets per scenario and make corresponding trade-offs in batching strategy, parallel partitioning, and resource allocation.
Conclusion
High output speed is not the product of any single technique, but of three layers working together. The engine layer rewrites the inference path and applies static scheduling to squeeze out single-GPU throughput. The scheduler layer uses dynamic batching to solve the latency problem under high concurrency. The infrastructure layer turns those gains into dependable production performance through co-tuning. Understanding the memory bandwidth bottleneck in the decode phase is the starting point for all of it, and the most valuable piece of knowledge for developers building highly responsive inference services.
