Decoding at 400 tokens/s: Three-Layer Optimization Practices for LLM Inference Systems

    This article explains how to achieve stable 400 tokens/s decoding performance for large language model inference. It breaks the optimization work into three layers—inference engine, scheduling, and infrastructure—and covers static execution planning, paged KV cache, chunked prefill, speculative decoding, quantization, continuous batching, admission control, interconnect topology, cold-start handling, observability, and pre-launch benchmarking.

    1. Why output speed has become a core inference-service metric

    Large language model inference has two main phases:

    • Prefill: processes the entire input prompt at once. It is compute-intensive.
    • Decode: generates output tokens one by one. It is memory-bandwidth-intensive.

    When users say that an inference service feels slow, the problem is often not a slow first token. More commonly, the delay appears as uneven gaps between tokens: generation starts smoothly but becomes noticeably choppy halfway through. Therefore, evaluating an inference service requires more than aggregate throughput measured in tokens/s. It also requires measuring the steady-state output rate for a single session.

    A decoding rate of 400 tokens/s is often treated as a representative high-watermark target. The latency budget can be derived directly:

    1. per-token interval = 1000ms / 400 = 2.5ms

    To sustain 400 tokens/s, every step in the end-to-end path—operator execution, KV cache reads and writes, sampling, scheduling, and network transfer—must fit within a millisecond-level budget. During decoding, each new token requires reading the model weights and the request’s entire KV cache again. The computation per token is relatively small, but the memory access is large. As a result, the 2.5ms budget is mostly consumed by memory traffic and framework overhead.

    This makes it impossible to reach the target through a single optimization. The engine, scheduling system, and infrastructure must be optimized together.

    2. Optimization framework overview

    Layer Main bottleneck Optimization goal
    Inference engine layer Single-device decode step time, memory access efficiency, operator scheduling overhead Reduce per-token computation and execution time
    Scheduling system layer Queue wait time, intra-batch interference, memory fragmentation Maintain latency stability under high concurrency
    Infrastructure layer Interconnect bandwidth, cold start, failure jitter, monitoring blind spots Make performance metrics sustainable and reproducible

    The division of work can be understood as follows:

    • The engine layer determines how fast one decode step is.
    • The scheduling layer determines how many requests are packed into each step and in what order.
    • The infrastructure layer determines whether the performance curve can hold under real traffic.

    3. Engine layer: making the inference path static

    3.1 Avoid over-dynamic runtime orchestration

    Many inference frameworks perform extensive shape inference, memory allocation, and graph partitioning at runtime. During prefill, these costs can be amortized. During decode, however, each step processes only one token per request, so even tens of microseconds of runtime decision-making can become a significant relative overhead.

    A common engineering approach is to move execution planning earlier:

    • Resolve tensor shapes, blocking strategies, and parallelism partitions at compile time.
    • Use static scheduling or graph capture to package a fixed kernel sequence into a single submission, reducing per-operator scheduling and synchronization.
    • Preallocate memory pools to avoid frequent memory allocation and release during decoding.

    3.2 Paged management of KV cache

    The KV cache is the largest source of memory traffic and memory consumption during decoding. Mature systems generally use block-based management: the cache is divided into fixed-size blocks, and logical sequences are mapped to physical blocks through a block table. This provides three direct benefits:

    • Eliminates memory waste caused by reserving space for the longest possible sequence.
    • Supports prefix reuse.
    • Makes it easier to share system prompts across requests.
    1. # Allocation and release of paged KV cache (pseudocode)
    2. class KVBlockPool:
    3. def can_allocate(self, req) -> bool:
    4. need = ceil_div(req.prompt_len + req.max_new_tokens, BLOCK_TOKENS)
    5. return self.free_count() >= need
    6. def alloc(self, req):
    7. blocks = self._take(ceil_div(req.prompt_len, BLOCK_TOKENS))
    8. req.block_table = blocks # logical block -> physical block mapping
    9. self._pin_shared_prefix(req) # reuse existing physical blocks for shared prefixes
    10. def release(self, req):
    11. for b in req.block_table:
    12. if self.ref_count(b) == 1: # do not reclaim if still referenced by shared prefix
    13. self._return(b)

    3.3 Chunked prefill and speculative decoding

    When long prefills and decoding steps are mixed in the same batch, a long prompt can block the decode steps of other requests, causing the output rate to drop sharply. Chunked prefill splits a long prefill into fixed-size chunks and interleaves them with decode steps. This trades a small amount of prefill latency for more predictable steady-state output speed.

    Speculative decoding is another orthogonal approach. A low-cost draft model or an n-gram strategy predicts multiple candidate tokens at once, and the main model verifies them in a single forward pass. When the acceptance rate is high, one forward pass can produce multiple tokens, giving a meaningful speedup. However, the acceptance rate varies significantly with task distribution, so an adaptive switch is needed to avoid increasing overhead when predictions are wrong.

    3.4 Quantization and operator fusion

    Weight quantization directly reduces the number of bytes that must be moved during decoding, making it one of the most direct levers for improving decode speed. Quantizing activations and the KV cache requires more careful evaluation of accuracy loss.

    Operator fusion—such as merging normalization, projection, and activation operations—reduces kernel launches and avoids extra memory round trips for intermediate tensors. For decode workloads characterized by small computation and many launches, this benefit is especially pronounced.

    4. Scheduling layer: latency stability under high concurrency

    4.1 Continuous batching replaces static batches

    Static batching must wait for the slowest request in the batch to finish before releasing resources. As a result, short requests can be held back by long ones.

    Continuous batching, also called dynamic batching or iteration-level scheduling, allows completed requests to exit and new requests to enter after each iteration. This improves both memory utilization and queue wait time.

    1. # Iteration-level scheduling loop (pseudocode)
    2. running, waiting = deque(), deque()
    3. while True:
    4. for req in finished_or_truncated():
    5. kv_pool.release(req) # 1. Release memory for completed requests
    6. while waiting and len(running) < MAX_CONCURRENCY \
    7. and kv_pool.can_allocate(waiting[0]): # 2. Admit new requests within budget
    8. running.append(waiting.popleft())
    9. batch = build_step_batch(running, token_budget=TOKEN_BUDGET)
    10. logits = engine.forward(batch) # 3. Chunked prefill plus one decode token per request
    11. for req, tok in zip(batch, logits.argmax(-1)):
    12. if tok == EOS or len(req.output) >= req.max_new_tokens:
    13. mark_finished(req)
    14. else:
    15. req.append(tok)

    4.2 Admission control: decide whether to accept requests based on SLO

    What often determines whether the latency curve stays smooth is not the scheduling algorithm alone, but admission control. If the system continues accepting requests after memory or concurrency has reached its limit, it can trigger queuing, memory preemption, and recomputation. The steady-state output rate can then collapse.

    A simple model can be used to estimate per-request time per output token:

    1. def estimate_tpot(per_token_ms, batch_size, overhead_ms):
    2. """Estimate per-request time per output token (illustrative model; parameters must be calibrated with measurements)"""
    3. return per_token_ms * batch_size + overhead_ms
    4. def admit(req, running, per_token_ms, overhead_ms, budget_ms=2.5):
    5. est = estimate_tpot(per_token_ms, len(running) + 1, overhead_ms)
    6. return est <= budget_ms # if over latency budget, queue or degrade

    Related scheduling strategies usually include:

    • Ordering requests by remaining output budget rather than input length to suppress head-of-line blocking.
    • Assigning different priorities and independent resource pools to interactive requests and batch offline tasks, preventing large offline jobs from crowding out interactive traffic.
    • Setting a per-request max_new_tokens limit to prevent a few long generations from occupying memory for extended periods.

    5. Infrastructure layer: making 400 tokens/s production-deliverable

    A high watermark measured in a single benchmark is not enough. Production readiness requires the target to hold under real traffic and fault injection.

    • Interconnect and parallel topology: Tensor parallelism is highly sensitive to interconnect bandwidth. Cross-node communication can become a hard lower bound on decode step time. Parallel groups with frequent communication should be placed within the same high-speed interconnect domain to avoid crossing domain boundaries.
    • Memory and compute ratio: Decoding is memory-bandwidth-bound, so simply adding compute provides limited benefit. Priority should be given to improving effective memory bandwidth and memory capacity utilization. Techniques such as quantization, prefix reuse, and KV cache tiering across memory, DRAM, and local storage can increase servable concurrency.
    • Elasticity and cold start: Loading model weights takes much longer than starting a container. Warm replica pools, readiness probes, and graceful traffic draining are needed to prevent first-token latency spikes during scale-out.
    • Observability: At minimum, collect time to first token, token interval, steady-state output rate, queue wait time, batch size distribution, KV cache utilization, and preemption counts, and configure tiered alerts. Missing any of these metrics makes it difficult to diagnose intermittent slowdowns.

    6. Pre-launch evaluation methodology

    1. import statistics as st
    2. def summarize(ms_samples):
    3. s = sorted(ms_samples)
    4. q = lambda p: s[min(len(s) - 1, int(len(s) * p))]
    5. return {"p50": q(0.5), "p95": q(0.95), "p99": q(0.99), "max": s[-1]}
    6. def session_tps(first_token_ms, token_gaps_ms, total_ms):
    7. out_tokens = len(token_gaps_ms) + 1
    8. steady = (total_ms - first_token_ms) / 1000.0 # steady decoding time (seconds)
    9. return out_tokens / steady if steady > 0 else float("inf")

    Three points deserve attention during evaluation:

    1. Report first-token latency and steady-state output rate separately, because they are governed by different bottlenecks.
    2. Run stepwise concurrency stress tests to find the point where the output rate begins to degrade, rather than reporting only the best result.
    3. Focus on long-tail latency. When p99 is an order of magnitude worse than the average, user experience is largely determined by the tail.

    Every engine or scheduling change should retain a baseline snapshot, forming a performance regression gate.

    7. Suitable scenarios and common pitfalls

    High output speed is most valuable in scenarios such as:

    • AI code completion, which requires fast, continuous generation of large code blocks.
    • Real-time voice interaction, where the voice path itself has strict end-to-end latency requirements.
    • Intelligent programming and debugging assistants.
    • Human-in-the-loop commercial decision support.

    These scenarios share one characteristic: each request produces long output, and the user remains attentive while waiting.

    Common pitfalls include:

    • Treating aggregate throughput as user experience.
    • Pursuing peak speed by disabling admission control, which causes collapse during traffic peaks.
    • Quantizing models without evaluating long-tail accuracy regressions.
    • Optimizing only the engine while ignoring scheduling and networking, so the 2.5ms budget is consumed by framework overhead and cross-node communication.

    In production deployments, it is better to reuse a common container platform, monitoring and alerting, and logging services to build the evaluation and observability loop, and to encode performance baselines into the release process rather than relying on manual stress testing every time.

    8. Summary

    The essence of 400 tokens/s is not an isolated technique. It is the result of precisely allocating and defending a per-token latency budget of about 2.5ms across three layers:

    • The engine layer reduces per-step cost through static scheduling, paged KV cache, chunked prefill, and quantization.
    • The scheduling layer maintains stability under concurrency through iteration-level batching, memory budgeting, and admission control.
    • The infrastructure layer makes the metrics reproducible through topology planning, elastic warm-up, and complete observability.

    Only when these three layers form a measurable, regression-tested engineering loop can high throughput and low latency become a sustainable delivery capability rather than an isolated number in a benchmark report.