High-Speed LLM Decoding: Technical Principles and Engineering Practices

    Large language model inference is often treated as a straight trade-off: go faster by using a smaller, weaker model. This article shows that decoding speed is an independent engineering variable, not a model attribute, and explains how to push it to the limit without touching model weights. It breaks "speed" into measurable metrics—TTFT, TPOT/ITL, end-to-end completion time, and aggregate throughput—explains why autoregressive decoding is memory-bound, and walks through the techniques that deliver real gains: continuous batching, KV cache paging, prefix-cache reuse, prefill/decode disaggregation, speculative decoding, quantization, and serving-layer optimizations. It also shows how sufficiently fast inference unlocks new product forms, how to benchmark your own deployment using percentiles instead of averages, and the common pitfalls that distort results.

    1. Speed and capability are not two ends of a seesaw

    Over the past year, a common assumption in the industry has been that “fast means lightweight”: to get lower latency, you switch to a smaller model, accept a hit to quality, and users quietly go along with the trade. This impression does not come from any law of physics. It comes from a shortcut people take out of habit—cutting model size to go faster.

    Separating the two concerns makes things clearer. A model’s capability is determined mainly by parameter count, training-data quality, and post-training strategy. How long a token takes to come out is determined by the inference serving system: scheduling, memory management, parallelism strategy, and numerical precision. The first is a property of the model; the second is a property of the deployment. As long as you leave the weights and the forward semantics untouched, decoding speed is an engineering curve that can be optimized independently.

    So the question worth discussing is not “can a flagship model be fast,” but: given a fixed compute and memory budget, how do you push the number of tokens produced per unit of memory access to its limit?

    2. Breaking “speed” into measurable metrics

    Talking about “speed” in the abstract is meaningless in engineering; it has to be pinned to four measurable definitions.

    • TTFT (time to first token): the wait from when a user hits enter to when they see the first character. It correlates strongly with input length, prefill-phase compute, and queuing delay. Interactive products are the most sensitive to it.
    • TPOT / ITL (time per output token / inter-token latency): this governs whether the text scrolls smoothly. In practice, an output rate in the range of tens of tokens per second already reads naturally; below that, users clearly notice “stuttering” text that pops out word by word.
    • End-to-end completion time: roughly TTFT + number of output tokens × TPOT. Note that it is not linear. If a task needs to produce 3,000 tokens, decoding alone takes about 60 seconds at 50 tok/s and about 7.5 seconds at 400 tok/s. The gap is only 8x, but inside a coding agent’s multi-turn loop—read file, edit code, run tests, read logs—that multiple gets multiplied by the number of loop iterations.
    • Aggregate throughput (tokens/s): the total number of tokens the entire service emits per unit time. As concurrency rises, per-stream speed necessarily drops while aggregate throughput rises—these are two ways of slicing the same resource, not the same metric.
    Metric Meaning Key factors Where to optimize
    TTFT Time to first token Input length, queuing, prefill compute Prefix reuse, prefill/decode separation, elastic concurrency
    TPOT Output smoothness Memory bandwidth, batch size, context length Continuous batching, KV paging, quantization
    End-to-end time Task completion time Both of the above combined Streaming rendering + lower per-step latency
    Aggregate throughput Total service capacity VRAM capacity, concurrency, parallelism strategy Batching strategy, sparsification, dedicated pools

    Before comparing any two sets of numbers, you must state the concurrency, the input and output length distribution, whether streaming is on, and whether the prefix cache is hit. Without these four preconditions, a tokens/s figure is not comparable.

    3. Why decoding is a memory-bound problem

    Before understanding the optimizations, you need to understand where the bottleneck is. Each token in autoregressive decoding requires reading the entire set of model weights, plus reading the KV cache up to the current position. The compute is small; the data movement is large. As a result, the GPU’s compute units sit largely idle while bandwidth is saturated first. This is what is meant by memory-bound.

    Two directional conclusions follow. First, increasing batch size raises arithmetic intensity: the same weight read is amortized across multiple sequences, so throughput goes up. This is the basis for optimizing aggregate throughput. Second, the ceiling on concurrency is set by memory, not compute. KV cache usage can be estimated approximately as:

    1. KV_bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × bytes_per_element × batch

    As context grows, the KV cache quickly eats the memory budget and drags down the concurrency you can serve. So the real proposition behind “make a flagship model fast” is: without changing the weights, increase the number of tokens you get back per byte of memory access, and divide the limited VRAM more finely across more requests.

    4. Engineering techniques that actually work

    Continuous batching. Static batching requires a batch of requests to start and finish together; short requests must wait for long ones, empty slots cannot be filled, and severe head-of-line blocking results. Continuous batching uses each sequence as the scheduling unit: as soon as a request finishes, it is swapped out and a waiting request is swapped in, pushing GPU utilization close to saturation. This is the single change with the largest impact on throughput.

    KV cache paging. Split each sequence’s KV cache into fixed-size blocks and map logical positions to physical blocks through a block table. The benefits are three-fold: it eliminates the VRAM waste from reserving for maximum length, it significantly reduces fragmentation, and it lets read-only historical blocks be shared across multiple requests.

    Prefix-cache reuse. In real workloads, system prompts, tool definitions, codebase context, and multi-turn history are highly repetitive. Once a prefix hit occurs, the prefill compute for that portion can be skipped, and TTFT drops off a cliff. There is an easily overlooked client-side design constraint here: put stable content first and volatile content last. Do not inject timestamps or random request IDs at the start of the system prompt, or the prefix will never hit and the cache is useless.

    Prefill/decode disaggregation. The two workload types have completely different compute profiles—prefill is compute-intensive, decoding is memory-intensive. Putting them in the same resource pool guarantees that one drags on the other through its policy. Once split into two pools, each can choose its own parallelism scheme, batching strategy, and scaling metrics, with the KV cache moved between them over a high-speed interconnect.

    Speculative decoding and multi-token prediction. A lightweight draft mechanism guesses several tokens ahead, and the main model then verifies them all in a single parallel pass. The key property is that the acceptance/rejection rule guarantees the final output distribution is strictly identical to the target model. This is the strongest technical argument for “faster without losing capability”—the draft only affects how fast you compute, not whether you compute correctly. A common analysis in the industry gives, if the per-token acceptance probability is α and the proposal length is γ, the average number of tokens advanced per verification cycle as about:

    1. E[accepted] = (1 - α^(γ+1)) / (1 - α)

    The real speedup still has to subtract draft overhead and batching interactions. α tends to be high on low-entropy tasks (structured code, formatted output) and drops noticeably on high-entropy open-ended generation, so γ needs to be tuned per workload rather than hard-coded.

    Quantization and operator-level optimization. Weight quantization and lower-precision compute directly compress the number of memory bytes, and are the most on-target fix for a memory-bound problem. Efficient attention implementations, operator fusion, and graph capture are used to cut kernel-launch and intermediate write-back overhead. It is worth emphasizing that quantization is the only approximate technique mentioned here, and its quality impact must be measured against the specific model and task.

    Serving layer. Streaming transport, persistent-connection reuse, nearby ingress, end-to-end cancellation propagation, and adaptive concurrency limiting. Many complaints that “the model is slow” actually come from gateway queuing and connection setup, not from decoding itself.

    5. What changes once speed crosses the threshold

    The point of a technical metric is ultimately what it enables. When per-step interaction latency is pushed down into the sub-second to seconds range, product forms that were previously infeasible because of waiting become viable:

    • Real-time generative visuals. A single natural-language description generates a shader or visualization code in real time, and the render window refreshes almost in sync. The human-machine relationship shifts from “submit—wait—view” to real-time collaboration on the same canvas.
    • Sketch to runnable page. With a hand-drawn wireframe as input, a page skeleton—including structure, style, and interaction logic—comes out within tens of seconds, compressing the design-to-implementation loop down to an interruptible granularity.
    • A real-time world that can be hot-swapped. Enter a theme word and the scene’s colors, elements, and overall visual switch instantly—well suited to demos, education, and content production.
    • Large-scale parallel agents. When dozens of roles or subtasks advance at once, what determines end-to-end latency is aggregate throughput and scheduling fairness, not single-stream peak speed. This also shows that aggregate throughput reflects real cost efficiency better than a single-stream metric.
    • Long coding-agent chains. Per-step latency multiplied by the number of steps means that pushing single-step TTFT down to the sub-second range is what keeps a long chain “interactive”; otherwise users can only wait in offline batch mode.

    The criterion is simple: latency determines which interaction forms are mathematically infeasible. Once the threshold is crossed, what opens up is a window for new products.

    6. How to validate it on your own service

    First, fix the definitions: concurrency, input and output length distribution, whether streaming is on, prefix-cache hit rate, and quantization precision. Then collect percentiles rather than means—among p50 / p95 / p99, the long tail is what users actually feel.

    The sample code below uses the Python standard library to measure TTFT and TPOT, and can be adapted to any inference service with a compatible API:

    1. import json, time, urllib.request
    2. url = "https://<inference-service-host>/v1/chat/completions" # Example endpoint; replace with your actual one
    3. payload = {
    4. "model": "<model-id>",
    5. "messages": [{"role": "user", "content": "Write a function decorator with caching"}],
    6. "stream": True,
    7. }
    8. req = urllib.request.Request(
    9. url,
    10. data=json.dumps(payload).encode(),
    11. headers={"Content-Type": "application/json"},
    12. )
    13. t0 = time.perf_counter()
    14. ttft = None
    15. chunks = 0
    16. with urllib.request.urlopen(req, timeout=120) as resp:
    17. for raw in resp:
    18. line = raw.decode().strip()
    19. if not line.startswith("data:"):
    20. continue
    21. body = line[5:].strip()
    22. if body == "[DONE]":
    23. break
    24. delta = json.loads(body)["choices"][0].get("delta", {})
    25. if delta.get("content"):
    26. if ttft is None:
    27. ttft = time.perf_counter() - t0
    28. chunks += 1
    29. print(delta["content"], end="", flush=True)
    30. total = time.perf_counter() - t0
    31. tpot_ms = (total - ttft) / max(chunks - 1, 1) * 1000
    32. print(f"\nTTFT={ttft:.3f}s TPOT={tpot_ms:.1f}ms "
    33. f"aggregate output rate={chunks / total:.1f} units/s")
    34. # Load-test note: a single streaming chunk is not one token;
    35. # for an exact token count, use the tokenizer or the usage field returned by the service.

    Optimizations on the client side are also part of “speed”: incremental rendering and skeleton screens that show content first, support for interruption and cancellation, layered timeout-and-retry design, idempotent failed requests, and asynchronous handling for long tasks with progress feedback.

    When deploying in the cloud, the reusable general-purpose capability categories include: a container platform to host inference services and scale elastically on queue depth, a message queue to absorb traffic spikes and carry asynchronous long tasks, object storage for model weights and generated artifacts, logging and monitoring/alerting to build a token-level observability system, and a gateway to handle rate limiting and multi-model routing. Using general-purpose compute and foundational cloud services—Baidu AI Cloud, for example—as the hosting foundation lets teams focus on tuning inference parameters and product interaction instead of building the underlying infrastructure themselves.

    For capacity planning, pin concurrency to two constraints: first, that p95 TTFT stays under the interaction threshold; second, that there is enough VRAM to hold the KV cache at the target context length. Only then can you back out the cost per million tokens from load-test results and make a judgment about economics.

    7. Common pitfalls

    • Looking only at tokens/s without checking what input length and concurrency it was measured at.
    • Reporting experience with means, so long-tail stutter gets averaged away.
    • Ignoring time-based jitter in the prefix-cache hit rate and provisioning capacity for the worst case, which leaves resources idle for long periods.
    • Turning on speculative decoding indiscriminately; on high-entropy creative tasks the draft acceptance rate is low, and it actually adds total overhead.
    • Focusing only on speed while ignoring attention-quality degradation and positional-extrapolation limits over long contexts, so key information in very long inputs gets “read off.”
    • Using single-stream speed as a cost metric, when real cost should be accounted by compute usage per token.

    8. Conclusion

    “Flagship capability” and “extreme speed” were never a proposition requiring an either/or choice. What really needs to happen is to reduce speed from a marketing slogan to a set of measurable architecture metrics: TTFT, TPOT, aggregate throughput, prefix-cache hit rate, and per-GPU token cost. When these metrics are explicitly defined and continuously monitored, every optimization to the inference system can be validated, and speed naturally grows into part of the product’s strength—what it widens is not the boundary of parameters, but the ceiling on usable interaction forms.