LLM Inference Speed: How Token Generation Rate Reshapes the User Experience
This article breaks inference latency down into its three measurable components — TTFT, ITL, and end-to-end duration — and explains why token generation rate, not raw model capability, determines whether a task feels usable. It covers the main engineering levers for speed (speculative decoding, paged KV cache and prefix reuse, KV cache quantization, continuous batching, chunked prefill, scheduling and admission control, parallelism, kernel and precision optimization, stream pass-through), how to build your own benchmark harness with percentile-based TTFT/ITL sampling and simultaneous quality checks, which product forms only become viable once throughput crosses a threshold, a production rollout checklist, and five common measurement mistakes that hide the latency users actually complain about.
Latency is not a single number, and treating it as one makes every comparison meaningless. Before asking whether a model is “fast,” it helps to separate the three numbers that make up response time, and to understand why generation rate rather than model size now decides whether a capability actually gets used.
1. Define the Metrics First: Latency Is Three Numbers, Not One
- TTFT (Time To First Token): the time from request dispatch to receiving the first token. It breaks down into three parts — network round trips and ingestion-side queueing, prefill of the prompt, and scheduling wait.
- ITL (Inter-Token Latency): the arrival interval between adjacent tokens in steady state, approximately the inverse of the output rate. 20 ms ITL corresponds to roughly 50 tok/s; 3 ms corresponds to roughly 300 tok/s.
- End-to-end duration: approximately
T_total ≈ TTFT + (N_out - 1) * ITL
This formula explains a fact that is often overlooked: for long outputs, TTFT is nearly irrelevant, and ITL is the variable that determines the experience. For a task that must generate 10,000 tokens, users barely notice whether TTFT is 0.5 seconds or 2 seconds. But when ITL drops from 20 ms to 3 ms, total duration falls from the ~200-second range to the ~30-second range — which sits exactly on the boundary between “users will watch it work” and “users switch to another tab.”
Coding agents, long-document generation, and front-end page generation share one trait: N_out is very large (thousands to tens of thousands of tokens). That is the technical reason why the combination of “flagship capability plus high output speed” carries product meaning: capability determines whether the task can be done, and output rate determines whether that capability can be used at all.
2. Why “Fast” Used to Mean “Small”
Most high-speed models over the past year have been lightweight, because speed and capability were coupled along several dimensions:
- Parameter count and memory traffic. Decoding is memory-bound: every token requires reading the weights again. The more parameters, the higher the per-token memory bandwidth cost.
- Conservatism in decoding strategy. When a small model is weak, it often compensates with longer chains of thought and more self-verification passes, which increases the number of output tokens and makes the request slower end to end.
- Serving configurations biased toward throughput. To keep per-request inference cost low, batch sizes are set large. Large batches raise throughput but push up per-request ITL. Throughput and latency are the two ends of a seesaw, so a lot of what looks like a “slow model” is actually a scheduling decision rather than a property of the model.
So “a flagship model running at high speed” is not, at its core, about the model being smaller. It is about the inference stack pushing down the cost of three things: memory traffic, scheduling, and caching.
3. The Main Engineering Levers for Speed
3.1 Decoding Side: Speculative Decoding
A cheap “draft source” (a small model, n-gram statistics, or a multi-head prediction structure) proposes k candidate tokens at once. The target large model then verifies them in a single forward pass, accepting the longest matching prefix and resampling from the point of divergence if a token is rejected. Mathematically, it does not change the output distribution, so it loses no quality — which is exactly what makes it better than “reducing the number of reasoning steps.”
The upper bound on the gain is roughly the average accepted length, and the acceptance rate is strongly correlated with the task distribution: scenarios with high local predictability, such as code completion, templated UI, and structured JSON, typically hit significantly higher acceptance rates than open-ended writing. In practice, tune k and the draft source per scenario rather than applying one global configuration.
3.2 Memory Management: Paged KV Cache and Prefix Reuse
- Paged management. Split the KV cache into fixed-size blocks allocated on demand, and map logical sequences to physical blocks with a page table. This removes the fragmentation caused by reserving memory for the “maximum possible length.” The number of concurrent sequences a single GPU can hold often increases severalfold, which indirectly reduces queueing time.
- Prefix caching. System prompts, tool definitions, repository context, and few-shot examples are highly repetitive within a session. On a cache hit for the same prefix, prefill can be skipped entirely, improving TTFT by an order of magnitude. This requires a prefix hashing and invalidation strategy, plus hit-rate monitoring.
- KV cache quantization. Low-bit compression of the cache trades a small amount of precision for substantially more concurrent capacity.
3.3 Scheduling: Continuous Batching, Chunked Prefill, and Priority
- Continuous Batching. Scheduling granularity moves from “request” to “decode iteration.” When a sequence finishes, its slot is released immediately for a new request, so requests no longer wait on each other within a batch.
- Chunked Prefill. Prefill of a long prompt is the number-one killer of ITL tail latency: a 50k-token prompt arriving stalls every request currently decoding in the same batch. Splitting prefill into chunks and interleaving it with decode can significantly reduce ITL P99.
- Priority and admission control. Interactive requests go into a high-priority queue; offline batch jobs go into a low-priority queue and may be preempted. Apply admission control and quotas to requests with estimated very long outputs, so a single request cannot drag down the whole batch.
- Length prediction. Predict
N_outwith a lightweight model or historical statistics, and use that to decide batch composition. It is more stable than scheduling requests one at a time as they arrive.
3.4 Parallelism and Kernel Optimization
- Tensor parallelism involves a lot of communication. For cross-node deployment, use hierarchical reduction and overlap communication with computation, or the all-reduce in the decode phase will eat the gains. Pipeline parallelism has to handle bubbles and micro-batch splitting.
- With small batches during decoding, kernel launch and scheduling overhead take up a large share. Operator fusion (merging several attention-related operators, for example), memory access pattern optimization, and graph capture to fix the execution sequence and eliminate launch overhead often yield considerable direct gains.
- Low-precision inference (FP8/INT8), combined with keeping sensitive layers at higher precision, is the most direct way to cut memory traffic.
3.5 The Network Path: Stream Pass-Through and Ingestion
However fast the server computes, buffering at the gateway undoes it. Practical points:
- Gateways and load balancers must disable buffering and response aggregation for streaming responses, or users see “frozen for 30 seconds, then everything floods in at once.”
- Nearest-region ingestion, connection reuse (HTTP/2 multiplexing or long-lived connection pools), and TLS session resumption cut the fixed overhead inside TTFT — invisible to users but real.
- Under insufficient capacity, TTFT rises non-linearly with concurrency. Autoscaling should key off queue depth rather than CPU/GPU utilization alone, and queueing duration should be exposed as a first-class monitoring metric.
4. How to Produce Numbers You Can Actually Trust
Rates published by vendors are results under specific configurations, and they almost never match your scenario. Build your own evaluation harness:
Control variables. Fix input length for the prompts (three tiers: 1k / 8k / 32k), fix the output limit, fix concurrency (1 / 4 / 16 / 64), and fix the sampling protocol (streaming or not, tool calls or not).
Sample metrics. TTFT P50/P95/P99, ITL P50/P95/P99, single-stream tok/s, aggregate tok/s, and effective tokens per unit cost. Looking only at the mean hides the long tail, and the long tail is where user complaints come from.
Timing convention. Standardize the client-side start point (use the moment the request is dispatched as zero), and compute the real rate from the token usage returned by the server rather than dividing received character counts by four — token density varies widely across prose, code, and indentation.
import time, statisticsdef measure(stream_call, prompts, max_tokens):"""Single-stream latency profile: TTFT / ITL / effective rate"""ttft, itl, speed = [], [], []for p in prompts:t0 = time.perf_counter()first, prev, chunks = None, None, 0for _ in stream_call(p, max_tokens): # consume the stream chunk by chunknow = time.perf_counter()if first is None:first = nowttft.append((first - t0) * 1000) # mselse:itl.append((now - prev) * 1000)prev = nowchunks += 1total = time.perf_counter() - t0# Rigorous approach: use the server's usage.completion_tokens instead of chunksspeed.append(chunks / total)def pct(xs, q):xs = sorted(xs)return xs[min(len(xs) - 1, int(len(xs) * q))]return {"ttft_ms_p50": round(pct(ttft, 0.50), 1),"ttft_ms_p95": round(pct(ttft, 0.95), 1),"itl_ms_p50": round(pct(itl, 0.50), 2),"itl_ms_p95": round(pct(itl, 0.95), 2),"tok_per_s": round(statistics.median(speed), 1),}
Verify quality at the same time. Speed alone is not the whole picture. Run A/B tests on your own task set — compile pass rate, unit test pass rate, human blind-test win rate — and confirm the optimizations did not come at the cost of quality. Speculative decoding and prefix caching should not change the output distribution in theory, but sampling parameters, temperature, and random seed handling in real implementations can still introduce bias, so measurement is mandatory.
5. Product Forms Unlocked Once the Rate Crosses the Threshold
The value of latency is non-linear: going from 5 seconds to 2 seconds improves the experience, but going from 2 seconds to 300 ms changes the nature of the interaction. Only when the output rate reaches the hundreds-of-tokens-per-second range do the following forms genuinely hold up:
- Real-time generative interfaces. Users state a change request and the code is modified within seconds and hot-reloaded, creating a fast iterate-on-demand rhythm. This requires incremental patching (rewriting only the changed fragments) rather than regenerating whole files, otherwise the speed advantage is cancelled out by output length.
- Live parametric graphics and shader authoring. The model emits shader code and the browser compiles and renders it in sync, so user and model face the same canvas. Fault tolerance requirements here are extreme — one syntax error means a black screen — so output validation and automatic rollback are mandatory.
- Hand-drawn sketch to runnable page. HTML, CSS, and interaction logic returned end to end in tens of seconds. Here TTFT and prefix caching matter more than ITL, because the image input and the newly generated component style conventions have to be prefilled together.
- Parallel multi-agent work. Even a high single-stream rate has a ceiling; the real challenge is aggregate throughput. With dozens of agents running concurrently, aggregate rate can reach several times a single stream, but the bottleneck moves from GPU decoding to the scheduler, gateway connection counts, and rate-limit quotas. What is needed here is session-level isolation, fair scheduling, and backpressure.
- Interruptible long tasks. An understated benefit of speed is that users dare to interrupt and retry. This presupposes three things in the product design: streaming visibility, interruptibility at any moment, and rollback via snapshots.
6. Production Rollout Checklist
| Layer | Key Actions |
|---|---|
| Ingestion | Nearest-region access, connection reuse, disable gateway buffering for streaming responses |
| Scheduling | Separate interactive and offline queues; scale on queue depth and TTFT P95 |
| Caching | Prefix/semantic cache plus object storage for models and static assets; monitor hit rate |
| Observability | Percentile instrumentation for TTFT / ITL / queueing duration / cache hit rate, sampled logging with retained traces, alerting bound to SLOs |
| Cost | Cost ≈ input length × unit price + output length × unit price, modulated by cache hit rate and concurrency peaks |
| Capacity | Message queues for peak shaving and backpressure; container platform for elastic scaling of inference instances |
On general-purpose cloud platforms such as Baidu AI Cloud, these capabilities typically map to the same generic building blocks: a container platform to host inference instances, object storage for models and artifacts, and logging and monitoring services to close the observability loop. Which specific services to select should be based on your own load-testing results.
7. Five Common Pitfalls
- Reporting the experience using average ITL. The long tail is completely hidden, while users perceive P99.
- Testing only a single stream. When concurrency climbs into the tens, a throughput-biased scheduling policy can degrade ITL several times over.
- Ignoring prompt length. For the same model, TTFT at 32k context and at 1k context can differ by an order of magnitude.
- Assuming “fast means dumber.” Rate and quality are decoupling, but you still have to verify it yourself rather than infer it.
- Overlooking the client environment. Weak networks, proxies, and enterprise gateway buffering can all destroy a streaming experience, so end-to-end load testing should include real network conditions.
Bottom Line
The core progress on the inference side is that capability and latency, two dimensions that used to be coupled, are gradually becoming orthogonal. When flagship-class models also land inside the interactive latency range, product designers finally gain real freedom. And the engineering center of gravity has already shifted — from “swap in a bigger model” to “build a metrics system, a scheduling policy, and a caching system.” Whoever’s inference stack can flatten the tails of TTFT and ITL is the one who can turn model capability into an experience users actually perceive.
