LLM Inference Speed: The Core Capability in the Second Half of the Model Race
As model capabilities converge and open-weight models proliferate, being able to complete a task is no longer the differentiator—how fast it completes one is. This article breaks down LLM inference speed into three measurable dimensions (time to first token, time per output token, and throughput), explains why prefill is compute-bound while decode is memory-bandwidth-bound, and shows why the same weights can run several times faster or slower depending on deployment. It then covers the main optimization techniques—prefill/decode disaggregation, speculative decoding, continuous batching, KV cache management, quantization, and parallel strategies—and closes with a practical loop for measuring, selecting, and tuning inference performance.
1. The Competitive Center of Gravity: Speed as a Standalone Capability
For years, model comparisons started with parameter count, leaderboard scores, and context length. As capabilities converge and open-weight models become widely available, “can it complete the task?” is no longer the interesting question. What actually separates user experiences is “how quickly does it complete it?” That is the hallmark of the second half of the race: speed is no longer treated as a by-product of training metrics, but as a product capability that can be designed, optimized, and delivered on its own.
From an engineering standpoint, this shift was inevitable. When a model reaches production, users rarely perceive its benchmark accuracy. They perceive when the first token appears, how smoothly tokens stream out, and whether the system stutters under load. Speed determines whether a product feels usable, and it also determines how many requests a unit of compute can carry and how low the cost can go. In other words, inference speed ties two sets of books together at once: user experience and commercial efficiency.
2. Getting “Speed” Right: The Metrics You Cannot Skip
The most common mistake is treating “fast” as a single, fuzzy quantity. In real engineering, at least three dimensions need separate measurement.
- Time To First Token (TTFT): the time from request dispatch to the first token emitted. It dominates the intuitive sense of “how long did I wait after I clicked?” and is driven mainly by the prefill phase—the step where the model processes the entire input prompt in one pass.
- Time Per Output Token (TPOT): the average time to generate one token during decoding. Its reciprocal approximates per-stream output tokens per second. This metric governs the streaming feel of “does the text come out smoothly?”
- System Throughput: the total number of tokens the service can produce per unit of time. It reflects “how many users can be served at once” and is a separate objective that must be balanced against per-request latency.
The key to understanding these metrics is that inference splits into a compute-bound prefill phase and a memory-bandwidth-bound decode phase. Their bottlenecks differ, and so do the optimization levers. Two measurement traps show up constantly: equating total throughput with single-stream speed, and looking only at average latency while ignoring the long tail.
3. Why Speed Has Moved to Center Stage Now
Interactive and streaming scenarios have become the mainstream. Chat assistants, code completion, AI coding tools, and agent task chains are all fundamentally humans talking to a model in real time. People are extremely sensitive to latency: a first token that takes a second or two, or output that stutters, causes perceived quality to collapse.
Long outputs and multi-step reasoning amplify the value of speed. When output grows from a few dozen characters to hundreds or thousands—or when the model must chain several reasoning steps—per-token cost accumulates repeatedly, and differences in speed scale up sharply.
Speed is approximately cost efficiency. On the same hardware, faster decoding and fuller batching mean less compute cost attributed to each request. Speed gains often translate directly into a larger servable user base and lower per-call cost.
4. One Model, Several Times the Speed Difference
A counterintuitive but extremely common observation: with the same model and the same weights, a different deployment can produce several times—sometimes ten-plus times—the actual speed. The cause is almost always the deployment and inference system, not the model. The main variables are:
- Hardware and parallel strategy: single-GPU versus multi-GPU, and how tensor and pipeline parallelism are partitioned, directly determine memory capacity and compute efficiency.
- Precision and quantization: loading weights at FP8, INT8, or lower bit widths substantially reduces the amount of data moved during decode, which raises speed.
- Batching and scheduling: whether continuous batching is used, and how request admission and exit are orchestrated, determines whether GPU utilization can be saturated.
- KV cache management: how cache memory is allocated, reused, and evicted directly affects how many requests can run concurrently and whether GPU memory can hold up.
So the “speed frontier” is the result of jointly optimizing model, inference system, and deployment—not a property dictated by the model alone.
5. Key Technical Paths for Faster Inference
Prefill/Decode Disaggregation (P/D)
Running prefill and decode on separate execution units prevents a long prompt’s prefill from blocking decodes already in flight, improving both time to first token and output stability.
Speculative Decoding
A smaller “draft” model or a cheaper mechanism proposes several tokens, which the large model then verifies in a single parallel pass; accepted proposals are taken as-is. It reduces the number of serial decode steps in the large model with little or no loss to the output distribution.
Continuous (In-Flight) Batching
Classic static batching holds resources until every request in the batch finishes, leaving compute idle. Continuous batching lets a request exit as soon as it completes and admits a new one immediately, substantially improving system throughput and GPU utilization.
KV Cache Optimization
Paged-attention-style cache management and prefix cache reuse (a repeated system prompt hits directly) reduce memory waste and redundant computation—critical for long context and high concurrency.
Quantization and Low-Precision Inference
Lowering weights and activations to reduced precision saves memory and relieves memory-bandwidth pressure. Because decode is memory-bandwidth-bound, quantization produces especially visible speed gains.
Parallel Strategies and Graph Optimization
Partitioning tensor and pipeline parallelism sensibly, and using graph capture and operator fusion to cut kernel launch overhead, further shrinks the fixed cost of each decode step.
The snippet below is a schematic decode loop that helps illustrate why KV caching and prefix reuse speed things up. It is for principle illustration only and is not a runnable implementation:
# Schematic: autoregressive decoding with a KV cachedef decode(model, prompt_ids, max_new_tokens, cache):logits = model.prefill(prompt_ids, cache) # prefill: process the input in one pass, populate the cachenext_id = sample(logits[:, -1, :])for _ in range(max_new_tokens - 1):# With a prefix-cache hit, skip the already-computed part and forward only the new tokenlogits = model.decode_step(next_id, cache) # decode: only the newly added token is computednext_id = sample(logits[:, -1, :])return next_id
6. Engineering Practice: Measuring, Choosing, and Tuning
To make speed an actual competitive advantage, it has to sit inside a measurable, regression-aware engineering loop:
- Define the SLA before optimizing. Separate interactive goals (prioritize TTFT) from batch goals (prioritize throughput), and set a metric baseline for each.
- Load test with realistic traffic. Use input-length distributions and concurrency levels close to production, and focus on P50/P95/P99 tails rather than averages alone.
- Set up monitoring and alerting. Continuously observe time to first token, per-token latency, throughput, and memory usage so regressions surface quickly.
- Trade off cost against speed. Quantization, parallelism, and batching all add complexity; choose combinations based on how latency-sensitive the business is instead of stacking techniques indiscriminately.
- Leverage general cloud capabilities. On the deployment side, you can rely on a container platform for elastic scaling, monitoring and alerting for metric visibility, and log services for request tracing—turning the inference service into a standardized, elastic, observable, and regression-testable pipeline.
7. Conclusion
In the second half of the LLM race, the question is no longer just “how big is the parameter count, how high is the leaderboard score,” but “can you deliver the capability fast, reliably, cost-efficiently, and at scale?” Understanding the three-way relationship between time to first token, output speed, and throughput, mastering optimization paths such as prefill/decode disaggregation, speculative decoding, continuous batching, KV cache management, and quantization, and bringing speed into the measurement and tuning loop is what turns “fast” from an accidental deployment outcome into a durable product advantage.
