Ultra-Fast APIs from Flagship Models: An Engineering Look at High-Speed LLM Inference
Flagship LLMs are usually assumed to be slow, but the bottleneck at the decode stage is not linearly tied to model size. With enough engineering work on scheduling, cache management, and decoding algorithms, a flagship-class model can deliver near-real-time output without sacrificing capability. This article explains the key metrics for measuring LLM speed (TTFT, output speed, and system throughput), then walks through five mainstream optimization levers—speculative decoding, paged attention, prefix caching, prefill–decode disaggregation, and quantization plus kernel tuning. It also covers how to verify that capability is preserved, four scenarios that benefit most from high-speed inference, and practical guidance for integrating a high-speed inference API.
1. The Question: Why Can Flagship Models “Spray” Code?
In most developers’ experience, the stronger a model is, the slower it responds. LLM inference is usually split into two phases: first it “reads” the input, then it “writes out” the answer token by token. The phases differ in their hardware bottlenecks and optimization approaches, summarized in the table below:
| Phase | Compute characteristics | Main bottleneck | Typical optimization directions |
|---|---|---|---|
| Prefill | Processes all input tokens in parallel at once | Compute | Chunked prefill, operator and kernel optimization, parallel partitioning |
| Decode | Generates only 1 token per step | Memory bandwidth and capacity | Continuous batching, speculative decoding, KV cache reuse and compression |
The prefill phase determines how long users wait before they see the first token, while the decode phase determines whether generation stays smooth. At the same time, the larger the model, the more complex each layer’s computation, and the more compute resources each inference call requires. So the intuition that “flagship models are slow by nature” seems to hold up.
But if you carefully break down where the slowness comes from, you find it is not insurmountable. Over the past year, when the industry advertised inference speed, it often assumed by default that “high speed” meant a smaller, lighter model: cut the parameter count, and speed goes up while capability takes some kind of hit. Users gradually grew used to this trade-off. The real question worth exploring is: when a model’s parameter scale and capability tier stay unchanged, where else can inference serving “squeeze out” several times the speed?
The answer is that the decode bottleneck is not linearly bound to model size. As long as you do enough engineering work around compute scheduling, cache management, and decoding algorithms, a flagship-class model can plausibly produce output speeds close to real time.
To take a generation speed of 400 tokens per second as an example: a human blink takes about 0.3 seconds, and in that time the model has already written roughly 120 tokens. That is enough to output a complete code function or a short natural-language section. Once generation speed exceeds a user’s reading speed, “waiting” shifts from a latency you have to endure into a nearly imperceptible process.
Coding tasks benefit especially. Developers read code far faster than they read natural language, and a model “spraying” code at several hundred tokens per second lets the user keep their train of thought rather than staring at the screen waiting for an answer. Of course, 400 tokens/s is better understood as a benchmark for the engineering level and performance target that high-speed inference serving represents, rather than a measured value that every deployment can reliably reproduce. Actual speed depends on input length, hardware configuration, concurrent load, network links, and other factors, and needs to be verified in your own scenario.
2. First, Clarify the Concepts: Which Metrics Actually Measure Model Speed
Before discussing “fast,” you need to turn the fuzzy sense of experience into quantifiable engineering metrics. LLM inference serving usually involves three key metrics:
- Time To First Token (TTFT): The time from when a request is sent to when the first output token arrives. It is determined mainly by the prefill phase and directly shapes the user’s first impression of “how fast it responds.”
- Output Speed: The average time to generate each token during the decode phase, usually expressed in tokens per second (tok/s). It determines whether long-form generation stays smooth.
- Overall system throughput: The total tokens a server produces per unit time across all concurrent requests. It measures the cluster’s carrying capacity and is a different layer of metric from the per-request speed a user perceives.
Note in particular that a fast single-request output speed does not mean high overall throughput. For example, batch inference with a small model can achieve considerable overall throughput, but too many queued requests lengthen TTFT. Conversely, simply scaling up batch size to raise overall throughput can also lengthen the generation interval for an individual request. Fixating on a single number easily leads to one-sided conclusions.
From this, three easily overlooked corollaries follow.
First, output length is a “multiplier” on speed. The end-to-end time of a typical conversation can be approximated as TTFT plus “number of output tokens ÷ output speed.” If TTFT is 0.8 seconds and output speed is 50 tokens/s, generating 2000 tokens takes about 40 seconds, for a total of roughly 41 seconds. Holding everything else constant, raising output speed to 400 tokens/s drops generation time to 5 seconds and end-to-end time to about 5.8 seconds—an overall reduction of roughly 86%. As output gets longer, the perceived improvement from higher speed becomes more pronounced, and tasks like code generation and long-form writing are exactly the typical long-output cases.
Second, speed headroom has phase-dependent value. A person’s Chinese reading speed is usually well below 100 tokens per second, and once a model’s output speed clearly exceeds reading speed, the marginal benefit of going faster for a single read-through starts to diminish. But that headroom gets “cashed in” again across a long task chain: when an agent runs a dozen-plus reasoning steps in a row, or a coding assistant rewrites multiple files back to back, the few seconds saved at each step keep accumulating.
Third, TTFT and output speed need to be optimized separately. Their bottlenecks differ and so do the techniques. Looking only at TTFT, you may miss the wait during long-form generation; looking only at output speed, you may miss the long blank stretch before the user even enters a waiting state.
In engineering, these two metrics are usually measured with a minimal streaming request like this:
import timedef measure_stream(client, prompt: str, max_tokens: int = 2048):start = time.perf_counter()ttft = None # Time To First Tokentoken_count = 0 # Number of tokens produced by this requestt_prev = startintervals = [] # Inter-arrival intervals between adjacent tokens, used to observe TPOTstream = client.chat.completions.create(model="flagship-model",messages=[{"role": "user", "content": prompt}],max_tokens=max_tokens,stream=True,)for chunk in stream:now = time.perf_counter()delta = chunk.choices[0].deltaif delta and delta.content:if ttft is None:ttft = now - startintervals.append(now - t_prev)t_prev = nowtoken_count += 1print(delta.content, end="", flush=True)total = time.perf_counter() - startprint(f"\nTTFT: {ttft:.2f}s | "f"Output speed: {token_count / max(total - ttft, 1e-6):.1f} tok/s | "f"End-to-end latency: {total:.2f}s")
This example uses the Chat Completions-style protocol that mainstream services commonly support, enabling streaming output with stream=True. When the first token arrives versus the pace at which subsequent tokens arrive correspond to TTFT and output speed respectively. You can also log the interval between each token (often called TPOT in the industry) to observe whether generation stalls anywhere.
One thing worth noting: in the underlying inference framework, continuous batching is usually scheduled on the granularity of a scheduling cycle. Each cycle has to balance prefill for new requests against decode for existing requests, so a single request’s token arrival intervals are not strictly uniform. Output speed should therefore be taken as the average over the entire generation process, not inferred from individual intervals. TTFT and end-to-end time, meanwhile, should be sampled multiple times to observe percentiles such as p50 and p95, rather than relying on a single result.
3. Five Key Levers for Speeding Up Flagship Models
Once the metrics are clear, the real engineering challenge is: how do you push TTFT under 1 second and raise output speed into the several-hundred-tokens-per-second range without sacrificing capability? The mainstream industry approaches to inference optimization can roughly be grouped into the following five directions.
1. Speculative Decoding: Break the Serial, Token-by-Token Constraint with “Draft with a Small Model, Verify with a Large One”
The fatal weakness of standard autoregressive decoding is its strict serial dependency: the nth token cannot start computing until the (n-1)th is finished, leaving much of the GPU’s compute idle while it waits. The idea behind Speculative Decoding is to have a lightweight “draft model” guess several candidate tokens in a row, then have the target model verify them all in parallel in one pass. A verified prefix is accepted directly; at the first mismatch, resampling happens from the target model’s own distribution, and the wrong guesses are discarded.
This mechanism has a key property: the output distribution is exactly the same as decoding with the target model alone, so it is lossless to model capability in principle, while the speedup depends on the draft model’s hit rate. Put simply, inference shifts from “one token per step” to “look at the answer first, then verify it in batches.”
A few boundaries are worth noting in practice:
- For highly predictable, structured content like code and JSON, the draft hit rate tends to be higher and more substantial speedups are easier to obtain;
- For divergent tasks like creative writing, hitting the draft is harder, so the payoff is correspondingly limited;
- The draft model itself consumes memory and compute, and needs to be co-tuned with the target model.
For a model pursuing flagship capability, speculative decoding is one of the few techniques that is “both fast and lossless to capability.”
2. Paged Attention: Allocate the KV Cache On Demand with Less Fragmentation
During decode, every new token requires reading the key-value cache (KV Cache) of all prior tokens. Cache memory grows linearly with “number of concurrent requests × context length.” In long-context scenarios, it quickly becomes a memory consumer on par with the model weights. The traditional approach reserves a contiguous block of memory for each request, and the unused portion cannot be lent to others—both wasteful and a limit on batch size.
Paged Attention borrows the idea of virtual-memory paging from operating systems, splitting the KV cache into fixed-size blocks allocated on demand, and maintaining a block table that maps logical sequences to physical blocks. The benefits are threefold:
- Memory fragmentation drops sharply, so the same amount of memory holds more concurrent requests;
- When different requests share an identical prefix, the corresponding physical blocks can be reused directly;
- With better memory utilization, continuous batching can “fit” larger batches.
The estimate below makes this more intuitive. Suppose a model has 60 layers, each with 8 key-value heads and a head dimension of 128, and the cache is stored in FP16 (2 bytes per element). Then each token’s KV cache is about:
60 layers × 2 (K and V) × 8 heads × 128 dims × 2 bytes = 245760 bytes ≈ 0.23 MiB/token
If you serve 256 requests at once with an average context length of 8192, the KV cache alone needs about 480 GiB of memory. And this is only the theoretical floor for a medium-scale dense model, before accounting for framework reserves and temporary buffers—which shows just how much memory pressure long-context inference creates. In addition, paged access by block is weaker on locality than large contiguous reads, so high-performance implementations often have to balance block size against memory access patterns.
3. Prefix Caching: Stop Making Every Request “Read From Scratch”
Many application scenarios have large repeated prefixes in their input: a fixed, unchanging system prompt, long reference documents, the message history of a multi-turn conversation, a code repository’s context, and so on. If every request recomputes all of this, it is not just wasted compute—it is one of the biggest sources of TTFT.
The idea behind Prefix Caching is to build an index via block-level hashing of token sequences, and on a hit, reuse the existing KV cache directly, skipping the corresponding prefill computation and computing only the new portion. A few things to note in practice:
- The cache key must cover every factor that affects the model input, including the model version and the sampling parameters that affect inference. Otherwise you may wrongly reuse another request’s result and cause answer cross-contamination;
- Only reuse a fully matching prefix, to guarantee semantic consistency;
- How the prefix is organized directly affects the hit rate. Putting stable content first and volatile content last (for example, placing the user’s current question at the end of the prompt) can significantly improve reuse efficiency.
How big is the effect? Consider an estimate. Suppose an input contains 8000 reusable prefix tokens and 500 new tokens, under a prefill compute rate of about 100,000 tokens per second:
- On a miss, prefill takes about 85 milliseconds;
- On a hit, only the new portion needs computing, about 5 milliseconds.
TTFT drops by about 94% this way. And if each conversation turn also carries a system prompt of a few thousand tokens, the cumulative benefit across a long session is even greater.
4. Prefill–Decode Disaggregation: Keep “Reading Fast” and “Writing Steadily” Out of Each Other’s Way
Prefill and decode make completely different demands on hardware:
- Prefill processes hundreds to thousands of input tokens at once—a compute-intensive task that “maxes out” compute;
- Decode generates only one token per step—a memory-access-intensive task that leaves most compute idle while waiting on memory data.
When the two request types share the same instance, one long-input prefill will “crowd out” the compute resources of requests that are decoding, causing output-speed jitter and token arrival intervals that swing long and short. Prefill–decode disaggregation (often called PD disaggregation in the industry) splits the two tasks into separate instance pools:
- The prefill pool handles “reading,” pursuing high compute and high concurrency;
- The decode pool handles “writing,” pursuing high memory bandwidth and low per-step latency;
- The two are connected by passing the KV cache over a high-speed network.
The ratio of the two can be adjusted dynamically based on the business’s input/output length distribution. Scenarios with long inputs and short outputs (such as document summarization) can allocate more prefill resources; scenarios with long outputs and short inputs (such as code generation) do the reverse. The core value of this architecture is isolating the performance of the two pipelines from each other, so each can be tuned independently.
5. Quantization and Inference Kernel Optimization: Cut Memory-Access Overhead, Squeeze Out Hardware Utilization
- Weight quantization: Low-precision formats such as FP8 and INT4 significantly reduce the amount of weight data that must be moved per decode step, directly easing memory-bandwidth pressure. But quantization is not lossless, and you must confirm output quality stays stable through systematic evaluation—especially for precision-sensitive tasks like mathematical reasoning and code generation;
- KV cache quantization: Lowering cache precision can multiply the concurrency the system can hold, making it an important lever for long-context serving;
- Inference kernel optimization: Fusing attention computation, reducing memory reads and writes for intermediate results, and using graph capture to eliminate kernel launch overhead can all compress per-step time at the operator level.
These techniques are not substitutes for one another; they stack. A mature high-speed inference service is usually a combination of “speculative decoding + paged KV cache + prefix reuse + continuous batching + kernel optimization.”
4. Verifying “No Capability Loss”: What Standard Tells You a Flagship Model Lives Up to Its Name
After the speed improvements, one question must be answered: is capability still at the flagship level? The third-party leaderboards common in the industry are mostly human-preference evaluations, measuring across two dimensions: output quality, i.e., the completeness and persuasiveness of generated results; and generation speed, i.e., how fast the model goes from its first reply to a complete result. Such leaderboards do provide valuable cross-comparison references, but they should also be viewed rationally:
- Leaderboards measure “preference,” not “correctness.” The generation-speed dimension rewards fast output, which is not fully aligned with the content correctness users care about;
- The distribution of competition problems is limited and struggles to cover the real data distribution of a company’s own business;
- Leaderboard rankings shift over time and should not be the sole basis for a decision.
A more reliable approach is to build an evaluation set tailored to your own business, covering at least three task types:
- Code generation and completion: Quantified with metrics like compilation pass rate and unit test pass rate;
- Structured output: Quantified with metrics like JSON field completeness and schema compliance rate;
- Instruction following: Quantified with metrics like format-constraint satisfaction rate and multi-turn context consistency.
At the same time, you should measure TTFT, output speed, and end-to-end time before and after optimization separately, and compare them with the same evaluation script under the same hardware configuration and concurrent load—only then can you judge whether the speedup came at the cost of quality or stability.
5. What High-Speed Inference Delivers: Four Typical Scenarios
The value of high-speed inference ultimately shows up in business scenarios. The following four task types are highly sensitive to TTFT and output speed and are worth integrating first.
The first is AI coding assistants. Code generation is a typical long-output task, and output speed directly determines the experience. When engineers write code, their train of thought is often fleeting; if the model needs tens of seconds to deliver a complete solution, attention has likely already shifted. A flagship-class model paired with a high-speed API can render code in real time at close to “typing” speed, and combined with streaming output and incremental editing, users can course-correct as generation proceeds.
The second is agents and tool calling. A complex task is often decomposed into multiple reasoning steps and multiple tool calls, and each step waits on the model’s output. The few seconds saved per step get amplified across a dozen-plus-step task chain into a reduction of tens of seconds or even minutes overall—especially critical for interactive agents that need fast feedback.
The third is long-document understanding and summarization. Inputs easily run to tens of thousands of words, and prefill time makes up a very large share of TTFT. Prefix caching and paged attention can significantly ease this bottleneck, turning long-context handling from “barely usable” into “smoothly usable.”
The fourth is low-latency conversational applications. Scenarios like real-time customer service and voice assistants have near-demanding requirements on TTFT. The high-speed version of flagship models makes these scenarios feasible for the first time with “strong capability + low latency.”
6. Integration Practice: How to Get the Most Out of a High-Speed Inference API
When choosing a high-speed inference API, a few engineering details are easy to overlook yet directly affect the final experience and cost.
First, prioritize streaming output. A non-streaming call can only wait for the full result to return, so in perceived terms TTFT equals end-to-end time and the high-speed API’s advantage is completely erased. A streaming call lets users see content sooner and also makes interruption control easier.
Second, load-test real throughput with concurrency, not serial requests. The peak speed from a single request does not represent how the service behaves under real load. You should use progressively increasing concurrent requests to measure how TTFT and output speed change as concurrency rises, find the “sweet spot” for your business, and avoid blindly chasing the theoretical limit.
Third, bring TTFT into your monitoring metrics. Routine monitoring often covers only end-to-end time and error rate, but TTFT is the core of how users perceive “fast” versus “slow.” It is advisable to compute percentiles for TTFT (such as p95 and p99) and set up independent alerts.
Fourth, get your timeout and retry strategy right. High-speed inference responses are usually fast, so the timeout threshold can be set shorter than for traditional LLM calls. But you should distinguish “first-token timeout” from “overall timeout”: the former indicates serious prefill queuing, while the latter may mean the output is too long. The two need different handling strategies.
Fifth, do capacity planning from a cost perspective. For the same workload, a higher-throughput inference service can support the same concurrency with fewer resource instances, and the per-token cost often has an advantage. When scaling out, base capacity estimates on the measured concurrency–latency curve rather than linearly extrapolating from single-request peak speed.
7. Conclusion: “Having Both” Speed and Capability Is a Victory of Inference Engineering
For a long time, developers choosing an LLM faced a single-choice question: capability or speed. Strong models were slow, fast models were limited in capability, and you could only make trade-offs per scenario. This situation is now being broken by advances in inference engineering. The maturing of techniques like speculative decoding, paged attention, prefix caching, PD disaggregation, and quantization optimization gives flagship-class models, for the first time, a real chance to deliver both “no capability loss” and “ultimate speed” at once.
For developers, this means a shift in product design logic. Features that used to require a compromise between “waiting” and “settling”—such as real-time code generation, real-time long-context Q&A, and multi-step agent task chains—can now all be designed in their ideal form.
But it is also important to see clearly that speed and capability are not equally important in every scenario. Batch offline tasks are insensitive to latency and pursue ultimate throughput and the lowest cost; it is user-facing interactive applications that are the real battleground where high-speed inference unleashes its value. When making a choice, base your judgment on the business’s real latency requirements rather than blindly chasing benchmark numbers. In engineering there is never an “all-purpose solution,” only the “most suitable one.”
