Flagship Models Hit 400 tok/s: Key Techniques Behind High-Speed LLM Inference
High-speed inference is enabling flagship LLMs to reach roughly 400 tokens per second without sacrificing capability. This article breaks down the key metrics—TTFT and TPOT—and explains the main optimization techniques behind fast inference, including speculative decoding, low-bit quantization, continuous batching, chunked prefill, prefill/decode disaggregation, prefix caching, and kernel-level optimizations. It also covers how callers can preserve end-to-end responsiveness with streaming output, incremental rendering, timeout and cancellation policies, and how to benchmark and choose high-speed flagship APIs for latency-sensitive workloads.
I. Introduction: Why Speed and Capability Were Once Hard to Combine
Over the past year, “fast” in LLM services has often meant “small.” Lightweight models respond quickly, but they may compromise on complex code understanding, long-context reasoning, and instruction following. Flagship models are more capable, but generating long responses can make users wait tens of seconds or even minutes while watching the cursor blink. Developers have gradually accepted this trade-off.
Interactive programming amplifies the tension. Asking a model to generate a complete personal blog page—with navigation, article cards, data charts, and other components—can produce tens of thousands of tokens. At typical output speeds, this can take several minutes, during which the developer can do little else.
Technical progress is changing this trade-off. High-speed APIs from leading providers have pushed flagship-model output to around 400 tokens per second, with time to first token kept under one second. If a blink lasts about 0.3 seconds, the model can write about 120 tokens during that time. For the same page-generation task, high-speed mode can finish in about one minute, while a conventional mode running at around 50 tokens per second may need about seven minutes. With the same model capability, end-to-end time can differ by nearly eight times.
A more representative example is real-time shader authoring. After entering “tunnel through black hole,” the model writes GLSL shader code in real time while a browser pane on the right renders it live. Within about 70 seconds, the model iterates through multiple natural-language instructions and produces a black-hole scene with an accretion disk, gravitational lensing, and particle effects. After each instruction, the code updates quickly and the visual result appears almost immediately. Shaders are a task with high code density and strong dependence on visual feedback. This shows that high-speed inference changes not only how long we wait, but also the shape of human-machine collaboration.
So how do we achieve “flagship capability + extreme speed”? The answer is to separate the problem into two categories and optimize them independently:
- Decode speed, which determines how quickly tokens are generated, is mainly constrained by memory bandwidth.
- Time to first token, which determines how soon generation starts, depends on queuing, prefill, and networking.
II. First, Clarify Which Metrics Actually Determine Speed
Many performance discussions mix different metrics together. To evaluate inference services objectively, we need to break down the time spent on a request:
End-to-end time = queueing + network transfer + prefill + decode + network transfer↓ ↓TTFT Output stage (TPOT × output tokens)
1. TTFT (Time To First Token)
The time from sending the request to receiving the first output token. It has the strongest impact on whether the model feels stuck. A commonly accepted threshold is 200 milliseconds for interactive experience: below that, users feel the system responds instantly; above one second, users notice a clear waiting break.
2. TPOT (Time Per Output Token)
Also called ITL (Inter-Token Latency), the average time required to generate one token during the output stage, measured in milliseconds. Its reciprocal is the generation speed often referenced in reading experience, measured in tokens per second:
- 1000 ms ÷ 50 = 20 ms/token, or about 50 tokens/s;
- 1000 ms ÷ 400 = 2.5 ms/token, or about 400 tokens/s.
For reference:
- Silent human reading is typically around 5–8 characters per second; LLM generation speed is usually higher than reading speed.
- A 2.5 ms token interval is shorter than one frame of a 60 Hz display, which is about 16.7 ms. Multiple tokens often arrive within the same frame, making text appear to pour onto the screen rather than trickle in one character at a time.
3. Output Throughput
The total number of tokens produced by all concurrent requests per unit time. This is a resource-efficiency metric most important to service providers, but users only perceive the speed of their own request. The two should be viewed separately.
4. End-to-End Latency
The total time the user actually waits. In interactive programming, this matters more than tok/s alone. For example, generating a page of around 15,000 tokens:
- 400 tokens/s: theoretical generation time of about 37.5 seconds;
- 50 tokens/s: about 300 seconds;
- Eight rounds of revision: about five minutes for the former, about forty minutes for the latter.
Adding roughly one second of TTFT and several tool-call rounds makes the gap even more obvious.
III. Why Speed and Capability Appear to Be Opposed
1. Decoding Is Reading Weights, Not Doing Math
Mainstream autoregressive LLMs generate tokens one by one. Each step must:
- Read all model weights;
- Read the KV Cache of the current sequence, which stores cached historical key-value vectors;
- Repeatedly read and write memory across matrix multiplication, attention, normalization, sampling, vocabulary projection, and other operations.
Under single-request or low-concurrency conditions, the computation in the decode stage is much smaller than the memory read/write volume, so the bottleneck is memory bandwidth. This means output speed is roughly proportional to memory bandwidth and has little to do with compute power.
For a 30-billion-parameter model in FP16:
- Weights occupy about 60 GB. With 3 TB/s memory bandwidth, the theoretical upper bound is about 50 tokens per second.
- If weights are compressed to 4 bits and occupy about 15 GB, the same bandwidth can raise the theoretical upper bound to about 200 tokens per second.
In practice, KV Cache reads, inference framework overhead, sampling, and other factors reduce the real number below the theoretical value. This analysis uses simplified assumptions and is only meant to show the order of magnitude.
2. The Cost of Traditional High-Speed Paths
Since speed is mainly constrained by how much data must be read, the most direct way to speed things up is to read less—for example, by switching to a smaller model. But reducing parameter scale often comes with:
- Weaker cross-file reasoning over complex code;
- Lower accuracy in retrieving key information from long contexts;
- Worse instruction following under strict format constraints.
This creates a loop: “fast means small, small means weak.”
3. The Right Direction: Change the Inference Stack, Not Model Capability
The new approach keeps the flagship model itself and changes how it is run faster:
- Produce multiple tokens per forward pass to reduce weight reads;
- Store weights at lower bit widths to reduce data read each time;
- Use more efficient scheduling to reduce interference between requests;
- Make the first-token stage almost negligible as a bottleneck;
- Cache and reuse repeated computation results.
Most of these optimizations do not change the probability distribution computed by the model, so they provide lossless acceleration. Flagship capability and extreme speed can coexist.
IV. Core Acceleration Techniques
Technique 1: Speculative Decoding—One Forward Pass, Multiple Tokens
The principle is draft-then-verify:
- A lightweight draft mechanism guesses several tokens in a row;
- The target model computes those positions in parallel with one forward pass;
- Each token is checked one by one: matching tokens are accepted; the first mismatch is corrected and decoding stops;
- The computation for skipped positions is not wasted—it happens in a forward pass the model would have executed anyway.
Therefore, one forward pass outputs at least one token, and can output several times more when speculation succeeds. The key is that verification uses the target model’s own probability distribution for acceptance/rejection sampling, so the output distribution is strictly identical to token-by-token decoding and there is no loss of capability. This is somewhat like CPU branch prediction: when prediction succeeds, the pipeline does not stall; when it fails, only a small number of cycles are lost.
There are three main sources of draft tokens:
| Draft source | Principle | Characteristics | Typical scenarios |
|---|---|---|---|
| Independent small model | A small model from the same family generates drafts, which are then verified by the larger model | Easy to understand; draft quality depends on distribution alignment between the small and large models | General conversation, long-form generation |
| Self-draft / multi-head prediction | Lightweight prediction heads, extra decoding layers, or reused shallow features are attached to the model | No second model to maintain; drafts are naturally from the same source as the target model | End-to-end low latency |
| Model-free draft | Retrieve fragments already present in the prompt or context, such as previously written import statements or repeated CSS rules | Almost zero extra cost; high benefit when it hits | Code completion, refactoring, long contexts |
The speed benefit can be understood with a simplified model:
Let α be the acceptance rate per token, γ be the number of draft steps, and c be the cost of each draft step relative to one target-model step:Expected tokens produced per verification ≈ (1 - α^(γ+1)) / (1 - α)Speedup ≈ expected output tokens / (1 + γ·c)
The formula yields three tuning rules:
- Draft quality, represented by α, is the first determining factor;
- There is an optimal number of draft steps: too few wastes verification opportunities, while too many increase draft cost and lower acceptance in later steps;
- The draft must be lightweight enough, meaning c must be small enough.
In code generation, token distributions are relatively predictable, so acceptance rates are often high and acceleration benefits are usually more obvious.
Technique 2: Low-Bit Quantization—Making Weights Lighter
Quantization represents weights with fewer bits. Some schemes also quantize activations and KV Cache. Common quantization approaches in LLM inference include:
| Scheme | Core idea | Main benefit | Caveats |
|---|---|---|---|
| Weight-only quantization, such as W8A16 or W4A16 | Weights are stored at low bit width and converted back to higher precision during computation | Significantly reduces weight reads and is relatively simple to implement | Dequantization adds a small amount of computation |
| Weight and activation quantization, such as W8A8 or W4A8 | Both weights and activations are quantized to lower precision, taking advantage of GPU low-precision tensor cores | Decode and prefill both benefit | Sensitive to outliers; requires calibration or mixed precision |
| Low-precision floating point, such as FP8 | 8-bit floating point balances dynamic range and precision | Nearly lossless when natively supported by hardware | Depends on hardware and inference framework support |
| KV Cache quantization | Cached keys and values are stored at lower bit widths | Reduces memory footprint and bandwidth usage for long contexts and high concurrency | May introduce slight accuracy loss under long contexts |
Quantization can be post-training quantization, or PTQ, or quantization-aware training, or QAT:
- PTQ requires only a small calibration dataset and is faster to deploy;
- QAT requires retraining, preserves accuracy better, and costs more.
Engineering experience:
- Per-channel or group-wise quantization and protecting outliers are common ways to control error;
- The inference framework must dequantize low-bit weights to higher precision when loading them; this overhead is usually overlapped with computation;
- Quantization is not lossless. Before launch, run regression evaluations on tasks such as code generation and mathematical reasoning to confirm quality is acceptable.
It is also important to distinguish between “low-precision storage with dequantization during computation” and “end-to-end low-precision computation.” Their error sources and acceleration sources are different.
Technique 3: Continuous Batching and Chunked Prefill—Improving GPU Utilization Without Hurting Interactivity
During decode of a single request, much GPU compute capacity is idle. Batching multiple requests can improve throughput, but traditional static batching must wait for the entire batch to finish before returning results, so short requests can be held back by long ones.
Modern inference services use continuous batching with iteration-level scheduling: each iteration can remove completed requests and add new ones. Its scheduling logic can be abstracted as:
while has_pending_requests():batch = scheduler.form_batch()# A batch can mix two types of work:# - Waiting requests: execute one prefill step# - Requests already generating output: execute one decode steplogits = model.forward(batch)for req, out in zip(batch, logits):token = sampler.sample(out, req.sampling_params)if token.is_eos or req.finished_len >= req.max_tokens:scheduler.finish(req) # Immediately release resources and return resultelse:req.append(token)
A further issue is that prefill for a long prompt is a compute-intensive operation that can take a long time. If it is mixed into the same batch as decoding, requests already streaming output may stall for a moment, causing token intervals to fluctuate.
Chunked prefill splits a long prompt into small chunks and schedules them together with decode requests:
- GPU utilization is maintained;
- Output streams become smoother;
- Time to first token becomes more controllable.
If a platform provides scheduling parameters such as priority or guaranteed tiers, interactive scenarios should declare them explicitly.
Technique 4: Prefill/Decode Disaggregation and Parallelism—Assigning Resources by Stage
Prefill and decode have opposite resource characteristics:
| Stage | Resource characteristics | Optimization goal |
|---|---|---|
| Prefill | Compute-intensive, high parallelism for large matrix multiplications | Compute, throughput |
| Decode | Memory-access-intensive, low parallelism | Memory bandwidth, per-step latency |
Separating the two stages into different instance pools allows independent choices for hardware mix, parallelism strategy, and batching parameters:
- Prefill instances are configured for compute needs and process prompts from many requests;
- Decode instances are configured for bandwidth and memory needs to maintain stable token intervals;
- KV Cache is transferred between stages over high-speed networks.
Model parallelism also directly affects latency:
- Tensor parallelism splits computation within a layer across multiple GPUs and requires frequent inter-GPU communication, making it suitable within nodes with high-bandwidth interconnect;
- Pipeline parallelism splits by layer and introduces pipeline bubbles;
- Expert parallelism for mixture-of-experts, or MoE, models adds routing communication.
Engineering implementations of “flagship model + low latency” usually require joint tuning of parallelism, topology, and batching parameters rather than changing a single setting.
Technique 5: Prefix Cache Reuse—Skipping Repeated Computation
System prompts, tool definitions, codebase context, and few-shot examples are often highly repetitive across turns or tasks. Prefix caching hashes and indexes each request’s prefix; when the prefix matches completely, cached KV can be reused directly and that portion of prefill is skipped.
The impact on TTFT is direct:
- On a cache miss, prefill time grows approximately linearly with prompt length;
- On a full hit, this portion is close to zero, leaving only processing for newly added content.
In the loop of writing code, running it, viewing the result, and revising, context changes only slightly each round, so prefix reuse is especially valuable.
Callers can actively improve hit rates:
| Practice | Explanation |
|---|---|
| Put stable content first | Place system prompts, tool definitions, project background, and coding rules at the beginning and keep them unchanged as much as possible |
| Put dynamic content last | Put timestamps, session IDs, currently selected code, and latest user input at the end of the prompt |
| Use stable serialization | Serialize tool definitions and messages in a fixed order to avoid field-order drift |
| Reuse session identifiers | If the platform supports session-level caching, reuse it within the same session |
Caveats:
- Similar length is not a hit; only token-for-token prefix identity counts as a hit;
- Caches have lifecycles; cache hits are an optimization, not a correctness guarantee, so business logic must not depend on their presence;
- When using caching, confirm that the platform provides request-level data isolation.
Technique 6: Kernel and Sampling Optimizations—Removing Invisible Latency
A decode step contains many small operations: normalization, positional encoding, attention, residual connections, vocabulary projection, and sampling. Each may take less than 1 ms, but at 400 tokens per second the budget per step is only 2.5 ms. Common engineering techniques in inference frameworks include:
- Operator fusion: reducing memory reads and writes and kernel launch overhead;
- Efficient attention kernels: lowering memory-access cost for long-sequence attention;
- Graph capture: reducing CPU-side scheduling overhead;
- Keeping sampling on the GPU: performing temperature, top-p, top-k, repetition penalties, and similar parameters on the GPU to avoid device-to-host copies at every step.
These optimizations do not change model outputs and are effectively “free” speedups.
V. From the Caller’s Side: Getting the Most Out of High-Speed APIs
Server-side speed is only an upper bound. The application must also avoid slowing the experience artificially.
1. Stream Output End to End
Non-streaming calls return only after the full result is generated. Even if the model is fast, users may perceive a long period of no response. Streaming turns “waiting for the full result” into “continuously seeing progress.”
A minimal example of metric collection:
import timestart = time.perf_counter()first_token_at = Nonechunks = 0for delta in stream_chat(messages=messages, temperature=0.2):# delta is the incremental textif first_token_at is None:first_token_at = time.perf_counter()chunks += 1render(delta) # Incrementally render to the UIttft = first_token_at - start
2. Incremental Rendering on the Front End
High-frequency small updates should not trigger a full-page reflow. Common practices:
- Merge UI refreshes at about 50 ms intervals;
- Incrementally highlight code blocks;
- Pre-generate preview content so writing and viewing happen in parallel.
3. Timeouts, Retries, and Cancellation
- Set separate timeout thresholds for TTFT and total generation to avoid requests hanging indefinitely;
- Use exponential backoff with jitter for retryable errors;
- When a user cancels or closes the page, actively terminate generation, stop billing, and avoid wasting quota on ineffective requests.
4. Concurrency and Quotas
High-speed APIs may trigger rate limits during load testing or high-concurrency batch scenarios. Recommended:
- Add client-side rate limiting and queuing;
- Use a sensible backoff strategy;
- Route asynchronous-capable tasks to asynchronous task modes.
5. Prompt Engineering
- Control output size: specify a line or token limit to avoid redundant content;
- Prefer structured output using the platform’s JSON mode or function-calling capabilities to reduce format correction costs;
- Express modification requests as clear incremental instructions, such as “change card spacing to 16px,” instead of asking the model to rewrite the whole file.
VI. How to Verify Whether Speed Is Real: Benchmarking Methods
Peak metrics published by vendors usually come from ideal conditions and must be validated against your own workload. Recommended benchmarking dimensions:
| Dimension | Recommendation |
|---|---|
| Metrics | Record TTFT, TPOT, end-to-end latency, and failure rate; distinguish P50, P95, and P99 |
| Load | Cover a typical prompt-length distribution, including long-context and tool-calling scenarios |
| Concurrency | Ramp up from single concurrency to the target concurrency and observe the latency knee |
| Output length | Fix maximum output length and sampling parameters so runs are comparable |
| Streaming | Streaming must be enabled; measure TTFT and TPOT under streaming |
Example output-side metric collection:
# tps: end-to-end average output speed; tpot: average token intervaltps = output_tokens / (end_time - start_time)tpot = (end_time - first_token_at) / max(output_tokens - 1, 1)
Additional benchmarking notes:
- Different output lengths naturally produce incomparable tok/s; compare under the same output budget;
- Peak speed often corresponds to low concurrency; the effective speed under high concurrency is what matters in production;
- Long-context scenarios should be tested separately because both TTFT and KV Cache overhead grow with length;
- If the baseline is a self-hosted or open-source inference stack, keep hardware and batching parameters consistent; otherwise conclusions will not hold.
VII. Selection Guidance: When to Choose High-Speed Flagship Models
High-speed APIs are not optimal for every scenario. Decide based on task characteristics:
| Scenario characteristics | Recommended strategy |
|---|---|
| Interactive programming, live previews, conversational edits | Prefer high-speed flagship models; TTFT and TPOT are core experience metrics |
| Agentic multi-step chains, such as plan, tool call, reason again | Prefer high-speed flagship models; per-step latency accumulates across rounds |
| High concurrency, short outputs, asynchronous possible | Consider lightweight models or asynchronous batch modes for lower cost |
| Offline batch generation, such as document summarization or data cleaning | Focus on throughput and unit price; TTFT is less important |
| Strong reasoning tasks, such as complex algorithms or long chains of reasoning | Prioritize capability; treat speed as a secondary metric |
If a task is latency-sensitive but still needs strong capability, a high-speed flagship model is often a balanced choice today. Before choosing, run a small-scale comparison with real business samples rather than relying only on leaderboard rankings.
VIII. Conclusion: Speed Is an Engineerable Product Capability
Broken down, 400 tokens per second is not the result of a single technique but the accumulation of multiple layers of optimization:
- Speculative decoding reduces weight reads;
- Quantization reduces data read per access;
- Continuous batching and chunked prefill keep GPUs busy and make output smoother;
- Prefill/decode disaggregation and parallelism tuning assign the right resources to each stage;
- Prefix caching removes repeated computation directly;
- Kernel and sampling optimizations eliminate millisecond-level hidden overhead.
For developers, the most direct benefit is a change in interaction form: from “send a request, wait, view the result” to “talk while editing and watch while editing.” The model shifts from a batch-oriented tool to a real-time collaborative pair programmer.
Three things matter most in practice:
- Measure experience with TTFT and TPOT, not a single tok/s number;
- Use streaming output, incremental rendering, and sensible timeout and cancellation policies to deliver speed to users;
- Make selection decisions based on benchmark data rather than marketing numbers.
When inference speed is no longer the bottleneck, product design can revolve around instant feedback. That may be a more important shift than any single metric.
