LLM Inference Acceleration: Key Techniques Behind 400 Tokens per Second

    A practical overview of the technologies that let flagship large language models serve hundreds of tokens per second without giving up capability. The article breaks "speed" into four distinct metrics—time to first token, single-stream output speed, end-to-end latency, and aggregate throughput—then explains how five categories of techniques work together: model architecture (sparse activation/MoE), decoding algorithms (speculative decoding), GPU memory and cache management, server-side scheduling, and low-level kernel optimizations. It also covers why many speedups are lossless, how lower latency reshapes product design, a client- and server-side tuning checklist with sample code, and common pitfalls to avoid.

    Q&A: How Do Flagship Models Balance Speed and Capability?

    Q: What are the key techniques that let a flagship model deliver high throughput and low latency at the same time?

    A: Speed can be broken down; capability cannot. Time to first token, output speed, end-to-end latency, and aggregate throughput are each governed by different parts of the pipeline. By combining techniques such as sparse activation, speculative decoding, cache reuse, continuous batching, and prefill/decode disaggregation, each stage can be optimized on its own—so speed no longer has to come at the expense of capability.

    In the past, “fast” often meant “small.” High-speed serving relied on lightweight models, and gains in speed came with a reduction in capability. Today, some vendors’ flagship models push streaming output to hundreds of tokens per second, keep time to first token under one second, and generate end to end several times faster than conventional serving. Flagship capability and responsiveness are no longer an either/or choice.

    1. First, Be Precise About “Fast”: Four Metrics You Shouldn’t Conflate

    • Time to first token (TTFT): The time from sending a request to receiving the first token. It determines “how soon you see a response.”
    • Single-stream output speed (tok/s): The average token generation rate of a single request during continuous streaming. It determines “how fast it reads.”
    • End-to-end latency: The total time from initiation to completion of generation. It depends on output length and equals “TTFT + generation time”—the time the user actually waits.
    • Aggregate throughput: The total tokens produced while the system serves many requests concurrently. It determines “how much concurrency the system can handle.”

    Once you separate these four metrics, it’s easy to understand “why a flagship model can still feel slow.” Higher-capability models tend to produce longer outputs with more complex reasoning chains, so their end-to-end latency is naturally longer. What “speedup” really targets is TTFT and single-stream speed.

    Here’s a quick comparison. A blink takes roughly 0.3 seconds, and typical reading speed is on the order of a dozen-plus tokens per second (varying for Chinese, code, and English). At a single-stream speed of 400 tokens per second, the model has already produced over a hundred tokens in the time it takes to blink once. When generation is far faster than reading, the user’s experience shifts from “waiting” to “flowing.”

    Metric Optimization levers Main benefit
    TTFT Cache reuse, prefill optimization, prefill/decode disaggregation, load balancing Starts “speaking” faster; interactions stay responsive
    Single-stream speed (tok/s) Sparse activation, speculative decoding, quantization, FlashAttention-style attention implementations Generation flows smoothly
    End-to-end latency Combined TTFT and output speed, output-length control Shortens the full wait; suits non-streaming scenarios
    Aggregate throughput Continuous batching, GPU memory and KV cache management, elastic scaling Stays stable under high concurrency

    2. Where Acceleration Comes From: Five Categories Working Together

    2.1 Model Architecture: Sparse Activation Makes “Big” Also “Fast”

    An autoregressive model runs a full forward pass for every token it generates, so the speed ceiling is set by “how many parameters are computed at each step.” A Mixture-of-Experts (MoE) architecture splits the model into multiple expert networks, and each token passes through only the few experts selected by the router.

    As a rough estimate using the scaling rules common to large models (based on the fitted formula proposed in the Chinchilla research; the numbers below are illustrative only): a dense model with 30B parameters and 100B training tokens sits roughly in the compute-optimal region, with training compute of about 6 × 30B × 100B ≈ 1.8×10²² FLOPs. Switching to a sparse model with 4B total parameters and 3B activated parameters would need roughly 1000B tokens to reach the same training compute; yet for the same prompt, the per-step compute is only about one-tenth of the original. Given comparable capability, single-stream speed has room for a several-fold improvement. This is the structural basis that lets “flagship capability + high speed” coexist.

    That said, sparsification alone does not equal low latency. Communication between experts, whether routing load is balanced, and whether GPU memory bandwidth becomes a bottleneck all directly determine how much of that speed can actually be realized.

    2.2 Decoding Algorithms: Let the Big Model Grade Its Own Drafts

    Autoregressive generation is inherently serial: the Nth token depends on the previous N-1 tokens, and one forward pass typically produces a single token. The idea behind speculative decoding is “draft first, then verify”:

    1. Use a lightweight draft model or extra heads to speculatively guess k candidate tokens in a row;
    2. The target model runs one forward pass over these k+1 positions, computing the next-token distribution at each position in parallel;
    3. Verify each one with an accept/reject rule equivalent to the original decoding strategy: if accepted, advance one step; at the first rejected position, resample from the target model’s own distribution, which guarantees the output distribution is unchanged;
    4. All draft tokens after the rejected position are discarded and filled in by a new round of drafting.

    A single verification pass can advance by up to k+1 tokens, and the average speedup depends on the draft’s acceptance rate. The higher the acceptance rate, the more effective tokens each target-model forward pass “produces.”

    An online draft model consumes extra compute. In practice, engineers often use extra prediction heads built into the model, n-gram/prompt retrieval, or pre-generate draft trees while the service is idle. Whatever the approach, raising the acceptance rate and lowering the drafting cost are the keys to making it work in production.

    2.3 GPU Memory and Cache: Stop Paying for Repeated Computation

    • Paged KV cache management: The key-value cache of historical tokens grows with the context. Managing the cache in fixed-size pages reduces memory fragmentation and increases concurrency capacity; PagedAttention is a representative implementation of this idea.
    • Prefix cache reuse: System prompts, long documents, code repository context, and tool definitions often stay unchanged across many consecutive turns. Caching and reusing their prefill results means later requests barely need to recompute, and TTFT drops significantly.
    • Cache-friendly routing: Session-sticky routing sends requests carrying the same prefix to the same instance, directly improving prefix-cache hit rate.
    • Output-side caching: In scenarios like code completion where prefixes are highly consistent, you can use approximate prefix matching to hit historical results and skip whole segments of generation.
    • Compression and quantization: Quantizing to lower precision (such as INT8 or FP8) reduces the memory and bandwidth overhead of weights and the KV cache. This comes with some loss of precision and needs to be paired with calibration and regression evaluation.

    2.4 Server-Side Scheduling: Keep the GPU From Sitting Idle

    • Continuous batching: Traditional static batching must wait for an entire batch of requests to finish before admitting new ones, so short requests get held back by long ones. Continuous batching dynamically adjusts batch membership at every decoding step: completed requests leave immediately and new requests join at any time, keeping the GPU at high utilization.
    • Chunked prefill: Prefilling a long prompt is a compute-intensive operation. Running it all at once ties up compute resources for a long time and slows down other requests’ token-by-token output. Splitting the prefill into chunks and interleaving them with decoding steps lowers inter-token latency under concurrency.
    • Prefill/decode disaggregation: Prefill is compute-bound, while decode is GPU-memory-bandwidth-bound; the two have entirely different resource needs. Splitting them into separate resource pools lets each scale independently according to its own bottleneck, connected via KV cache transfer. This both improves TTFT and prevents long and short requests from interfering with each other.
    • Request admission and prioritization: Routing based on request length and setting priorities for interactive requests prevents long tasks from monopolizing resources.

    2.5 Low-Level Execution: Squeeze Out Hardware Efficiency

    • IO-aware attention implementations such as FlashAttention reduce GPU memory reads and writes through tiled computation, and have become a common foundational component for high-speed inference.
    • Techniques like operator fusion and computation-graph capture reduce kernel launch and scheduling overhead—especially critical in the decoding phase, where each step computes just one token. Here the GPU is often not short of compute but held back by memory access and small-operator scheduling.
    • In multi-GPU deployments, tensor parallelism and expert parallelism combined with high-speed interconnects spread a single request’s compute and communication overhead.

    3. Speed Goes Up—Why Doesn’t Capability Drop?

    Splitting the speedup techniques into two categories makes this clear:

    • Lossless: Continuous batching, chunked prefill, prefill/decode disaggregation, cache reuse, and scheduling optimizations change only “how it’s computed,” not the model parameters or output distribution.
    • Lossy: Low-precision quantization, plus distillation and pruning done for speed. These techniques can cause capability loss and must be gated by evaluation.

    Speculative decoding also comes in two variants. If the draft comes from an independent small model, resampling at rejected positions happens only on paths that have already diverged from the target distribution, and the final output stays consistent with the target model. If the draft is taken directly from the target model itself, then when a draft token happens to be the maximum-probability token under greedy decoding, it is always accepted; otherwise the target model’s distribution is resampled directly. Both approaches are equivalent to the original decoding strategy. In principle, speculative decoding trades extra compute for latency, not for capability.

    So the real solution for flagship high-speed serving is usually a combination:

    1. Use lossless optimizations to push throughput and single-stream speed high;
    2. Use techniques like sparse activation to reduce per-token compute at the structural level;
    3. Use lossy compression sparingly, and guard the capability floor with an evaluation suite.
    Evaluation dimension Representative methods Focus
    Knowledge General knowledge QA, multi-discipline comprehensive exams Factual accuracy, depth of expertise
    Reasoning Math and logic problems, multi-step reasoning tasks Correctness and stability of the reasoning chain
    Code Code completion, repository-level editing, unit-test-driven evaluation Run rate, first-pass success rate, multi-file consistency
    Interaction Multi-turn dialogue, tool calling, long-context QA Instruction following, context consistency
    Engineering Load-test latency percentiles, token cost accounting P95/P99 latency, scaling curves, cost per task

    A practical rule of thumb: pursue losslessness at the serving layer, and require evaluation for any model-layer optimization. Public leaderboards measure the capability ceiling, but what production really has to defend is the latency percentiles and the cost per task.

    4. Once Latency Crosses the Threshold, How Do Product Forms Change?

    Speed matters beyond the numbers. When the wait shrinks to negligible, interactions that were previously impossible become possible.

    From “wait and check” to “see it as it’s built.” When generating shaders, configuring visual components, or writing frontend code, waiting a dozen-plus seconds to see the effect of a one-line prompt tweak quickly breaks a person’s flow. When output speed holds steady at hundreds of tokens per second and each change shows results within a few seconds, the model feels like it shares the same canvas as the person—editing and viewing as it goes.

    From sketch to usable prototype. Hand the model a hand-drawn wireframe and, within tens of seconds, get a complete page with structure, styling, and interaction logic. Design review can then start at the prototype stage instead of waiting until implementation is done.

    From “one assistant” to “a parallel team.” When running many tasks in parallel, a single fast request isn’t enough—system-level throughput is the ceiling. When dozens of concurrent requests reach an aggregate speed of over a thousand tokens per second, batch code review, parallel generation of multiple options, and multi-role simulation become practically useful, and the bottleneck shifts from “generation is too slow” to “task orchestration.”

    Takeaways for product design:

    • Include TTFT as an acceptance metric and render the first-screen skeleton first;
    • Support incremental updates and partial refreshes instead of redrawing the whole block;
    • Provide progress, pause, and resume capabilities for long tasks;
    • Keep dual channels of “low-latency mode” and “deep-reasoning mode”: fast responses for everyday interaction, and a stronger reasoning mode for complex problems.

    5. Putting It into Practice: An Integration and Tuning Checklist

    5.1 Client: Make “Fast” Actually Felt by the User

    No matter how fast the server is, blocking calls will consume all the gains. The core practices are streaming reception, incremental rendering, and setting a separate budget for each type of delay:

    1. # Illustrative code: pseudocode, not any vendor's real SDK, shown only to demonstrate the integration approach
    2. from some_sdk import Client # placeholder name; replace with the actual SDK import
    3. import time
    4. client = Client(base_url=BASE_URL, api_key=API_KEY)
    5. FIRST_TOKEN_BUDGET = 1.0 # time-to-first-token budget (seconds)
    6. STALL_BUDGET = 3.0 # stall threshold: no output for this long (seconds)
    7. HARD_TIMEOUT = 120.0 # hard timeout for a single request (seconds)
    8. def stream_completion(prompt: str, on_delta):
    9. started = time.monotonic()
    10. stream = client.chat.completions.create(
    11. model=MODEL, # flagship high-speed model
    12. messages=[{"role": "user", "content": prompt}],
    13. stream=True, # streaming must be enabled
    14. temperature=0.2,
    15. max_tokens=8192,
    16. )
    17. first, last = None, started
    18. try:
    19. for chunk in stream:
    20. now = time.monotonic()
    21. if now - started > HARD_TIMEOUT:
    22. raise TimeoutError("hard timeout")
    23. piece = chunk.choices[0].delta.content or ""
    24. if not piece:
    25. continue
    26. first = first or now
    27. if now - last > STALL_BUDGET:
    28. raise TimeoutError("stream stalled")
    29. last = now
    30. on_delta(piece) # incremental write to output area / incremental rendering
    31. finally:
    32. if first:
    33. metrics.observe(
    34. ttft=first - started, # report time to first token
    35. inter_token=(last - first) / max(1, est_tokens),
    36. )
    37. return "".join(...) # assemble the complete text
    • Tiered timeouts: Set and report separate thresholds for first token, token interval, and total duration, so you can tell whether it’s “slow first token” or “stall midway.”
    • Retry and fallback: For recoverable errors like connection drops and rate limiting, use exponential backoff with random jitter and set a retry cap. Write operations that need idempotency should carry an idempotency key, and long tasks can be persisted and resumed.
    • Incremental rendering: For structured output (like page code), render the skeleton first and fill in content, using partial replacement instead of redrawing the whole page.
    • Session stickiness: Route the same session to the same inference instance as much as possible, so the prefix cache keeps hitting.
    • Context governance: Relative to the model’s context limit, KV cache capacity and the maximum length processable per request are the real constraints. It’s advisable to periodically compress or summarize historical messages to avoid context bloat slowing down every response.

    5.2 Server: Make Latency a Sustainable Metric

    • Elastic scaling: Replace “scale on request count” with “scale on queue length and token rate.” Generative workloads vary greatly in duration, so looking at request count alone can easily misjudge.
    • Capacity planning: Estimate compute needs as “peak concurrency × average output length ÷ target tok/s,” then tune parameters against the dual goals of TTFT and tok/s.
    • Segmented load testing: Construct two traffic types separately: “short prompt + long output” and “long prompt + short output.” The former tests decoding efficiency; the latter tests prefill and cache hit rate.
    • Observability: Monitor the percentiles of TTFT, inter-token latency, output speed, aggregate throughput, queueing time, cache hit rate, and error rate.
    • Release strategy: Roll out a new version first with shadow traffic or a percentage-based canary, compare latency and quality metrics before ramping up, and keep a one-click rollback.

    5.3 Cloud Architecture: Organize Infrastructure Around Latency Goals

    If you don’t want to run your own inference cluster, you can use large-model inference services on a cloud platform directly. Platforms such as Baidu AI Cloud offer corresponding capabilities in model serving, container platforms, and monitoring and alerting, for example. Using a cloud platform lets you focus your attention on the business itself:

    • Access layer: A gateway handles unified authentication, rate limiting, routing, and session stickiness.
    • Inference layer: Deploy model services on a container platform, with elastic scaling driven by queue length and throughput targets.
    • Cache layer: Centralize management of high-frequency results and prefix caches, optionally with a distributed cache service.
    • Async layer: Route long tasks like batch generation and code review through message queues so they don’t slow down interactive requests.
    • Observability layer: Connect log services and monitoring/alerting to a latency dashboard and alert rules.
    • Configuration layer: Use a configuration center to manage multi-tier model routing—for example, simple requests to a lightweight model and complex requests to a flagship high-speed model.

    6. Common Pitfalls and How to Avoid Them

    1. Watching only tok/s and ignoring TTFT: In interactive scenarios, the first token is what users perceive first. If the first token takes more than a second or two, no matter how high the tok/s, it won’t hold their attention.
    2. Directly comparing speeds across different tokenizers: The same character can split into multiple tokens under different tokenization schemes. “How many tokens per second” is only comparable under the same tokenizer; for cross-model comparisons, focus more on end-to-end latency.
    3. Hoping for “fast” from a single optimization: Speedup is the product of architecture, decoding, caching, scheduling, and low-level execution. If any one link lags, the whole is constrained.
    4. Overlooking the hidden cost of long context: The longer the context, the slower the prefill and the larger the KV cache footprint. Caching and summarization mechanisms should be standard, not optional.
    5. Ignoring recoverable errors: Under high concurrency, rate limiting and transient timeouts are hard to avoid. Lacking backoff/retry and a fallback channel can amplify an occasional failure into a widespread outage.

    7. Conclusion

    Flagship-class high-speed inference at hundreds of tokens per second is essentially a systematic engineering rebuild:

    • Model side: Use sparse activation to reduce per-token compute;
    • Decoding side: Use speculative decoding to turn serial waiting into parallel verification;
    • Serving side: Use cache reuse and continuous batching to squeeze out hardware efficiency;
    • Integration side: Use streaming, tiered timeouts, and observability to defend the experience.

    For developers, what really matters isn’t mastering a single number, but breaking latency into measurable, optimizable, and sustainable engineering metrics. Only when responses are faster than the psychological threshold of “waiting” will product forms like real-time co-creation, sketch-to-prototype, and multi-task parallelism truly land.