LLM Inference Speed Optimization: How to Approach 1,000 Tokens per Second
This article breaks down what actually determines large language model inference speed and why "1,000 tokens per second" is a matter of engineering rather than magic. You will learn the difference between total and activated parameters, why inference splits into prefill and decode phases, how memory bandwidth and KV Cache limit generation, and which levers matter most: sparse model structures like MoE, attention optimizations, inference engine techniques (kernel fusion, paged KV Cache, prefix caching, speculative decoding), quantization, parallelism, and disaggregated deployment. It also covers how to measure speed with TTFT, TPOT, ITL, and TPS, includes a Python script for benchmarking streaming responses, and offers practical guidance on model selection, throughput-versus-latency trade-offs, and common misconceptions—so you can build a verifiable performance framework for your own workloads.
1. Introduction: The LLM Race Is Shifting from “Capability” to “Capability × Speed”
Over the past two years, both the parameter counts and the generative capabilities of large language models have grown rapidly. For many teams, the real bottleneck is no longer “can the model do it,” but “can it do it fast enough.” Code completion needs millisecond-level first-token feedback, agentic tasks need many tool calls to converge quickly, and online services need stable latency under peak traffic. All of these scenarios push inference speed to the center of engineering optimization.
Not long ago, “trillion parameters” was often seen as a badge of model capability and was casually equated with “slow.” Yet over the past two years, the industry has indeed seen ultra-large models running inference at hundreds or even over a thousand tokens per second. Such speeds are not the product of any single “black box” trick; they are the combined result of model architecture, the inference engine, quantization, parallelism strategy, and service deployment.
To be more precise at the headline level, it is better to frame this as “how to approach thousand-token-class speed” rather than broadly promising that “a trillion-parameter model can hit thousands of tokens per second.” Reaching a thousand tokens per second usually depends on a stack of conditions: sparse model architectures, high-bandwidth memory, low-precision quantization, batching, speculative decoding, prefix caching, parallel sharding, and disaggregated deployment. Without these conditions, model scale alone cannot deliver a stable thousand-token experience.
The goal of this article is not to promote a single number, but to help developers build a verifiable framework for analyzing inference speed: what determines speed, which techniques are common, how to benchmark, and how to make trade-offs for your own business.
2. Parameter Count vs. Generation Speed: What Really Matters Is “Activated Parameters”
Before discussing inference speed, we first need to understand what a “parameter” actually is.
A useful analogy is the human brain: it has roughly 86 billion neurons connected by synapses. Learning new knowledge and forming new habits often corresponds to changes in the strength of synaptic connections. The parameters in an artificial neural network are similar to these “synapses”—each parameter stores a numerical value, and by continuously adjusting these values during training, the model gradually learns language patterns, world knowledge, and task skills.
More parameters mean the model can store richer knowledge, represent more complex relationships, and make finer distinctions:
- A small model with a few billion parameters is like a student with a solid foundation—capable of common Q&A, but likely to struggle with complex reasoning;
- A large model with a trillion parameters is more like a multi-domain expert—broad coverage and more nuanced expression.
But “more parameters” does not necessarily mean “slower at every step.” What truly determines generation speed is not the total parameter count, but the number of parameters that actually participate in computation when generating each token. Sparse architectures, especially Mixture-of-Experts (MoE) models, are the key idea for decoupling “parameter scale” from “compute.”
If you remember only one sentence: total parameter count determines knowledge capacity; activated parameter count determines per-step compute cost.
3. Why Inference Is Slow: The Prefill and Decode Phases
LLM inference is typically divided into two phases: prefill and decode. Their performance bottlenecks differ, and so do their optimization strategies.
3.1 Prefill: Determines How Long the “First Token” Takes
In the prefill phase, the model processes the entire user prompt in parallel. This phase is compute-intensive, and its bottleneck is primarily compute. The wait between the user pressing Enter and the first character appearing on screen is typically called time-to-first-token latency, or TTFT.
The longer the prompt, the slower the prefill. System prompts, long documents, and code repository context can all significantly increase TTFT.
3.2 Decode: Determines How Fast the “Output” Is
In the decode phase, the model generates new tokens one at a time. Each generated token requires reading the model weights and the existing KV Cache. The KV Cache stores the key and value vectors produced during attention computation for every previous token, avoiding recomputation of historical content.
The per-step computation in the decode phase is relatively small, so the bottleneck is primarily memory bandwidth—that is, how much data can be moved through memory per second.
3.3 Estimating Single-Request Decode Speed
We can estimate the upper bound of single-request decode speed with a simplified model:
Single-request decode speed upper bound ≈ effective memory bandwidth ÷ (weight bytes read per token + KV Cache bytes read per token)
Assume a theoretical memory bandwidth of about 8 TB/s, and ignore the impact of the KV Cache for now:
| Model size & precision | Estimated weight size | Upper bound from weight reads only |
|---|---|---|
| 7B parameters, FP16 | ~14 GB | ~550 tokens/s |
| 700B parameters, FP8 | ~0.7 TB | ~11 tokens/s |
A clarification is important here: going from a 7B-parameter model in FP16 to a 700B-parameter model in FP8 increases the parameter count by about 100x, but because FP8 weight bytes are about half of FP16, the total weight bytes grow from roughly 14 GB to about 0.7 TB—about a 50x increase. So in the simplified estimate above, the single-request decode speed upper bound drops by about 50x.
If precision stays constant while the parameter count grows 100x, the weight reads grow by about 100x, and the single-request decode upper bound also drops by nearly 100x. Therefore, we must distinguish three concepts: parameter count, weight precision, and actual bytes read. We cannot simply equate “more parameters” with “definitely 100x slower.”
Real systems are also affected by operator efficiency, memory occupancy, context length, batch size, quantization strategy, and hardware bandwidth utilization. This estimate only helps illustrate one thing: the decode phase is very sensitive to memory bandwidth.
3.4 Why KV Cache Becomes the Main Bottleneck Under Long Contexts
The size of the KV Cache can be roughly estimated as:
KV Cache bytes per token ≈ 2 × number of layers × number of KV heads × head dimension × bytes per data type
For example, a model with 70 layers, 8 KV heads, 128 dimensions per head, using FP16, requires about this many bytes per generated token:
2 × 70 × 8 × 128 × 2 ≈ 0.27 MB
If the context reaches 100,000 tokens, the KV Cache alone can approach 27 GB. In long-context scenarios, the volume of KV Cache reads can even exceed the model weights themselves, making it a significant bottleneck in the decode phase.
This is also why inference speed tends to drop as context grows, and why prefix caching, paged KV Cache, KV Cache quantization, and context compression are so important in long-text scenarios.
3.5 Why Batching Improves Throughput
Within the same decode step, every request in a batch uses the same copy of the model weights. If 32 requests are processed simultaneously, reading the weights once can generate one token for each of the 32 requests. As a result:
- Weight reads are amortized, so system throughput increases significantly;
- But each request’s KV Cache still needs to be read separately, and this overhead is hard to amortize under long contexts.
This explains the common tension between high throughput and low latency: increasing the batch size can raise system throughput, but the latency of an individual request may actually get worse. Therefore, interactive services care more about low latency, while offline batch processing cares more about high throughput.
4. Speed Techniques: From Model Architecture to System Deployment
Improving LLM inference speed usually involves four layers: model architecture, inference engine, quantization, and parallel & service deployment.
4.1 Model Architecture: Making Each Token Compute a Little Less
Mixture-of-Experts (MoE) is one of the most important ideas for speeding up ultra-large models. The model contains multiple expert sub-networks, and each token is routed to only a few experts for computation. This means the total parameter count can be very large—yielding higher knowledge capacity—while only a small fraction of the parameters are actually activated per token, keeping the compute within a reasonable range.
This is one of the key reasons why “large total parameter count” and “still-decent decode speed” can coexist.
Attention structure optimization is equally important. Standard multi-head attention requires caching the key-value pairs of all historical tokens, so the KV Cache grows larger as the context lengthens. Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) let multiple query heads share key-value heads, significantly compressing the size of the KV Cache.
In addition, model distillation is a common technique. It uses a large model to train a smaller one so that the smaller model approaches the large model’s results on common tasks, then serves online traffic at lower cost.
4.2 Inference Engine: Moving Less Data and Scheduling More Smoothly at Every Step
The inference engine has a large impact on decode speed. Common optimizations include:
- Kernel fusion and specialized kernels: reducing memory reads/writes and kernel-launch overhead;
- Graph capture and replay: capturing the fixed decode computation graph ahead of time to reduce per-operator scheduling cost;
- Paged KV Cache: borrowing the idea of OS paging to allocate memory in blocks, reducing fragmentation;
- Prefix caching: reusing the KV Cache for identical prefixes such as system prompts, long documents, and code context to lower TTFT;
- Speculative decoding: using a draft model or extra prediction heads to guess several tokens in a row, then having the large model verify them in parallel, improving decode speed when acceptance rates are high.
The key to speculative decoding is the verification mechanism. As long as the large model verifies multiple candidate positions in parallel, it can increase the number of tokens produced per step while typically keeping output quality close to the original.
4.3 Quantization: Trading Lower Precision for Higher Effective Bandwidth
Quantization compresses weights from FP16 to FP8, INT4, or even lower precision. Once weight bytes shrink, the bandwidth-bound decode phase effectively gains usable bandwidth.
However, quantization can introduce precision loss. In practice, its effect needs to be validated against a business evaluation set, and different strategies are often applied to sensitive layers, output layers, or the prefill phase.
4.4 Parallelism Strategy: Spreading a Large Model Across Multiple GPUs
Common parallelism methods include:
- Tensor parallelism: splitting each layer’s weight matrices across multiple GPUs, where the cards compute and then merge via collective communication;
- Expert parallelism: distributing different experts across different cards, which naturally fits MoE structures;
- Pipeline parallelism: splitting the model into segments by layer, with data flowing sequentially through the devices.
The higher the degree of parallelism, the more pronounced the communication overhead, and the gains gradually diminish. Engineering decisions must weigh inter-card interconnect bandwidth, model size, batch size, and concurrency.
4.5 Disaggregated Deployment: Letting Prefill and Decode Each Do Their Own Job
Prefill is compute-bound and decode is bandwidth-bound, so putting them in the same resource pool tends to cause interference. For example, prefilling a long prompt may block the decoding of other requests, causing stuttering output.
Disaggregated deployment—also called PD separation—places prefill and decode in different resource pools:
- The prefill pool emphasizes compute;
- The decode pool emphasizes memory capacity and bandwidth;
- The two transfer the KV Cache between them over a high-speed network.
This allows independent scaling, independent tuning of batching strategies, and reduced cross-phase interference. On the cloud, such services are typically deployed in a containerized way and paired with logging, monitoring, and alerting to continuously track metrics such as TTFT, TPOT, P95, and P99.
5. How to Evaluate Inference Speed: Metrics, Code, and a Benchmark Checklist
Evaluating an inference service cannot rely on “how many seconds it took in total” alone. Common metrics are as follows:
| Metric | Meaning | Affects experience | Main bottleneck |
|---|---|---|---|
| TTFT | Time from sending a request to receiving the first token | Perceived waiting | Prefill compute, queuing |
| TPOT | Average time per token during generation | Output smoothness | Memory bandwidth |
| ITL | Time interval between two adjacent tokens | Stuttering | Scheduling and batching |
| TPS | Tokens generated per second | Reading speed | Memory bandwidth |
| System throughput | Total tokens produced by all requests per unit time | Service capacity and cost | Batch size, parallelism efficiency |
Among these, TPOT and TPS are closely related. If TPOT is measured in milliseconds, then TPS is roughly:
TPS ≈ 1000 ÷ TPOT
But real experience also depends on P95 and P99 tail latency. A good average does not mean a good long-tail experience. The industry often uses effective throughput—the number of requests a system can actually complete while meeting latency targets—to measure service quality.
The following is illustrative code that assumes the server provides SSE streaming responses and uses the model’s own tokenizer to count tokens. Compared with estimating tokens by dividing UTF-8 byte length by 4, this approach is more reliable in mixed Chinese-English scenarios.
import jsonimport time# Dependencies: pip install transformers requestsfrom transformers import AutoTokenizerTOKENIZER = AutoTokenizer.from_pretrained("your-model-name", trust_remote_code=True)def post_stream(payload, url="http://localhost:8080/v1/completions"):import requestswith requests.post(url, json=payload, stream=True) as resp:resp.raise_for_status()for raw in resp.iter_lines():if not raw:continueline = raw.decode("utf-8") if isinstance(raw, bytes) else rawif not line.startswith("data: "):continuedata = line[6:]if data == "[DONE]":breakobj = json.loads(data)choices = obj.get("choices", [{}])if not choices:continuechoice = choices[0]# Support both completion text and chat delta response formatstext = choice.get("text", "")if not text and isinstance(choice.get("delta"), dict):text = choice["delta"].get("content", "")if text:yield textdef measure_speed(payload, max_tokens=512):payload = dict(payload)payload["max_tokens"] = max_tokenspayload["stream"] = Truestart = time.perf_counter()ttft = Noneoutput_text = ""for chunk in post_stream(payload):now = time.perf_counter()if ttft is None:ttft = now - startoutput_text += chunk# On the business side, you can render output hereend = time.perf_counter()# Use the tokenizer to count tokens, avoiding the Chinese estimation bias from UTF-8 bytes divided by 4tokens = len(TOKENIZER.encode(output_text, add_special_tokens=False))# Defensive handling: skip TPOT when there is no output, no first token, or abnormal timingif ttft is None or end <= ttft or tokens <= 0:return {"ttft_ms": None if ttft is None else round(ttft * 1000, 2),"tpot_ms": None,"tps": None,"tokens": tokens,}tpot_ms = (end - ttft) / tokens * 1000tps = tokens / (end - ttft)return {"ttft_ms": round(ttft * 1000, 2),"tpot_ms": round(tpot_ms, 2),"tps": round(tps, 2),"tokens": tokens,}payload_example = {"prompt": "Summarize the key points of speeding up LLM inference.","max_tokens": 512,"stream": True,"temperature": 0.2,"top_p": 0.9,}
To make benchmark results reproducible and comparable, you should record at least the following during load testing:
- GPU model, memory capacity, and inter-card interconnect bandwidth;
- Inference engine name and version;
- Model parameter count, activated parameter count, precision, and quantization method;
- Batch size, concurrency, maximum input length, and maximum output length;
- The distribution of input and output lengths, ideally close to real business traffic;
- TTFT, TPOT, ITL, TPS, and system throughput;
- P50, P95, and P99 tail latency;
- Quality metrics such as accuracy, human ratings, or automated evaluation scores.
Without these details, simply saying “X tokens per second” makes it hard to judge whether the result is real and whether it transfers to your business.
6. What Does a Thousand Tokens per Second Actually Mean?
Translating 1,000 tokens per second into a reading experience makes it more intuitive.
A token is the basic unit the model uses to process text, and it does not strictly correspond to one Chinese character or one word. In English, a rough rule of thumb is that 1 token is about 4 characters or 0.75 words; in Chinese, a token may correspond to one to a few characters, depending on the tokenizer. Conservatively, 1,000 tokens per second is roughly equivalent to outputting five or six hundred Chinese characters per second.
An average person reads Chinese at about 300 to 500 characters per minute, or 5 to 8 characters per second. If the model generates over a thousand tokens per second, it can be far faster than reading speed. At that point, the user’s attention shifts from “waiting for it to write” to “watching and judging as it goes.”
The significance for typical scenarios includes:
- Code generation: hundreds of lines of code can appear within seconds;
- Agentic tasks: faster per call, so multi-step tasks take less total time;
- Online Q&A: smoother streaming output and less perceived waiting.
But this should be viewed rationally. A thousand tokens per second usually appears under specific model architectures, hardware configurations, and concurrency conditions; it is not a universal level across all models and all scenarios. The end-to-end experience also depends on first-token latency, network conditions, output length, output stuttering, and quality.
FAQ: How to Choose a Model and How to Optimize
How should I choose a model?
Start with the quality threshold. If the task demands complex reasoning, code, or long-document understanding, favor larger-parameter or stronger MoE models; for simple Q&A, classification, or extraction, consider smaller or distilled models. For long-document scenarios, pay special attention to prefix caching and long-context support.
When should I prioritize throughput?
Scenarios like batch summarization, data processing, and offline content production usually care more about throughput and cost per token. Here you can increase the batch size moderately and accept slightly higher per-request latency.
When should I prioritize low latency?
Interactive chat, code completion, and customer service bots should focus on TTFT and TPOT first. In these cases, control the batch size, enable speculative decoding and prefix caching, and avoid letting long prompts block short requests.
What common pitfalls should I watch for?
Do not look only at averages—look at P95 and P99. Do not track only generation speed—monitor quality alongside it. Do not rely only on demo results—run load tests against your real input/output distribution. Do not ignore the impact of network, client rendering, and queuing strategy on the end-to-end experience.
7. Engineering Best Practices
7.1 Set Optimization Goals by Scenario
Different businesses have different goals:
- Interactive chat, code completion: prioritize TTFT and TPOT;
- Batch summarization, data processing, offline content production: prioritize system throughput and cost per token;
- Multi-step agentic tasks: aim for fast single calls while reducing unnecessary rounds and repeated context.
7.2 Model Tiering and Intelligent Routing
A common approach is to route by task difficulty:
- Send simple requests to a small model;
- Route long-context requests through prefix caching first;
- Send complex reasoning to the large model.
It is advisable to add fallback strategies: automatically escalate to the large model and retry when the small model has low confidence, fails to parse the format, times out, or fails quality checks. At the same time, monitor routing hit rate, escalation rate, error rate, P95 latency, and quality changes to avoid the routing strategy drifting over time.
7.3 Control Output Length
The longer the output, the higher the time and cost. Constraining prompts to “output only what is necessary,” avoiding redundant reasoning, and generating long outputs in segments is often more direct than merely tuning engine parameters.
7.4 Resource Isolation and Elastic Scaling
Set up independent resource pools and quotas for different business lines or priorities to prevent long tasks from taking resources away from interactive requests. Combine auto-scaling to handle traffic peaks, and use rate limiting, queuing, and graceful degradation to protect core requests under overload.
On the cloud, enterprises can use containerized platforms to host inference services and combine logging, monitoring, and alerting to continuously observe TTFT, TPOT, throughput, and error rates.
7.5 Build a Continuous Evaluation Loop
It is advisable to establish a fixed evaluation process:
- Build up an evaluation set that covers real input and output lengths;
- Track both quality metrics and performance metrics;
- Re-run regression after every model upgrade, quantization change, or batching-parameter adjustment;
- Analyze long-tail production requests specifically.
This helps avoid the situation where “speedups degrade quality” but the average hides it.
8. Common Misconceptions
“More parameters always means slower” is not necessarily true. Sparse structures decouple total parameter count from compute, so ultra-large models can still have good generation speeds. What matters is the activated parameter count and the actual bytes read.
“High throughput” does not mean “feels fast.” Increasing the batch size can raise system throughput, but per-request latency may go up. High-concurrency experience should be measured with tail latency.
“Demo speed” does not mean “business speed.” Demos are often run on ideal hardware, with a single request and shorter outputs. Production deployments must re-evaluate against your own traffic model.
“Fast” is not just the engine’s job. Prompt length, output length, concurrency strategy, network paths, and client rendering all affect the final experience.
9. Conclusion
LLM inference speed is not the credit of any single technique, but a systems-engineering effort across model architecture, inference engine, quantization, parallelism, and deployment.
- At the model layer, sparsification, attention structure optimization, and distillation make each token compute a little less;
- At the engine layer, kernel fusion, paged caching, prefix caching, and speculative decoding make each step compute more cleverly;
- At the system layer, quantization, parallelism, and disaggregated deployment make better use of the hardware.
For developers and enterprises, rather than chasing a single isolated number, it is better to start from the business scenario and build a complete evaluation system covering TTFT, TPOT, throughput, P95, P99, and quality metrics—finding the balance point between quality, speed, and cost that best fits your needs.
