# LLM Inference at 400 tokens/s: A Three-Layer Optimization and Load-Testing Guide
400 tokens/s is not a universal service-level promise. It is the steady streaming output rate a single request can sustain under a specific mix of model version, hardware, input and output lengths, sampling parameters, concurrency, and measurement point. This article explains why output speed is an engineering metric in its own right, how it relates to TTFT and total generation time, and how to move from tens of tokens/s to hundreds without sacrificing stability under load. It covers three cooperating layers (inference engine, scheduling, infrastructure), the variables that must be fixed for reproducible load tests, a streaming benchmark script in Python, which workloads genuinely benefit from this speed tier, and the most common measurement and tuning mistakes.
TL;DR: What Does 400 tokens/s Actually Mean?
In large language model (LLM) inference, 400 tokens/s usually refers to the steady-state streaming output rate that a single request can sustain under a specific combination of hardware, model version, input/output length, and concurrency. It should not be read as a service-wide guarantee, and it cannot be discussed in isolation from the following variables:
- Single-request speed vs. aggregate throughput across many concurrent requests;
- Input length and maximum generation length;
- Sampling parameters such as temperature and top_p;
- Measurement point: client, gateway, or inside the inference engine;
- Model version, inference engine version, and hardware model;
- Whether quantization, speculative decoding, continuous batching, and other optimizations are enabled.
A more defensible way to state the same result is: given an explicitly defined set of variables, a single-request streaming output rate of 400 tokens/s typically lands in the range where real-time interaction, real-time voice, and code completion feel good.
A number of cloud model services, open-source inference frameworks, and self-hosted inference clusters are now approaching or reaching the 400 tokens/s tier for single requests. This is not purely a spec race: that tier happens to sit close to the experience boundary of real-time interaction and real-time voice.
1. Why Output Speed Deserves Its Own Engineering Metric
Traditional web services are usually measured with QPS and P99 latency. LLM inference is different: it does not return a complete response in one shot, but streams tokens one by one. User experience is shaped by two phases together.
1.1 TTFT: Time to First Token
TTFT (Time to First Token) is the time from sending a request to receiving the first token. It is largely determined by the prefill phase and correlates strongly with input length, contention inside the batch, queueing, and network round trips.
TTFT governs “how long after I click does something happen.” If TTFT is too high, the system feels sluggish even when subsequent output is fast.
1.2 ITL / TPOT: Inter-Token Latency
ITL (Inter-Token Latency) or TPOT is the average time needed to generate one token during decoding. Its reciprocal is the single-request output speed.
ITL governs “does the text flow fast enough.” If ITL is too high, users watch words crawl out one at a time and the interaction feels poor.
1.3 Getting Total Time Right
The total duration of a generation can be approximated as:
total time ≈ TTFT + output tokens / output speed
Strictly speaking, if the first token already arrives at TTFT, the remaining tokens number output tokens - 1, so:
total time ≈ TTFT + (output tokens - 1) / output speed
Example:
- TTFT: 0.4 s
- Output tokens: 800
- Output speed: 400 tokens/s
0.4 + (800 - 1) / 400 = 0.4 + 799 / 400 ≈ 2.4 s
The total is about 2.4 s, not about 2 s. If the business requires total time under 2 s while TTFT is 0.4 s, the output speed must be higher or the output must be shorter:
remaining decode time = 2.0 - 0.4 = 1.6 srequired speed ≈ 799 / 1.6 ≈ 500 tokens/s
So when discussing 400 tokens/s, you must also state the total output length and TTFT. Otherwise, the experience is easy to misjudge.
2. Three-Layer Optimization: From Inference Engine to Infrastructure
Raising output speed from tens of tokens/s to hundreds of tokens/s cannot be done through single-point tuning. A more reliable approach is to split the problem into three layers:
- Inference engine: reduce single-step decode time;
- Scheduling system: hold latency under concurrency;
- Infrastructure: keep hardware and interconnects from becoming the ceiling.
2.1 Inference Engine: Rewriting the Core Inference Path
Decoding is a classic memory-bound workload. Generating each token requires reading model weights and the KV Cache, so compute units frequently wait on data. The goal at the engine layer is therefore to reduce wall-clock time per decode step.
Common optimizations include:
- Static scheduling instead of dynamic dispatch: compile an execution plan at startup to cut runtime branching, operator selection, memory allocation, and other overhead.
- Operator fusion: fuse adjacent operators such as attention, normalization, and activation to reduce reads and writes of intermediate tensors.
- Paged KV Cache management: reduce GPU memory fragmentation and increase the number of concurrent sequences a single device can hold.
- Weight quantization: lower weight bit width to cut memory traffic, while watching for accuracy loss.
- KV Cache compression: reduce the cache read/write pressure during decoding.
- Speculative decoding: use a draft model to propose multiple tokens and have the target model verify them. It fits scenarios with low sampling randomness, but gains can shrink in high-entropy, long-tail generation.
These optimizations should never be judged by benchmark numbers alone. They must be regression-tested on your business evaluation set.
2.2 Scheduling System: Holding Latency Under Concurrency
A fast single request does not make a fast service. The real challenge is whether P99 latency stays stable when dozens of requests arrive at once.
Core tools include:
- Continuous batching: traditional static batching waits for the slowest request in the batch, so short requests get dragged down by long ones. Continuous batching schedules sequences at iteration-step granularity and raises GPU utilization.
- Chunked prefill: prefilling a long input occupies compute for a long time and causes ITL spikes for other requests in the same batch. Splitting prefill into chunks and interleaving it with decode flattens the tail.
- Disaggregated prefill and decode: prefill is compute-heavy, decode is memory-heavy. Running them on separate resource pools allows each to be tuned with its own parallelism strategy and batch parameters.
- Priority and preemption: online interactive traffic should have its own priority queues so that offline batch jobs cannot block the online path.
A useful rule of thumb: batch size is the knob between latency and throughput. Too small, and GPU utilization suffers; too large, and the latency tail rises. Find the inflection point through load testing rather than guessing from experience.
2.3 Infrastructure: Keeping Hardware from Becoming the Ceiling
No matter how fine-grained the scheduling is, it still runs on real hardware, interconnects, and memory bandwidth. Many teams hit the “400 tokens/s in the lab, half that in production” problem precisely because the infrastructure layer was not aligned.
Things to watch:
- Topology-aware parallel sharding: tensor parallelism and pipeline parallelism must match inter-GPU interconnect bandwidth. Cross-node communication on the critical path slows down every decode step.
- NUMA affinity and memory bandwidth isolation: CPU-side data preparation, post-sampling processing, and log writes can contend with GPU inference for memory bandwidth. Binding affinity helps improve the tail.
- Capacity planning and elastic scaling: model loading and KV Cache warm-up take time. High-speed services should keep reserved, always-on replicas to avoid cold starts hurting experience.
- End-to-end observability: collect at least P50/P95/P99 of TTFT and ITL, average sequences per batch, GPU memory usage, queueing time, and network time. Without queueing time, it is hard to tell whether the problem is a slow engine or a long queue.
2.4 Three-Layer Comparison
| Layer | Optimization goal | Main benefit | Main cost | Applicable boundary |
|---|---|---|---|---|
| Inference engine | Cut single-step decode time | Directly raises single-request speed | High engineering complexity, possible accuracy risk | Low latency and high single-stream speed |
| Scheduling system | Improve stability under concurrency | Better P99, higher throughput | Complex configuration, batch tuning required | Multi-request concurrent online services |
| Infrastructure | Reduce hardware and interconnect bottlenecks | Less jitter, smaller performance gap | Requires hardware operations and observability | Production stability guarantees |
3. How to Run a Credible LLM Inference Load Test
Many teams lack a shared definition of “400 tokens/s,” which inflates their load-test results. Reproducible load testing requires fixing at least the following variables.
3.1 Required Variables
| Category | Required items |
|---|---|
| Model | Model version, weight precision, tokenizer, context window |
| Hardware | GPU/CPU model, memory capacity, inter-GPU interconnect, node topology |
| Inference engine | Engine name and version, batching configuration, quantization settings, KV Cache strategy |
| Client | Client region, network location, gateway in path or not, cross-region or not |
| Request parameters | Input length, max_tokens, temperature, top_p, seed |
| Concurrency | Single-request or multi-concurrent test, concurrency level, request arrival distribution |
| Statistics | Warm-up rounds, repetitions, TTFT, ITL, tokens/s, P50/P95/P99 |
Without this information, 400 tokens/s on a single request and 400 tokens/s per stream at 64 concurrent requests describe completely different capability levels.
3.2 Reproducible Load-Test Script
Below is an illustrative streaming load-test script. Its key points are:
- Handling the
data:prefix of SSE lines; - Skipping
[DONE]; - Adding timeout, retry, and exception handling;
- Preferring the server-returned
usage.completion_tokens; - Falling back to chunk-count approximation when usage is absent, with a note on error sources.
# Illustrative code: streaming load test reporting TTFT, output speed, and measurement methodimport timeimport jsonimport statisticsimport urllib.requestdef bench_once(url,model,prompt,max_tokens=512,temperature=0.0,top_p=1.0,timeout=120,retries=3,):payload = {"model": model,"messages": [{"role": "user", "content": prompt}],"max_tokens": max_tokens,"temperature": temperature,"top_p": top_p,"stream": True,}last_error = Nonefor attempt in range(retries):try:req = urllib.request.Request(url,data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json"},)start = time.perf_counter()first_token_at = Nonechunk_count = 0usage_tokens = Nonecontent_parts = []with urllib.request.urlopen(req, timeout=timeout) as resp:for raw in resp:line = raw.decode("utf-8", errors="ignore").strip()if not line:continue# SSE lines typically start with data:if not line.startswith("data:"):continuedata = line[len("data:"):].strip()# End-of-stream markerif data == "[DONE]":breaktry:obj = json.loads(data)except json.JSONDecodeError:# Ignore heartbeats, comments, or non-standard linescontinue# If the service returns usage, prefer it for token countingusage = obj.get("usage") or {}if usage.get("completion_tokens") is not None:usage_tokens = usage["completion_tokens"]choices = obj.get("choices") or []if not choices:continuedelta = choices[0].get("delta") or {}content = delta.get("content")if content:if first_token_at is None:first_token_at = time.perf_counter()chunk_count += 1content_parts.append(content)end = time.perf_counter()if first_token_at is None:raise RuntimeError("no streamed content tokens")ttft_s = first_token_at - startdecode_s = end - first_token_atif usage_tokens is None:# Without usage, speed can only be approximated from streamed chunks.# chunk_count is not the token count; the error comes from tokenization,# chunk splitting, punctuation, and gateway forwarding.tokens_for_speed = chunk_counttoken_source = "chunk approximation; verify with a tokenizer or usage"else:tokens_for_speed = usage_tokenstoken_source = "usage"tokens_per_s = 0.0if decode_s > 0 and tokens_for_speed > 1:tokens_per_s = (tokens_for_speed - 1) / decode_sreturn {"ttft_s": round(ttft_s, 4),"tokens_per_s": round(tokens_per_s, 2),"token_source": token_source,"token_count": tokens_for_speed,"chunk_count": chunk_count,"elapsed_s": round(end - start, 4),"sample": "".join(content_parts)[:200],}except Exception as e:last_error = e# Exponential backofftime.sleep(min(8, 2 ** attempt))raise RuntimeError(f"bench failed after retries: {last_error}")def bench_summary(url, model, prompt, runs=5, **kwargs):results = [bench_once(url, model, prompt, **kwargs) for _ in range(runs)]return {"runs": runs,"ttft_median_s": round(statistics.median(r["ttft_s"] for r in results),4),"tokens_per_s_median": round(statistics.median(r["tokens_per_s"] for r in results),2),"results": results,}
3.3 Key Principles
Keep the client close to the server.
Cross-region testing folds network round trips into TTFT, so the resulting numbers are not comparable across runs.Separate single-request speed from aggregate throughput.
400 tokens/s at concurrency 1 and 400 tokens/s per stream at concurrency 64 represent completely different system capabilities. Any external statement must specify the concurrency level.Fix the input and output length distribution.
If output length is unbounded, setmax_tokensexplicitly, otherwise batching behavior is uncontrolled.Report percentiles, not averages.
Interactive experience is usually decided by P95/P99. Averages hide the tail.Warm up before measuring.
Model loading, KV Cache warm-up, and connection-pool setup all affect the first runs. Warm up for several rounds, repeat the measurements, and take the median.
4. Which Workloads Really Need 400 tokens/s
High-speed inference raises cost, so it should be selected by business value rather than pursued as the maximum speed in every scenario.
| Scenario | Needs 400 tokens/s? | Key reason |
|---|---|---|
| AI coding tools | High | Code completion requires fast responses; speed correlates with acceptance rate |
| Real-time voice interaction | Medium | Speech playback is typically slower than text generation; stable TTFT matters more |
| Conversational AI and agent loops | High | Multi-turn tool calls amplify the wait of each step |
| Batch offline processing | Low | Unit token cost and aggregate throughput matter more |
| Document summarization, data extraction | Low | Can be processed asynchronously; not sensitive to ITL |
Scenario Checklist
Before deciding whether you need the high-speed tier, work through the following:
- Is streaming output mandatory?
- Do users read the output in real time while it is generated?
- Can you tolerate a higher time to first token?
- Are there multi-turn tool calls or agent loops?
- Is real-time voice playback required?
- Can you accept higher GPU cost?
- Have you measured P95/P99, not just the average?
- Have you clearly separated single-request speed from concurrent throughput?
5. Common Mistakes and a Troubleshooting Checklist
Mistake 1: Conflating TTFT with ITL
High TTFT usually points to input that is too long, slow prefill, heavy queueing, a slow gateway, or a slow network. High ITL usually points to memory-bound decoding, a large KV Cache, too large a batch, or insufficient memory bandwidth.
Mistake 2: Comparing numbers across regions
TTFT measured in different regions is heavily affected by network round trips. For a meaningful side-by-side comparison, fix the client location, the gateway path, and the server location.
Mistake 3: Reporting only averages
Averages hide the tail. Report at least P50, P95, and P99, together with the concurrency level, input length, and output length.
Mistake 4: Ignoring quantization accuracy
Quantization reduces the pressure of reading weights, but it can cost accuracy. Before launch, regression-test on your business evaluation set instead of looking only at the speed gain.
Mistake 5: Assuming speculative decoding always pays off
Speculative decoding clearly helps on predictable text, but gains may shrink in high-entropy generation, complex reasoning, or open-ended question answering, and verification overhead can even make it slower.
Mistake 6: Treating single-request speed as a service-wide promise
If you advertise 400 tokens/s, state whether it is single-request or concurrent, and specify input/output length, sampling parameters, and the measurement point.
6. Practical Advice for Using Cloud-Hosted Model Services
For teams that do not want to run their own inference clusters, calling a cloud-hosted model service on demand is the more economical path. It removes the burden of GPU capacity planning, inference engine tuning, and model upgrades, but you still need to check:
- Whether streaming output is supported;
- Per-account concurrency and rate quotas;
- Rate limiting, backoff, and retry behavior;
- Whether observability instrumentation is available;
- Whether long context or specific model versions are supported;
- Whether billing is per input/output token or per request.
If an organization has data-compliance requirements, it can build a full call chain by combining its cloud provider’s model service, object storage, logging, and monitoring/alerting products. When evaluating any provider — for example, the relevant Baidu AI Cloud capabilities — verify the actual service terms, quotas, SLA, and observability support, and do not turn a single load-test number into a capacity commitment.
Three actionable takeaways:
- Measure before optimizing. Use percentile metrics to determine whether the bottleneck is in the inference engine, the scheduling system, or the infrastructure.
- Put the SLO into the client. Set timeouts and fallback strategies for TTFT, and use streaming rendering and cancellable requests for long answers.
- Speed is not the only dimension. At the same experience tier, also compare accuracy, stability, unit token cost, and operational complexity.
