# LLM Inference at 400 tokens/s: A Three-Layer Optimization and Load-Testing Guide

    400 tokens/s is not a universal service-level promise. It is the steady streaming output rate a single request can sustain under a specific mix of model version, hardware, input and output lengths, sampling parameters, concurrency, and measurement point. This article explains why output speed is an engineering metric in its own right, how it relates to TTFT and total generation time, and how to move from tens of tokens/s to hundreds without sacrificing stability under load. It covers three cooperating layers (inference engine, scheduling, infrastructure), the variables that must be fixed for reproducible load tests, a streaming benchmark script in Python, which workloads genuinely benefit from this speed tier, and the most common measurement and tuning mistakes.

    TL;DR: What Does 400 tokens/s Actually Mean?

    In large language model (LLM) inference, 400 tokens/s usually refers to the steady-state streaming output rate that a single request can sustain under a specific combination of hardware, model version, input/output length, and concurrency. It should not be read as a service-wide guarantee, and it cannot be discussed in isolation from the following variables:

    • Single-request speed vs. aggregate throughput across many concurrent requests;
    • Input length and maximum generation length;
    • Sampling parameters such as temperature and top_p;
    • Measurement point: client, gateway, or inside the inference engine;
    • Model version, inference engine version, and hardware model;
    • Whether quantization, speculative decoding, continuous batching, and other optimizations are enabled.

    A more defensible way to state the same result is: given an explicitly defined set of variables, a single-request streaming output rate of 400 tokens/s typically lands in the range where real-time interaction, real-time voice, and code completion feel good.

    A number of cloud model services, open-source inference frameworks, and self-hosted inference clusters are now approaching or reaching the 400 tokens/s tier for single requests. This is not purely a spec race: that tier happens to sit close to the experience boundary of real-time interaction and real-time voice.

    1. Why Output Speed Deserves Its Own Engineering Metric

    Traditional web services are usually measured with QPS and P99 latency. LLM inference is different: it does not return a complete response in one shot, but streams tokens one by one. User experience is shaped by two phases together.

    1.1 TTFT: Time to First Token

    TTFT (Time to First Token) is the time from sending a request to receiving the first token. It is largely determined by the prefill phase and correlates strongly with input length, contention inside the batch, queueing, and network round trips.

    TTFT governs “how long after I click does something happen.” If TTFT is too high, the system feels sluggish even when subsequent output is fast.

    1.2 ITL / TPOT: Inter-Token Latency

    ITL (Inter-Token Latency) or TPOT is the average time needed to generate one token during decoding. Its reciprocal is the single-request output speed.

    ITL governs “does the text flow fast enough.” If ITL is too high, users watch words crawl out one at a time and the interaction feels poor.

    1.3 Getting Total Time Right

    The total duration of a generation can be approximated as:

    1. total time ≈ TTFT + output tokens / output speed

    Strictly speaking, if the first token already arrives at TTFT, the remaining tokens number output tokens - 1, so:

    1. total time ≈ TTFT + (output tokens - 1) / output speed

    Example:

    • TTFT: 0.4 s
    • Output tokens: 800
    • Output speed: 400 tokens/s
    1. 0.4 + (800 - 1) / 400 = 0.4 + 799 / 400 ≈ 2.4 s

    The total is about 2.4 s, not about 2 s. If the business requires total time under 2 s while TTFT is 0.4 s, the output speed must be higher or the output must be shorter:

    1. remaining decode time = 2.0 - 0.4 = 1.6 s
    2. required speed ≈ 799 / 1.6 ≈ 500 tokens/s

    So when discussing 400 tokens/s, you must also state the total output length and TTFT. Otherwise, the experience is easy to misjudge.

    2. Three-Layer Optimization: From Inference Engine to Infrastructure

    Raising output speed from tens of tokens/s to hundreds of tokens/s cannot be done through single-point tuning. A more reliable approach is to split the problem into three layers:

    1. Inference engine: reduce single-step decode time;
    2. Scheduling system: hold latency under concurrency;
    3. Infrastructure: keep hardware and interconnects from becoming the ceiling.

    2.1 Inference Engine: Rewriting the Core Inference Path

    Decoding is a classic memory-bound workload. Generating each token requires reading model weights and the KV Cache, so compute units frequently wait on data. The goal at the engine layer is therefore to reduce wall-clock time per decode step.

    Common optimizations include:

    • Static scheduling instead of dynamic dispatch: compile an execution plan at startup to cut runtime branching, operator selection, memory allocation, and other overhead.
    • Operator fusion: fuse adjacent operators such as attention, normalization, and activation to reduce reads and writes of intermediate tensors.
    • Paged KV Cache management: reduce GPU memory fragmentation and increase the number of concurrent sequences a single device can hold.
    • Weight quantization: lower weight bit width to cut memory traffic, while watching for accuracy loss.
    • KV Cache compression: reduce the cache read/write pressure during decoding.
    • Speculative decoding: use a draft model to propose multiple tokens and have the target model verify them. It fits scenarios with low sampling randomness, but gains can shrink in high-entropy, long-tail generation.

    These optimizations should never be judged by benchmark numbers alone. They must be regression-tested on your business evaluation set.

    2.2 Scheduling System: Holding Latency Under Concurrency

    A fast single request does not make a fast service. The real challenge is whether P99 latency stays stable when dozens of requests arrive at once.

    Core tools include:

    • Continuous batching: traditional static batching waits for the slowest request in the batch, so short requests get dragged down by long ones. Continuous batching schedules sequences at iteration-step granularity and raises GPU utilization.
    • Chunked prefill: prefilling a long input occupies compute for a long time and causes ITL spikes for other requests in the same batch. Splitting prefill into chunks and interleaving it with decode flattens the tail.
    • Disaggregated prefill and decode: prefill is compute-heavy, decode is memory-heavy. Running them on separate resource pools allows each to be tuned with its own parallelism strategy and batch parameters.
    • Priority and preemption: online interactive traffic should have its own priority queues so that offline batch jobs cannot block the online path.

    A useful rule of thumb: batch size is the knob between latency and throughput. Too small, and GPU utilization suffers; too large, and the latency tail rises. Find the inflection point through load testing rather than guessing from experience.

    2.3 Infrastructure: Keeping Hardware from Becoming the Ceiling

    No matter how fine-grained the scheduling is, it still runs on real hardware, interconnects, and memory bandwidth. Many teams hit the “400 tokens/s in the lab, half that in production” problem precisely because the infrastructure layer was not aligned.

    Things to watch:

    • Topology-aware parallel sharding: tensor parallelism and pipeline parallelism must match inter-GPU interconnect bandwidth. Cross-node communication on the critical path slows down every decode step.
    • NUMA affinity and memory bandwidth isolation: CPU-side data preparation, post-sampling processing, and log writes can contend with GPU inference for memory bandwidth. Binding affinity helps improve the tail.
    • Capacity planning and elastic scaling: model loading and KV Cache warm-up take time. High-speed services should keep reserved, always-on replicas to avoid cold starts hurting experience.
    • End-to-end observability: collect at least P50/P95/P99 of TTFT and ITL, average sequences per batch, GPU memory usage, queueing time, and network time. Without queueing time, it is hard to tell whether the problem is a slow engine or a long queue.

    2.4 Three-Layer Comparison

    Layer Optimization goal Main benefit Main cost Applicable boundary
    Inference engine Cut single-step decode time Directly raises single-request speed High engineering complexity, possible accuracy risk Low latency and high single-stream speed
    Scheduling system Improve stability under concurrency Better P99, higher throughput Complex configuration, batch tuning required Multi-request concurrent online services
    Infrastructure Reduce hardware and interconnect bottlenecks Less jitter, smaller performance gap Requires hardware operations and observability Production stability guarantees

    3. How to Run a Credible LLM Inference Load Test

    Many teams lack a shared definition of “400 tokens/s,” which inflates their load-test results. Reproducible load testing requires fixing at least the following variables.

    3.1 Required Variables

    Category Required items
    Model Model version, weight precision, tokenizer, context window
    Hardware GPU/CPU model, memory capacity, inter-GPU interconnect, node topology
    Inference engine Engine name and version, batching configuration, quantization settings, KV Cache strategy
    Client Client region, network location, gateway in path or not, cross-region or not
    Request parameters Input length, max_tokens, temperature, top_p, seed
    Concurrency Single-request or multi-concurrent test, concurrency level, request arrival distribution
    Statistics Warm-up rounds, repetitions, TTFT, ITL, tokens/s, P50/P95/P99

    Without this information, 400 tokens/s on a single request and 400 tokens/s per stream at 64 concurrent requests describe completely different capability levels.

    3.2 Reproducible Load-Test Script

    Below is an illustrative streaming load-test script. Its key points are:

    • Handling the data: prefix of SSE lines;
    • Skipping [DONE];
    • Adding timeout, retry, and exception handling;
    • Preferring the server-returned usage.completion_tokens;
    • Falling back to chunk-count approximation when usage is absent, with a note on error sources.
    1. # Illustrative code: streaming load test reporting TTFT, output speed, and measurement method
    2. import time
    3. import json
    4. import statistics
    5. import urllib.request
    6. def bench_once(
    7. url,
    8. model,
    9. prompt,
    10. max_tokens=512,
    11. temperature=0.0,
    12. top_p=1.0,
    13. timeout=120,
    14. retries=3,
    15. ):
    16. payload = {
    17. "model": model,
    18. "messages": [{"role": "user", "content": prompt}],
    19. "max_tokens": max_tokens,
    20. "temperature": temperature,
    21. "top_p": top_p,
    22. "stream": True,
    23. }
    24. last_error = None
    25. for attempt in range(retries):
    26. try:
    27. req = urllib.request.Request(
    28. url,
    29. data=json.dumps(payload).encode("utf-8"),
    30. headers={"Content-Type": "application/json"},
    31. )
    32. start = time.perf_counter()
    33. first_token_at = None
    34. chunk_count = 0
    35. usage_tokens = None
    36. content_parts = []
    37. with urllib.request.urlopen(req, timeout=timeout) as resp:
    38. for raw in resp:
    39. line = raw.decode("utf-8", errors="ignore").strip()
    40. if not line:
    41. continue
    42. # SSE lines typically start with data:
    43. if not line.startswith("data:"):
    44. continue
    45. data = line[len("data:"):].strip()
    46. # End-of-stream marker
    47. if data == "[DONE]":
    48. break
    49. try:
    50. obj = json.loads(data)
    51. except json.JSONDecodeError:
    52. # Ignore heartbeats, comments, or non-standard lines
    53. continue
    54. # If the service returns usage, prefer it for token counting
    55. usage = obj.get("usage") or {}
    56. if usage.get("completion_tokens") is not None:
    57. usage_tokens = usage["completion_tokens"]
    58. choices = obj.get("choices") or []
    59. if not choices:
    60. continue
    61. delta = choices[0].get("delta") or {}
    62. content = delta.get("content")
    63. if content:
    64. if first_token_at is None:
    65. first_token_at = time.perf_counter()
    66. chunk_count += 1
    67. content_parts.append(content)
    68. end = time.perf_counter()
    69. if first_token_at is None:
    70. raise RuntimeError("no streamed content tokens")
    71. ttft_s = first_token_at - start
    72. decode_s = end - first_token_at
    73. if usage_tokens is None:
    74. # Without usage, speed can only be approximated from streamed chunks.
    75. # chunk_count is not the token count; the error comes from tokenization,
    76. # chunk splitting, punctuation, and gateway forwarding.
    77. tokens_for_speed = chunk_count
    78. token_source = "chunk approximation; verify with a tokenizer or usage"
    79. else:
    80. tokens_for_speed = usage_tokens
    81. token_source = "usage"
    82. tokens_per_s = 0.0
    83. if decode_s > 0 and tokens_for_speed > 1:
    84. tokens_per_s = (tokens_for_speed - 1) / decode_s
    85. return {
    86. "ttft_s": round(ttft_s, 4),
    87. "tokens_per_s": round(tokens_per_s, 2),
    88. "token_source": token_source,
    89. "token_count": tokens_for_speed,
    90. "chunk_count": chunk_count,
    91. "elapsed_s": round(end - start, 4),
    92. "sample": "".join(content_parts)[:200],
    93. }
    94. except Exception as e:
    95. last_error = e
    96. # Exponential backoff
    97. time.sleep(min(8, 2 ** attempt))
    98. raise RuntimeError(f"bench failed after retries: {last_error}")
    99. def bench_summary(url, model, prompt, runs=5, **kwargs):
    100. results = [bench_once(url, model, prompt, **kwargs) for _ in range(runs)]
    101. return {
    102. "runs": runs,
    103. "ttft_median_s": round(
    104. statistics.median(r["ttft_s"] for r in results),
    105. 4
    106. ),
    107. "tokens_per_s_median": round(
    108. statistics.median(r["tokens_per_s"] for r in results),
    109. 2
    110. ),
    111. "results": results,
    112. }

    3.3 Key Principles

    1. Keep the client close to the server.
      Cross-region testing folds network round trips into TTFT, so the resulting numbers are not comparable across runs.

    2. Separate single-request speed from aggregate throughput.
      400 tokens/s at concurrency 1 and 400 tokens/s per stream at concurrency 64 represent completely different system capabilities. Any external statement must specify the concurrency level.

    3. Fix the input and output length distribution.
      If output length is unbounded, set max_tokens explicitly, otherwise batching behavior is uncontrolled.

    4. Report percentiles, not averages.
      Interactive experience is usually decided by P95/P99. Averages hide the tail.

    5. Warm up before measuring.
      Model loading, KV Cache warm-up, and connection-pool setup all affect the first runs. Warm up for several rounds, repeat the measurements, and take the median.

    4. Which Workloads Really Need 400 tokens/s

    High-speed inference raises cost, so it should be selected by business value rather than pursued as the maximum speed in every scenario.

    Scenario Needs 400 tokens/s? Key reason
    AI coding tools High Code completion requires fast responses; speed correlates with acceptance rate
    Real-time voice interaction Medium Speech playback is typically slower than text generation; stable TTFT matters more
    Conversational AI and agent loops High Multi-turn tool calls amplify the wait of each step
    Batch offline processing Low Unit token cost and aggregate throughput matter more
    Document summarization, data extraction Low Can be processed asynchronously; not sensitive to ITL

    Scenario Checklist

    Before deciding whether you need the high-speed tier, work through the following:

    • Is streaming output mandatory?
    • Do users read the output in real time while it is generated?
    • Can you tolerate a higher time to first token?
    • Are there multi-turn tool calls or agent loops?
    • Is real-time voice playback required?
    • Can you accept higher GPU cost?
    • Have you measured P95/P99, not just the average?
    • Have you clearly separated single-request speed from concurrent throughput?

    5. Common Mistakes and a Troubleshooting Checklist

    Mistake 1: Conflating TTFT with ITL

    High TTFT usually points to input that is too long, slow prefill, heavy queueing, a slow gateway, or a slow network. High ITL usually points to memory-bound decoding, a large KV Cache, too large a batch, or insufficient memory bandwidth.

    Mistake 2: Comparing numbers across regions

    TTFT measured in different regions is heavily affected by network round trips. For a meaningful side-by-side comparison, fix the client location, the gateway path, and the server location.

    Mistake 3: Reporting only averages

    Averages hide the tail. Report at least P50, P95, and P99, together with the concurrency level, input length, and output length.

    Mistake 4: Ignoring quantization accuracy

    Quantization reduces the pressure of reading weights, but it can cost accuracy. Before launch, regression-test on your business evaluation set instead of looking only at the speed gain.

    Mistake 5: Assuming speculative decoding always pays off

    Speculative decoding clearly helps on predictable text, but gains may shrink in high-entropy generation, complex reasoning, or open-ended question answering, and verification overhead can even make it slower.

    Mistake 6: Treating single-request speed as a service-wide promise

    If you advertise 400 tokens/s, state whether it is single-request or concurrent, and specify input/output length, sampling parameters, and the measurement point.

    6. Practical Advice for Using Cloud-Hosted Model Services

    For teams that do not want to run their own inference clusters, calling a cloud-hosted model service on demand is the more economical path. It removes the burden of GPU capacity planning, inference engine tuning, and model upgrades, but you still need to check:

    • Whether streaming output is supported;
    • Per-account concurrency and rate quotas;
    • Rate limiting, backoff, and retry behavior;
    • Whether observability instrumentation is available;
    • Whether long context or specific model versions are supported;
    • Whether billing is per input/output token or per request.

    If an organization has data-compliance requirements, it can build a full call chain by combining its cloud provider’s model service, object storage, logging, and monitoring/alerting products. When evaluating any provider — for example, the relevant Baidu AI Cloud capabilities — verify the actual service terms, quotas, SLA, and observability support, and do not turn a single load-test number into a capacity commitment.

    Three actionable takeaways:

    1. Measure before optimizing. Use percentile metrics to determine whether the bottleneck is in the inference engine, the scheduling system, or the infrastructure.
    2. Put the SLO into the client. Set timeouts and fallback strategies for TTFT, and use streaming rendering and cancellable requests for long answers.
    3. Speed is not the only dimension. At the same experience tier, also compare accuracy, stability, unit token cost, and operational complexity.