LLM Inference Throughput Gains: Metrics, Attribution, and Benchmarking

    This article explains how to interpret large language model inference throughput claims, especially single-GPU decode throughput improvements. It defines comparable throughput metrics, models decode performance using memory-bandwidth and roofline analysis, examines expert parallelism and interconnect-domain effects, reviews software optimization mechanisms, and provides a reproducible benchmark methodology with attribution and capacity-planning practices.

    Introduction: How to Read a “Tens of Times Single-GPU Throughput Gain” Claim

    A recent inference performance result has attracted considerable attention: a team behind a new-generation rack-scale acceleration system published benchmark results for a very large mixture-of-experts (MoE) model. The report claimed that single-GPU decode throughput was more than an order of magnitude higher than that of the previous-generation rack-scale system.

    Such numbers spread quickly, but comparing them horizontally without decomposing them can lead to the incorrect conclusion that simply replacing hardware will increase throughput by tens of times.

    When people search for questions about large-model inference throughput or benchmark selection, they usually want to answer two practical questions:

    • Can the reported multiplier be trusted?
    • If throughput does not scale after an upgrade, what should be checked first?

    This article does not evaluate the merits of any specific hardware or product. Instead, it focuses on three tasks:

    • Decompose the real sources of throughput gains and attribute the multiplier to verifiable variables.
    • Provide a reproducible and comparable benchmarking method, including runnable code.
    • Summarize practical guidance for capacity planning and continuous observation after deployment.

    1. Align the Definition First: What Is a Throughput Number Actually Measuring?

    The same term, “throughput,” can mean numbers that differ by several times across reports. Before comparing results, at least five dimensions should be aligned.

    • Throughput unit: There are three common definitions:

      • Output-token throughput only.
      • Combined input and output token throughput.
      • Completed requests per second.

      In scenarios with long inputs, the difference among these three can be very large.

    • Measurement scope: Is the number an aggregate rack throughput divided by the number of GPUs, or a measured value from a single GPU independently hosting the model?

    • Model and precision: Are the parameter count, activated-parameter ratio, weight precision, and KV-cache precision the same? The same model at different precisions is not directly comparable.

    • Software configuration: Are speculative decoding, prefix caching, and chunked prefill enabled? Comparing a fully optimized configuration against a default configuration usually measures software-stack differences rather than hardware differences alone.

    • Workload distribution: Are input length, output length, concurrency, and sampling parameters fixed? If output length is not fixed, throughput can be amplified by random factors.

    Among these, “single-GPU throughput” is especially easy to misread. For clarity, this article defines it as follows:

    Single-GPU throughput in this article = sum of decode throughput across all requests within the interconnect domain ÷ number of GPUs in that interconnect domain.

    This metric is only meaningful when bound to the interconnect-domain size and parallelism strategy. It is not the same as the capability of one GPU independently hosting the model. An average obtained by dividing aggregate rack throughput by GPU count implicitly assumes that multiple GPUs are collaborating within the same high-bandwidth interconnect domain.

    A configuration manifest can help keep these variables fixed across comparisons:

    1. # bench-config.yaml
    2. # Confirm each item before cross-generation comparison to avoid metric drift.
    3. model:
    4. total_params: 671B # Total parameter count determines memory capacity requirement
    5. activated_params_per_token: 37B # Activated parameters per token determine bandwidth requirement
    6. weight_dtype: fp8 # Weight precision
    7. kv_dtype: fp8 # KV-cache precision
    8. parallelism:
    9. expert_parallel_size: 72 # Expert parallelism
    10. tensor_parallel_size: 8 # Tensor parallelism within a node
    11. dense_replication: per_card # Whether dense components keep one copy per GPU
    12. workload:
    13. input_len: { p50: 1024, p99: 8192 }
    14. output_len: 512 # Fixed output length
    15. concurrency: [1, 8, 32, 128, 512]
    16. sampling:
    17. temperature: 0.0 # Disable randomness for reproducibility
    18. features:
    19. chunked_prefill: true
    20. prefix_cache: true
    21. speculative_decoding: false
    22. metrics:
    23. - output_tps
    24. - ttft_p99
    25. - tpot_p99
    26. - goodput

    2. Root-Cause Analysis: Decode-Stage Performance Is Bounded by Memory Bandwidth

    1. Prefill and Decode Have Different Performance Models

    Autoregressive inference has two stages with very different bottlenecks:

    • Prefill: Processes the entire input sequence in parallel. Matrix multiplications have high arithmetic intensity, so the bottleneck is compute.
    • Decode: Generates one token per step. It must move weights from memory into compute units, with very low data reuse. The bottleneck is memory bandwidth.

    Using the roofline model: when an operator’s arithmetic intensity, measured in FLOPs/Byte, is below the hardware ridge point, defined as peak compute divided by peak bandwidth, performance is determined by bandwidth. Decode-phase matrix-vector operations are a typical low-arithmetic-intensity workload. The theoretical upper bound of single-GPU decode throughput can be approximated as:

    1. Decode throughput upper bound ≈ memory bandwidth ÷ weight bytes moved per generated token

    Using the earlier configuration as an example: if each token activates 37 billion parameters, FP8 uses approximately 1 byte per parameter, or about 37 GB per token. If a GPU has 8 TB/s memory bandwidth:

    1. 8 TB/s ÷ 37 GB/token ≈ 216 tokens/s

    This estimate ignores KV-cache reads and writes, communication, and operator efficiency. It is only an order-of-magnitude reference and does not represent any specific product measurement.

    2. Batching Amortizes Fixed Costs, While KV Cache Determines the Inflection Point

    The weights must be read completely at every step. This is a fixed cost. Each additional request requires only its own KV cache to be read, which is a marginal cost.

    KV-cache size per token can be calculated as follows, with separate K and V entries:

    1. KV bytes per token = 2 × number of layers × number of KV heads × head dimension × bytes per cache element

    Using illustrative parameters: 61 layers, 4 KV heads per layer, head dimension 128, and FP16 cache with 2 bytes per element:

    1. 2 × 61 × 4 × 128 × 2 B = 124,928 B ≈ 125 KB/token

    A single sequence of length 8192 therefore occupies:

    1. 8192 × 125 KB ≈ 1.02 GB

    Even if all 256 GB of memory were used for KV cache, it could hold only about 250 full-length 8K sequences. At 90% usable capacity, that becomes about 225 sequences. The number of sequences that can fit must be derived from the same per-token calculation; otherwise the capacity conclusion becomes distorted.

    Batch size Context Weight read per step KV read per step Total data per step Step time at ideal 8 TB/s Aggregate output throughput
    1 2K 37 GB 0.26 GB 37.26 GB 4.66 ms about 215 tokens/s
    16 2K 37 GB 4.09 GB 41.09 GB 5.14 ms about 3110 tokens/s
    128 2K 37 GB 32.8 GB 69.8 GB 8.72 ms about 14,700 tokens/s
    128 8K 37 GB 131 GB 168 GB 21.0 ms about 6100 tokens/s

    Note: This table assumes a single GPU independently hosting the workload. It illustrates trends only. It does not include operator efficiency, communication, or sampling overhead, so real measurements are usually lower than the ideal values. The fourth row is calculated using 125 KB/token; its 131 GB KV cache already consumes about half of 256 GB of memory, meaning capacity rather than bandwidth becomes the constraint.

    Three conclusions can be drawn:

    1. When context is short, weight reading dominates. Throughput grows approximately linearly with batch size. Going from batch size 1 to 128 increases throughput by about 68×.

    2. As context length increases, KV-cache reads and writes replace weight reading as the main overhead. At the same batch size of 128, 8K context throughput is only about 41% of 2K context throughput.

    3. When context is sufficiently long, total data moved per step becomes approximately proportional to batch size × context length. Improving throughput in this regime requires wider bandwidth, more efficient attention operators, or lower-precision KV cache.

    Low latency and throughput are not the same objective. At batch size 1, per-token latency is about 4.7 ms. At batch size 128, theoretical step time rises only to 8.7 ms, but queuing and inter-step waiting can significantly increase tail latency. Online services should maximize throughput under explicit latency constraints, as discussed in Section 5.

    3. Topology Attribution: How Parallelism Strategy Determines “Single-GPU Throughput”

    When a single GPU cannot hold the complete model, performance depends on how the model is partitioned and how the resulting parts communicate.

    Common parallelism methods include:

    • Tensor parallelism: A single layer’s weights are split across multiple GPUs, requiring all-reduce synchronization per layer.
    • Pipeline parallelism: Layers are partitioned across devices, which can introduce pipeline bubbles.
    • Expert parallelism: Different experts of an MoE model are placed on different GPUs. After routing, tokens require all-to-all dispatch and gathering.
    • Data parallelism: Multiple full model replicas are deployed, with a front-end load balancer distributing requests.

    The key difference in rack-scale systems is the interconnect domain. Accelerator cards across multiple servers can form a unified interconnect domain through a high-bandwidth, low-latency switching network. A larger interconnect domain allows each GPU to hold fewer expert shards, reducing the weight data that must be moved per step.

    1. A Checkable Model: Expert Sharding Plus Dense Replication

    Assume that in a certain MoE model, experts account for 90% of activated weights, while dense components such as attention account for 10%. Using the same basis as Section 2:

    • Dense portion D = 41.1 GB × 10% ≈ 4.11 GB
    • Activated expert portion E = 41.1 GB × 90% ≈ 37 GB

    Let N be the number of GPUs in the interconnect domain. Expert weights are evenly distributed across the GPUs, while the dense portion keeps one copy on every GPU; dense weights partitioned by tensor parallelism also need to reside in each GPU of the tensor-parallel group. The actual weight read per step on a single GPU is:

    1. R(N) = D + E / N = 4.11 GB + 37 GB / N

    Therefore, an 8-GPU configuration cannot be estimated by simply computing 41.1 ÷ 8, because the dense replica cannot be amortized.

    Configuration Per-GPU read per step Single-GPU decode throughput at ideal 8 TB/s Per-GPU weight residency if 671GB is evenly split
    Single GPU hosting, theoretical 41.11 GB about 195 tokens/s 671 GB, does not fit
    8 GPUs in one interconnect domain 4.11 + 4.63 = 8.74 GB about 915 tokens/s 84 GB/GPU
    32 GPUs in one interconnect domain 4.11 + 1.16 = 5.27 GB about 1520 tokens/s 21 GB/GPU
    72 GPUs in one interconnect domain 4.11 + 0.51 = 4.62 GB about 1730 tokens/s 9.3 GB/GPU
    N → ∞, upper bound 4.11 GB about 1945 tokens/s —

    Note: The four measured rows use the same D and E parameters and the same bandwidth assumption. Only N changes. Communication overhead is not included.

    2. Three Counterintuitive Conclusions

    1. Simple division overestimates throughput by about 70%. Using 41.1 ÷ 8 = 5.14 GB would imply 1557 tokens/s, while the formula gives 915 tokens/s. The entire gap comes from redundant reads of the dense replica.

    2. Single-GPU throughput has a ceiling determined by the dense portion. In this example, the upper bound is about 1945 tokens/s. Moving from 32 to 72 GPUs improves throughput by only about 14%, whereas moving from 1 to 8 GPUs improves it by about 4.7×. The higher the expert ratio, the greater the benefit from enlarging the interconnect domain.

    3. Interconnect-domain size itself is a source of throughput. With the same 32 GPUs, splitting them into four 8-GPU domains gives aggregate throughput of approximately 32 × 915 ≈ 29,300 tokens/s. Organizing them into one 32-GPU domain gives approximately 32 × 1520 ≈ 48,600 tokens/s, a difference of about 1.66×. This portion of the gain should be attributed to interconnect and parallelism strategy, not to each individual GPU becoming faster.

    There are also two additional discounts:

    • Communication overhead: all-to-all transfer moves a relatively small amount of token data, but decode steps require synchronization. Cross-node latency and congestion can make measured results significantly lower than ideal values.
    • Load imbalance: shards containing popular experts finish first, but the whole step is determined by the slowest shard.

    Therefore, a reported “tens of times single-GPU throughput gain” is usually an average throughput per GPU. Topology typically contributes a few times. The remaining multiplier must be explained jointly by precision changes, such as roughly 2× from FP16 to FP8, higher per-GPU bandwidth, software-stack differences, and model architecture differences. These four factors should be measured separately.

    4. Software Stack: Turning Hardware Potential into Actual Throughput

    Hardware specifications determine the upper bound. The software stack determines how much of that upper bound can be realized. Common techniques include:

    1. Scheduling optimization

      • Continuous batching: release slots as soon as requests complete, eliminating gaps in static batching.
      • Chunked prefill: split long inputs into chunks and mix them with decode execution, preventing long requests from blocking ongoing generation.
      • Priority and preemption: prioritize interactive requests and swap out low-priority requests when necessary to stabilize tail latency.
    2. Memory management

      • Paged KV-cache management: allocate cache in blocks to reduce fragmentation.
      • Prefix caching: reuse computation results for identical system prompts, document prefixes, and other shared prefixes.

      In multi-turn customer-service scenarios where each turn carries the same long knowledge-base context, prefix-cache hit rate directly determines cost.

    3. Quantization

      • Weight quantization, such as FP8 or W4A8, directly reduces the number of bytes moved per step.
      • KV-cache quantization, such as FP8, increases the number of concurrent sequences that can be held.

      Quantization introduces accuracy loss and should be evaluated with business-specific regression tests.

    4. Operator and execution optimization

      • High-throughput attention operators.
      • Operator fusion.
      • Graph capture to reduce launch overhead.
      • Warp-level reduction.
      • Fuse expert routing, grouped GEMM, and communication to reduce intermediate-result reads and writes.
    5. Speculative decoding

      • A small model or extra prediction head proposes multiple candidate tokens at once, and the large model verifies them in parallel.

      Higher acceptance rates yield greater gains. Candidate generation and verification still incur overhead, so this technique is best suited to workloads with relatively deterministic output.

    6. Prefill-decode disaggregation

      • The two stages have different bottlenecks. Mixing them can cause interference.

      Deploying them in separate instance groups and transferring KV cache over a high-speed network allows each pool to be sized according to its own bottleneck: compute-bound prefill or bandwidth-bound decode.

    Roofline Theoretical Upper-Bound Calculation

    1. def theoretical_decode_tps(
    2. activated_params: float, # Activated parameters per token
    3. bytes_per_param: float, # Bytes per parameter implied by weight precision; FP8 = 1
    4. bandwidth_bps: float, # Memory bandwidth in bytes per second
    5. ) -> float:
    6. """Estimate single-GPU decode throughput upper bound from memory bandwidth (tokens/s).
    7. This only considers weight reads and ignores KV cache, communication, and operator efficiency.
    8. The result is intended for order-of-magnitude checks, not as a product benchmark result.
    9. """
    10. return bandwidth_bps / (activated_params * bytes_per_param)
    11. def kv_bytes_per_token(
    12. layers: int, kv_heads: int, head_dim: int, dtype_bytes: int
    13. ) -> int:
    14. """KV-cache bytes per token, counting both K and V."""
    15. return 2 * layers * kv_heads * head_dim * dtype_bytes
    16. def per_card_read_gb(
    17. dense_gb: float, activated_expert_gb: float, domain_size: int
    18. ) -> float:
    19. """Per-GPU bytes read per step.
    20. Dense components keep one replica per GPU, while activated expert components are evenly split across GPUs in the interconnect domain.
    21. """
    22. return dense_gb + activated_expert_gb / domain_size
    23. if __name__ == "__main__":
    24. print(f"{theoretical_decode_tps(37e9, 1, 8e12):.0f} tokens/s") # about 216
    25. print(f"{kv_bytes_per_token(61, 4, 128, 2) / 1024:.0f} KB/token") # about 122 KiB, roughly 125 KB
    26. for n in (8, 32, 72):
    27. r = per_card_read_gb(4.11, 37.0, n)
    28. tps = theoretical_decode_tps(1, 1, 8e12 / (r * 1e9))
    29. print(f"N={n}: {r:.2f} GB/step -> {tps:.0f} tokens/s")

    5. Benchmarking: Making Comparisons Reproducible and Attributable

    1. Metric Selection

    Before answering how to choose a benchmark, define the metrics:

    • Throughput: output tokens generated per unit time. Specify whether input tokens are included.
    • TTFT: time to first token. It determines how quickly an interactive application responds after the user presses Enter.
    • TPOT: time per output token. It determines the smoothness of reading and voice playback.
    • Goodput: throughput contributed only by requests that satisfy latency constraints. Counting only successful requests avoids the illusion of “high throughput while many requests time out.”
    • Resource efficiency: throughput per GPU and throughput per megawatt, useful for cost and energy decisions.

    2. Test Procedure

    1. Fix the configuration manifest from Section 1 and record hardware model, driver version, and framework version.
    2. Warm up: complete weight loading, graph capture, and cache warmup before starting measurement.
    3. Fix output length by setting a minimum token count, and disable sampling randomness.
    4. Send several hundred requests at each concurrency level, such as 1, 8, 32, 128, and 512. The sample size should be large enough to cover P99; at least 300 requests per level is recommended.
    5. Record TTFT, TPOT, end-to-end latency, and completed token count. Compute P50/P95/P99. Also record client and gateway CPU, memory, and connection counts.
    6. Plot the throughput-latency curve and identify the maximum concurrency that satisfies the SLO. The throughput at that point is the goodput.

    3. Example Code

    Dependencies: Python 3.9+ and standard library only.

    Server requirements: an OpenAI-style /v1/chat/completions endpoint that supports stream=true and a min_tokens parameter for fixed output length. The SSE parser should tolerate empty lines, heartbeat frames, empty data: frames, and partial JSON. Set a per-request timeout of 300 seconds and retry twice.

    1. # benchmark.py
    2. from __future__ import annotations
    3. import json
    4. import queue
    5. import threading
    6. import time
    7. import urllib.error
    8. import urllib.request
    9. from typing import Any
    10. ENDPOINT = "http://127.0.0.1:8000/v1/chat/completions" # Internal self-hosted inference service
    11. MODEL = "local-model"
    12. INPUT_LEN = 1024
    13. OUTPUT_LEN = 512
    14. WARMUP = 8
    15. TIMEOUT = 300
    16. RETRY = 2
    17. PROMPT = "Please reason step by step and answer:" + "context example." * (INPUT_LEN // 6)
    18. def one_request() -> dict[str, float]:
    19. """Send one streaming request and return ttft, latency, and tokens.
    20. Exceptions are propagated to the caller.
    21. """
    22. body = json.dumps(
    23. {
    24. "model": MODEL,
    25. "messages": [{"role": "user", "content": PROMPT}],
    26. "max_tokens": OUTPUT_LEN,
    27. "min_tokens": OUTPUT_LEN, # Fix output length to keep metrics comparable
    28. "temperature": 0.0, # Disable random sampling
    29. "stream": True,
    30. }
    31. ).encode()
    32. req = urllib.request.Request(
    33. ENDPOINT,
    34. data=body,
    35. headers={"Content-Type": "application/json"},
    36. )
    37. t0 = time.perf_counter()
    38. first, tokens = None, 0
    39. with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
    40. for raw in resp: # Read SSE line by line
    41. line = raw.decode("utf-8", "replace").strip()
    42. if not line.startswith("data:"):
    43. continue # Tolerate empty lines and heartbeats
    44. payload = line[len("data:"):].strip()
    45. if not payload:
    46. continue # Tolerate empty data frames
    47. if payload == "[DONE]":
    48. break
    49. try:
    50. chunk = json.loads(payload)
    51. except json.JSONDecodeError:
    52. continue # Ignore partial or malformed frames
    53. choices = chunk.get("choices") or []
    54. if not choices:
    55. continue
    56. delta = (choices[0].get("delta") or {}).get("content") or ""
    57. if not delta:
    58. continue
    59. now = time.perf_counter()
    60. if first is None:
    61. first = now
    62. tokens += 1
    63. last = time.perf_counter()
    64. if first is None or tokens == 0:
    65. raise RuntimeError(
    66. "No content deltas received. Confirm that the server supports stream and min_tokens."
    67. )
    68. return {"ttft": first - t0, "latency": last - t0, "tokens": float(tokens)}
    69. def call_with_retry(attempts: int = RETRY) -> dict[str, float]:
    70. last_exc: Exception | None = None
    71. for _ in range(attempts + 1):
    72. try:
    73. return one_request()
    74. except (urllib.error.URLError, TimeoutError, RuntimeError) as exc:
    75. last_exc = exc
    76. time.sleep(0.5)
    77. raise RuntimeError("Request failed after retries") from last_exc
    78. def percentile(values: list[float], p: float) -> float:
    79. """Linear interpolation percentile.
    80. A sample size of at least 300 is recommended for a stable P99 estimate.
    81. """
    82. s = sorted(values)
    83. if len(s) == 1:
    84. return s[0]
    85. k = (len(s) - 1) * p
    86. lo = int(k)
    87. hi = min(lo + 1, len(s) - 1)
    88. return s[lo] + (s[hi] - s[lo]) * (k - lo)
    89. def benchmark(concurrency: int, total: int = 512) -> dict[str, Any]:
    90. """Fixed-concurrency benchmark.
    91. Workers pull tasks from a shared queue, and the timer stops only after total requests complete.
    92. """
    93. if total < concurrency:
    94. total = concurrency
    95. for _ in range(WARMUP): # Warm up weights, graph capture, and cache allocation
    96. call_with_retry()
    97. work: queue.Queue[int] = queue.Queue()
    98. for i in range(total):
    99. work.put(i)
    100. results: queue.Queue[dict[str, float]] = queue.Queue()
    101. errors: queue.Queue[BaseException] = queue.Queue()
    102. def worker() -> None:
    103. while True:
    104. try:
    105. work.get_nowait()
    106. except queue.Empty:
    107. return # Exit when the queue is empty
    108. try:
    109. results.put(call_with_retry())
    110. except BaseException as exc: # Report failures so the main thread does not hang
    111. errors.put(exc)
    112. threads = [threading.Thread(target=worker, daemon=True) for _ in range(concurrency)]
    113. wall_start = time.perf_counter()
    114. for t in threads:
    115. t.start()
    116. for t in threads:
    117. t.join() # Stop the timer only after all workers finish
    118. wall = time.perf_counter() - wall_start
    119. collected = []
    120. while not results.empty():
    121. collected.append(results.get())
    122. if not errors.empty():
    123. print(f"[warn] {errors.qsize()} failed requests were excluded from statistics")
    124. if len(collected) < max(30, total * 0.8):
    125. raise RuntimeError(
    126. f"Only {len(collected)} valid samples collected, which is insufficient for P99"
    127. )
    128. ttft = [r["ttft"] for r in collected]
    129. tpot = [(r["latency"] - r["ttft"]) / max(r["tokens"] - 1, 1) for r in collected]
    130. return {
    131. "concurrency": concurrency,
    132. "samples": len(collected),
    133. "output_tps": sum(r["tokens"] for r in collected) / wall,
    134. "ttft_p50": round(percentile(ttft, 0.50), 3),
    135. "ttft_p99": round(percentile(ttft, 0.99), 3),
    136. "tpot_p99": round(percentile(tpot, 0.99), 4),
    137. }
    138. if __name__ == "__main__":
    139. for c in (1, 8, 32, 128, 512):
    140. print(json.dumps(benchmark(c), ensure_ascii=False))

    Notes:

    • The script targets internal self-hosted services. For production load testing, restrict source IP addresses and add access control. Do not expose an unauthenticated inference service directly to the public internet.
    • Under high concurrency, a single-threaded client can itself become a bottleneck. For formal testing, use an asynchronous client or a multi-machine distributed load generator, and include client resource specifications in the report.
    • Report output-token throughput and input-token throughput separately. Do not combine them into one number and compare it with someone else’s output-token throughput.

    3.1 Attribution Experiment: Turning Off Optimizations One by One

    When comparing across generations, design four groups:

    • Old hardware + old software stack: control group.
    • Old hardware + new software stack: group A.
    • New hardware + old software stack: group B.
    • New hardware + new software stack: experimental group.

    Assume the control group is normalized to 1. Measured results are A = 2.5, B = 4, and experimental group = 58. Even if hardware and software contributions were completely independent, 2.5 × 4 = 10, which is still 5.8× short of 58. This indicates that some factors were not measured separately. Common sources include:

    • Parallelism changes caused by a larger interconnect domain.
    • Differences in the test model architecture.
    • The baseline group did not enable key optimizations.
    • Resource contention exists in one of the configurations.

    Use a unified record table for item-by-item attribution:

    Variable category Control A B Experimental Notes
    Hardware, including model, per-GPU bandwidth, and memory
    Software, including framework, version, and scheduling features
    Parallelism, including interconnect-domain size, EP/TP/PP
    Model, including total parameters, activated parameters, and precision
    Workload, including input distribution, output length, and concurrency
    Measured output throughput, tokens/s 1.0 2.5 4.0 58

    Rule: change one cell at a time and rerun the benchmark. The final multiplier must be explainable by the changes recorded in the table.

    3.2 Capacity-Planning Example

    Assume a typical latency constraint, for illustration only: TTFT P99 ≤ 2 s and TPOT P99 ≤ 50 ms. A service instance has the following measured results. Note that throughput below counts output tokens only.

    Concurrency Output throughput, tokens/s TTFT P99 TPOT P99 Meets SLO?
    1 about 200 0.3 s 4.7 ms Yes
    32 about 6,000 0.9 s 18 ms Yes
    128 about 15,000 1.6 s 38 ms Yes
    512 about 18,000 4.2 s 85 ms No

    Note: At 512 concurrency, throughput is only about 20% higher than at 128 concurrency, while latency more than doubles. Therefore, the effective throughput satisfying the SLO for this instance is 15,000 output tokens/s, not the peak 18,000.

    Assume peak business traffic is 300 QPS, average input is 2048 tokens, and output is 512 tokens. Capacity must be calculated under two separate units:

    1. Decode side, output tokens: 300 × 512 = 153,600 output tokens/s
    2. Prefill side, input tokens: 300 × 2048 = 614,400 input tokens/s
    • Decode resource pool: 153,600 ÷ 15,000 ≈ 10.24 → 11 instances. Add N+2 fault redundancy → 13 instances.
    • Prefill resource pool: if one instance can support 40,000 input tokens/s under TTFT P99 ≤ 2 s, then 614,400 ÷ 40,000 ≈ 15.36 → 16 instances. Add redundancy → 18 instances.

    The two numbers must not be mixed. Dividing 153,600 by 15,000 gives the number of decode instances. Dividing 614,400 by 15,000 is a unit error. Similarly, if peak 18,000 is mistakenly used instead of effective 15,000, the result becomes 8.5 → 9 instances, underestimating capacity by about 20% and causing widespread timeouts during peak periods.

    6. Capacity Planning and Cost: Move From “Peak Throughput” to “Effective Unit Throughput”

    • Plan with effective throughput. Estimate instance count using goodput under the SLO, and reserve capacity for failure redundancy and traffic fluctuation. Retest promptly when input or output length distributions change, such as when long-document summarization is introduced.
    • Split resource pools by stage. Prefill consumes compute, while decode consumes bandwidth and memory capacity. After separation, each pool can be scaled independently according to its own bottleneck.
    • Pay attention to energy efficiency. Rack-scale systems typically have significantly higher power consumption and cooling density than traditional servers. Tokens per megawatt and cost per thousand tokens should be included in decision-making.
    • Optimize input structure. Shorten prompts, reuse fixed prefixes, and set context windows reasonably. Reducing unnecessary tokens is the most direct way to lower cost.
    • Use general cloud capabilities when appropriate. When deploying on major public clouds, teams can combine neutral capabilities such as:
      • Object storage for model weights and datasets.
      • Container services for hosting inference instances and elastic scaling.
      • Message queues for smoothing traffic spikes.
      • Monitoring, alerting, and logging services for collecting runtime metrics and setting threshold alerts.

    7. Continuous Observation After Launch

    Benchmark results only represent a point-in-time state. After deployment, collect the following continuously:

    • Latency metrics: P50/P95/P99 of TTFT and TPOT, with particular attention to tail latency.
    • Service status: queue time, preemption and swap-out counts, goodput ratio.
    • Cache effects: prefix-cache hit rate and KV-cache block utilization.
    • Resource metrics: memory usage, interconnect-link bandwidth utilization, power, and temperature.

    It is recommended to establish a shadow-traffic mechanism. New hardware, new software versions, or new models can first receive replicated production traffic in an equivalent environment. Results are recorded but not returned to users. After covering several peak and low-traffic periods, traffic can then be shifted. At the same time, configure threshold alerts on key metrics to prevent performance regressions caused by version upgrades.

    8. Frequently Asked Questions

    1. Does “tens of times single-GPU throughput gain” mean that each GPU became tens of times faster?

      Not necessarily. This metric is usually aggregate throughput within an interconnect domain divided by GPU count. It depends on interconnect-domain size, parallelism strategy, and model architecture. Under the model in this article, topology typically contributes a few times, and there is an upper bound determined by dense replicas.

    2. What should I do if throughput drops sharply as context length increases?

      This happens because KV-cache reads and writes replace weight reads as the bottleneck. Options include KV-cache quantization, improving bandwidth utilization, rate-limiting long-context requests separately, or adopting architectures with lower cache complexity.

    3. Does speculative decoding always improve speed?

      Not always. The benefit depends on acceptance rate and draft overhead. A/B testing with real business corpora is recommended.

    4. After a hardware upgrade, throughput improvement is lower than expected. What should I check first?

      Check in this order:

      • Parallelism strategy and load balancing.
      • Communication bottlenecks, especially interconnect bandwidth utilization.
      • KV-cache capacity limits on concurrency.
      • Whether the client or gateway has become the bottleneck.

    Common benchmark mistakes:

    • Starting measurement without warmup.
    • Failing to fix output length, causing metric drift.
    • Failing to disable sampling randomness.
    • The client or gateway becomes a bottleneck, but its resource usage is not recorded.
    • Sample size is insufficient to support P99.
    • Mixing input-token and output-token metrics.
    • The baseline group did not enable key optimizations.
    • Reporting only peak throughput and not goodput.
    • The report does not state hardware model, driver version, and framework version.

    Conclusion

    Behind a claim such as “tens of times single-GPU throughput gain” are the combined effects of model architecture, parallelism topology, interconnect capability, and software scheduling, rather than a linear mapping of a single hardware parameter.

    A more reliable approach is to:

    • Align benchmark definitions, including a precise definition of single-GPU throughput.
    • Use roofline analysis and ablation experiments to attribute multipliers to individual variables.
    • Plan capacity by effective throughput and separate input and output units under SLO constraints.
    • Continue observing and validating with canary or shadow traffic after launch.

    Only in this way can a multiplier on paper become measurable improvements in responsiveness, stability, and cost efficiency.