LLM Inference Throughput Gains: Metrics, Attribution, and Benchmarking
This article explains how to interpret large language model inference throughput claims, especially single-GPU decode throughput improvements. It defines comparable throughput metrics, models decode performance using memory-bandwidth and roofline analysis, examines expert parallelism and interconnect-domain effects, reviews software optimization mechanisms, and provides a reproducible benchmark methodology with attribution and capacity-planning practices.
Introduction: How to Read a “Tens of Times Single-GPU Throughput Gain” Claim
A recent inference performance result has attracted considerable attention: a team behind a new-generation rack-scale acceleration system published benchmark results for a very large mixture-of-experts (MoE) model. The report claimed that single-GPU decode throughput was more than an order of magnitude higher than that of the previous-generation rack-scale system.
Such numbers spread quickly, but comparing them horizontally without decomposing them can lead to the incorrect conclusion that simply replacing hardware will increase throughput by tens of times.
When people search for questions about large-model inference throughput or benchmark selection, they usually want to answer two practical questions:
- Can the reported multiplier be trusted?
- If throughput does not scale after an upgrade, what should be checked first?
This article does not evaluate the merits of any specific hardware or product. Instead, it focuses on three tasks:
- Decompose the real sources of throughput gains and attribute the multiplier to verifiable variables.
- Provide a reproducible and comparable benchmarking method, including runnable code.
- Summarize practical guidance for capacity planning and continuous observation after deployment.
1. Align the Definition First: What Is a Throughput Number Actually Measuring?
The same term, “throughput,” can mean numbers that differ by several times across reports. Before comparing results, at least five dimensions should be aligned.
Throughput unit: There are three common definitions:
- Output-token throughput only.
- Combined input and output token throughput.
- Completed requests per second.
In scenarios with long inputs, the difference among these three can be very large.
Measurement scope: Is the number an aggregate rack throughput divided by the number of GPUs, or a measured value from a single GPU independently hosting the model?
Model and precision: Are the parameter count, activated-parameter ratio, weight precision, and KV-cache precision the same? The same model at different precisions is not directly comparable.
Software configuration: Are speculative decoding, prefix caching, and chunked prefill enabled? Comparing a fully optimized configuration against a default configuration usually measures software-stack differences rather than hardware differences alone.
Workload distribution: Are input length, output length, concurrency, and sampling parameters fixed? If output length is not fixed, throughput can be amplified by random factors.
Among these, “single-GPU throughput” is especially easy to misread. For clarity, this article defines it as follows:
Single-GPU throughput in this article = sum of decode throughput across all requests within the interconnect domain ÷ number of GPUs in that interconnect domain.
This metric is only meaningful when bound to the interconnect-domain size and parallelism strategy. It is not the same as the capability of one GPU independently hosting the model. An average obtained by dividing aggregate rack throughput by GPU count implicitly assumes that multiple GPUs are collaborating within the same high-bandwidth interconnect domain.
A configuration manifest can help keep these variables fixed across comparisons:
# bench-config.yaml# Confirm each item before cross-generation comparison to avoid metric drift.model:total_params: 671B # Total parameter count determines memory capacity requirementactivated_params_per_token: 37B # Activated parameters per token determine bandwidth requirementweight_dtype: fp8 # Weight precisionkv_dtype: fp8 # KV-cache precisionparallelism:expert_parallel_size: 72 # Expert parallelismtensor_parallel_size: 8 # Tensor parallelism within a nodedense_replication: per_card # Whether dense components keep one copy per GPUworkload:input_len: { p50: 1024, p99: 8192 }output_len: 512 # Fixed output lengthconcurrency: [1, 8, 32, 128, 512]sampling:temperature: 0.0 # Disable randomness for reproducibilityfeatures:chunked_prefill: trueprefix_cache: truespeculative_decoding: falsemetrics:- output_tps- ttft_p99- tpot_p99- goodput
2. Root-Cause Analysis: Decode-Stage Performance Is Bounded by Memory Bandwidth
1. Prefill and Decode Have Different Performance Models
Autoregressive inference has two stages with very different bottlenecks:
- Prefill: Processes the entire input sequence in parallel. Matrix multiplications have high arithmetic intensity, so the bottleneck is compute.
- Decode: Generates one token per step. It must move weights from memory into compute units, with very low data reuse. The bottleneck is memory bandwidth.
Using the roofline model: when an operator’s arithmetic intensity, measured in FLOPs/Byte, is below the hardware ridge point, defined as peak compute divided by peak bandwidth, performance is determined by bandwidth. Decode-phase matrix-vector operations are a typical low-arithmetic-intensity workload. The theoretical upper bound of single-GPU decode throughput can be approximated as:
Decode throughput upper bound ≈ memory bandwidth ÷ weight bytes moved per generated token
Using the earlier configuration as an example: if each token activates 37 billion parameters, FP8 uses approximately 1 byte per parameter, or about 37 GB per token. If a GPU has 8 TB/s memory bandwidth:
8 TB/s ÷ 37 GB/token ≈ 216 tokens/s
This estimate ignores KV-cache reads and writes, communication, and operator efficiency. It is only an order-of-magnitude reference and does not represent any specific product measurement.
2. Batching Amortizes Fixed Costs, While KV Cache Determines the Inflection Point
The weights must be read completely at every step. This is a fixed cost. Each additional request requires only its own KV cache to be read, which is a marginal cost.
KV-cache size per token can be calculated as follows, with separate K and V entries:
KV bytes per token = 2 × number of layers × number of KV heads × head dimension × bytes per cache element
Using illustrative parameters: 61 layers, 4 KV heads per layer, head dimension 128, and FP16 cache with 2 bytes per element:
2 × 61 × 4 × 128 × 2 B = 124,928 B ≈ 125 KB/token
A single sequence of length 8192 therefore occupies:
8192 × 125 KB ≈ 1.02 GB
Even if all 256 GB of memory were used for KV cache, it could hold only about 250 full-length 8K sequences. At 90% usable capacity, that becomes about 225 sequences. The number of sequences that can fit must be derived from the same per-token calculation; otherwise the capacity conclusion becomes distorted.
3. Batch Size and Throughput Are Not Linearly Related
| Batch size | Context | Weight read per step | KV read per step | Total data per step | Step time at ideal 8 TB/s | Aggregate output throughput |
|---|---|---|---|---|---|---|
| 1 | 2K | 37 GB | 0.26 GB | 37.26 GB | 4.66 ms | about 215 tokens/s |
| 16 | 2K | 37 GB | 4.09 GB | 41.09 GB | 5.14 ms | about 3110 tokens/s |
| 128 | 2K | 37 GB | 32.8 GB | 69.8 GB | 8.72 ms | about 14,700 tokens/s |
| 128 | 8K | 37 GB | 131 GB | 168 GB | 21.0 ms | about 6100 tokens/s |
Note: This table assumes a single GPU independently hosting the workload. It illustrates trends only. It does not include operator efficiency, communication, or sampling overhead, so real measurements are usually lower than the ideal values. The fourth row is calculated using 125 KB/token; its 131 GB KV cache already consumes about half of 256 GB of memory, meaning capacity rather than bandwidth becomes the constraint.
Three conclusions can be drawn:
When context is short, weight reading dominates. Throughput grows approximately linearly with batch size. Going from batch size 1 to 128 increases throughput by about 68×.
As context length increases, KV-cache reads and writes replace weight reading as the main overhead. At the same batch size of 128, 8K context throughput is only about 41% of 2K context throughput.
When context is sufficiently long, total data moved per step becomes approximately proportional to batch size × context length. Improving throughput in this regime requires wider bandwidth, more efficient attention operators, or lower-precision KV cache.
Low latency and throughput are not the same objective. At batch size 1, per-token latency is about 4.7 ms. At batch size 128, theoretical step time rises only to 8.7 ms, but queuing and inter-step waiting can significantly increase tail latency. Online services should maximize throughput under explicit latency constraints, as discussed in Section 5.
3. Topology Attribution: How Parallelism Strategy Determines “Single-GPU Throughput”
When a single GPU cannot hold the complete model, performance depends on how the model is partitioned and how the resulting parts communicate.
Common parallelism methods include:
- Tensor parallelism: A single layer’s weights are split across multiple GPUs, requiring all-reduce synchronization per layer.
- Pipeline parallelism: Layers are partitioned across devices, which can introduce pipeline bubbles.
- Expert parallelism: Different experts of an MoE model are placed on different GPUs. After routing, tokens require all-to-all dispatch and gathering.
- Data parallelism: Multiple full model replicas are deployed, with a front-end load balancer distributing requests.
The key difference in rack-scale systems is the interconnect domain. Accelerator cards across multiple servers can form a unified interconnect domain through a high-bandwidth, low-latency switching network. A larger interconnect domain allows each GPU to hold fewer expert shards, reducing the weight data that must be moved per step.
1. A Checkable Model: Expert Sharding Plus Dense Replication
Assume that in a certain MoE model, experts account for 90% of activated weights, while dense components such as attention account for 10%. Using the same basis as Section 2:
- Dense portion D = 41.1 GB × 10% ≈ 4.11 GB
- Activated expert portion E = 41.1 GB × 90% ≈ 37 GB
Let N be the number of GPUs in the interconnect domain. Expert weights are evenly distributed across the GPUs, while the dense portion keeps one copy on every GPU; dense weights partitioned by tensor parallelism also need to reside in each GPU of the tensor-parallel group. The actual weight read per step on a single GPU is:
R(N) = D + E / N = 4.11 GB + 37 GB / N
Therefore, an 8-GPU configuration cannot be estimated by simply computing 41.1 ÷ 8, because the dense replica cannot be amortized.
| Configuration | Per-GPU read per step | Single-GPU decode throughput at ideal 8 TB/s | Per-GPU weight residency if 671GB is evenly split |
|---|---|---|---|
| Single GPU hosting, theoretical | 41.11 GB | about 195 tokens/s | 671 GB, does not fit |
| 8 GPUs in one interconnect domain | 4.11 + 4.63 = 8.74 GB | about 915 tokens/s | 84 GB/GPU |
| 32 GPUs in one interconnect domain | 4.11 + 1.16 = 5.27 GB | about 1520 tokens/s | 21 GB/GPU |
| 72 GPUs in one interconnect domain | 4.11 + 0.51 = 4.62 GB | about 1730 tokens/s | 9.3 GB/GPU |
| N → ∞, upper bound | 4.11 GB | about 1945 tokens/s | — |
Note: The four measured rows use the same D and E parameters and the same bandwidth assumption. Only N changes. Communication overhead is not included.
2. Three Counterintuitive Conclusions
Simple division overestimates throughput by about 70%. Using 41.1 ÷ 8 = 5.14 GB would imply 1557 tokens/s, while the formula gives 915 tokens/s. The entire gap comes from redundant reads of the dense replica.
Single-GPU throughput has a ceiling determined by the dense portion. In this example, the upper bound is about 1945 tokens/s. Moving from 32 to 72 GPUs improves throughput by only about 14%, whereas moving from 1 to 8 GPUs improves it by about 4.7×. The higher the expert ratio, the greater the benefit from enlarging the interconnect domain.
Interconnect-domain size itself is a source of throughput. With the same 32 GPUs, splitting them into four 8-GPU domains gives aggregate throughput of approximately 32 × 915 ≈ 29,300 tokens/s. Organizing them into one 32-GPU domain gives approximately 32 × 1520 ≈ 48,600 tokens/s, a difference of about 1.66×. This portion of the gain should be attributed to interconnect and parallelism strategy, not to each individual GPU becoming faster.
There are also two additional discounts:
- Communication overhead: all-to-all transfer moves a relatively small amount of token data, but decode steps require synchronization. Cross-node latency and congestion can make measured results significantly lower than ideal values.
- Load imbalance: shards containing popular experts finish first, but the whole step is determined by the slowest shard.
Therefore, a reported “tens of times single-GPU throughput gain” is usually an average throughput per GPU. Topology typically contributes a few times. The remaining multiplier must be explained jointly by precision changes, such as roughly 2× from FP16 to FP8, higher per-GPU bandwidth, software-stack differences, and model architecture differences. These four factors should be measured separately.
4. Software Stack: Turning Hardware Potential into Actual Throughput
Hardware specifications determine the upper bound. The software stack determines how much of that upper bound can be realized. Common techniques include:
Scheduling optimization
- Continuous batching: release slots as soon as requests complete, eliminating gaps in static batching.
- Chunked prefill: split long inputs into chunks and mix them with decode execution, preventing long requests from blocking ongoing generation.
- Priority and preemption: prioritize interactive requests and swap out low-priority requests when necessary to stabilize tail latency.
Memory management
- Paged KV-cache management: allocate cache in blocks to reduce fragmentation.
- Prefix caching: reuse computation results for identical system prompts, document prefixes, and other shared prefixes.
In multi-turn customer-service scenarios where each turn carries the same long knowledge-base context, prefix-cache hit rate directly determines cost.
Quantization
- Weight quantization, such as FP8 or W4A8, directly reduces the number of bytes moved per step.
- KV-cache quantization, such as FP8, increases the number of concurrent sequences that can be held.
Quantization introduces accuracy loss and should be evaluated with business-specific regression tests.
Operator and execution optimization
- High-throughput attention operators.
- Operator fusion.
- Graph capture to reduce launch overhead.
- Warp-level reduction.
- Fuse expert routing, grouped GEMM, and communication to reduce intermediate-result reads and writes.
Speculative decoding
- A small model or extra prediction head proposes multiple candidate tokens at once, and the large model verifies them in parallel.
Higher acceptance rates yield greater gains. Candidate generation and verification still incur overhead, so this technique is best suited to workloads with relatively deterministic output.
Prefill-decode disaggregation
- The two stages have different bottlenecks. Mixing them can cause interference.
Deploying them in separate instance groups and transferring KV cache over a high-speed network allows each pool to be sized according to its own bottleneck: compute-bound prefill or bandwidth-bound decode.
Roofline Theoretical Upper-Bound Calculation
def theoretical_decode_tps(activated_params: float, # Activated parameters per tokenbytes_per_param: float, # Bytes per parameter implied by weight precision; FP8 = 1bandwidth_bps: float, # Memory bandwidth in bytes per second) -> float:"""Estimate single-GPU decode throughput upper bound from memory bandwidth (tokens/s).This only considers weight reads and ignores KV cache, communication, and operator efficiency.The result is intended for order-of-magnitude checks, not as a product benchmark result."""return bandwidth_bps / (activated_params * bytes_per_param)def kv_bytes_per_token(layers: int, kv_heads: int, head_dim: int, dtype_bytes: int) -> int:"""KV-cache bytes per token, counting both K and V."""return 2 * layers * kv_heads * head_dim * dtype_bytesdef per_card_read_gb(dense_gb: float, activated_expert_gb: float, domain_size: int) -> float:"""Per-GPU bytes read per step.Dense components keep one replica per GPU, while activated expert components are evenly split across GPUs in the interconnect domain."""return dense_gb + activated_expert_gb / domain_sizeif __name__ == "__main__":print(f"{theoretical_decode_tps(37e9, 1, 8e12):.0f} tokens/s") # about 216print(f"{kv_bytes_per_token(61, 4, 128, 2) / 1024:.0f} KB/token") # about 122 KiB, roughly 125 KBfor n in (8, 32, 72):r = per_card_read_gb(4.11, 37.0, n)tps = theoretical_decode_tps(1, 1, 8e12 / (r * 1e9))print(f"N={n}: {r:.2f} GB/step -> {tps:.0f} tokens/s")
5. Benchmarking: Making Comparisons Reproducible and Attributable
1. Metric Selection
Before answering how to choose a benchmark, define the metrics:
- Throughput: output tokens generated per unit time. Specify whether input tokens are included.
- TTFT: time to first token. It determines how quickly an interactive application responds after the user presses Enter.
- TPOT: time per output token. It determines the smoothness of reading and voice playback.
- Goodput: throughput contributed only by requests that satisfy latency constraints. Counting only successful requests avoids the illusion of “high throughput while many requests time out.”
- Resource efficiency: throughput per GPU and throughput per megawatt, useful for cost and energy decisions.
2. Test Procedure
- Fix the configuration manifest from Section 1 and record hardware model, driver version, and framework version.
- Warm up: complete weight loading, graph capture, and cache warmup before starting measurement.
- Fix output length by setting a minimum token count, and disable sampling randomness.
- Send several hundred requests at each concurrency level, such as 1, 8, 32, 128, and 512. The sample size should be large enough to cover P99; at least 300 requests per level is recommended.
- Record TTFT, TPOT, end-to-end latency, and completed token count. Compute P50/P95/P99. Also record client and gateway CPU, memory, and connection counts.
- Plot the throughput-latency curve and identify the maximum concurrency that satisfies the SLO. The throughput at that point is the goodput.
3. Example Code
Dependencies: Python 3.9+ and standard library only.
Server requirements: an OpenAI-style /v1/chat/completions endpoint that supports stream=true and a min_tokens parameter for fixed output length. The SSE parser should tolerate empty lines, heartbeat frames, empty data: frames, and partial JSON. Set a per-request timeout of 300 seconds and retry twice.
# benchmark.pyfrom __future__ import annotationsimport jsonimport queueimport threadingimport timeimport urllib.errorimport urllib.requestfrom typing import AnyENDPOINT = "http://127.0.0.1:8000/v1/chat/completions" # Internal self-hosted inference serviceMODEL = "local-model"INPUT_LEN = 1024OUTPUT_LEN = 512WARMUP = 8TIMEOUT = 300RETRY = 2PROMPT = "Please reason step by step and answer:" + "context example." * (INPUT_LEN // 6)def one_request() -> dict[str, float]:"""Send one streaming request and return ttft, latency, and tokens.Exceptions are propagated to the caller."""body = json.dumps({"model": MODEL,"messages": [{"role": "user", "content": PROMPT}],"max_tokens": OUTPUT_LEN,"min_tokens": OUTPUT_LEN, # Fix output length to keep metrics comparable"temperature": 0.0, # Disable random sampling"stream": True,}).encode()req = urllib.request.Request(ENDPOINT,data=body,headers={"Content-Type": "application/json"},)t0 = time.perf_counter()first, tokens = None, 0with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:for raw in resp: # Read SSE line by lineline = raw.decode("utf-8", "replace").strip()if not line.startswith("data:"):continue # Tolerate empty lines and heartbeatspayload = line[len("data:"):].strip()if not payload:continue # Tolerate empty data framesif payload == "[DONE]":breaktry:chunk = json.loads(payload)except json.JSONDecodeError:continue # Ignore partial or malformed frameschoices = chunk.get("choices") or []if not choices:continuedelta = (choices[0].get("delta") or {}).get("content") or ""if not delta:continuenow = time.perf_counter()if first is None:first = nowtokens += 1last = time.perf_counter()if first is None or tokens == 0:raise RuntimeError("No content deltas received. Confirm that the server supports stream and min_tokens.")return {"ttft": first - t0, "latency": last - t0, "tokens": float(tokens)}def call_with_retry(attempts: int = RETRY) -> dict[str, float]:last_exc: Exception | None = Nonefor _ in range(attempts + 1):try:return one_request()except (urllib.error.URLError, TimeoutError, RuntimeError) as exc:last_exc = exctime.sleep(0.5)raise RuntimeError("Request failed after retries") from last_excdef percentile(values: list[float], p: float) -> float:"""Linear interpolation percentile.A sample size of at least 300 is recommended for a stable P99 estimate."""s = sorted(values)if len(s) == 1:return s[0]k = (len(s) - 1) * plo = int(k)hi = min(lo + 1, len(s) - 1)return s[lo] + (s[hi] - s[lo]) * (k - lo)def benchmark(concurrency: int, total: int = 512) -> dict[str, Any]:"""Fixed-concurrency benchmark.Workers pull tasks from a shared queue, and the timer stops only after total requests complete."""if total < concurrency:total = concurrencyfor _ in range(WARMUP): # Warm up weights, graph capture, and cache allocationcall_with_retry()work: queue.Queue[int] = queue.Queue()for i in range(total):work.put(i)results: queue.Queue[dict[str, float]] = queue.Queue()errors: queue.Queue[BaseException] = queue.Queue()def worker() -> None:while True:try:work.get_nowait()except queue.Empty:return # Exit when the queue is emptytry:results.put(call_with_retry())except BaseException as exc: # Report failures so the main thread does not hangerrors.put(exc)threads = [threading.Thread(target=worker, daemon=True) for _ in range(concurrency)]wall_start = time.perf_counter()for t in threads:t.start()for t in threads:t.join() # Stop the timer only after all workers finishwall = time.perf_counter() - wall_startcollected = []while not results.empty():collected.append(results.get())if not errors.empty():print(f"[warn] {errors.qsize()} failed requests were excluded from statistics")if len(collected) < max(30, total * 0.8):raise RuntimeError(f"Only {len(collected)} valid samples collected, which is insufficient for P99")ttft = [r["ttft"] for r in collected]tpot = [(r["latency"] - r["ttft"]) / max(r["tokens"] - 1, 1) for r in collected]return {"concurrency": concurrency,"samples": len(collected),"output_tps": sum(r["tokens"] for r in collected) / wall,"ttft_p50": round(percentile(ttft, 0.50), 3),"ttft_p99": round(percentile(ttft, 0.99), 3),"tpot_p99": round(percentile(tpot, 0.99), 4),}if __name__ == "__main__":for c in (1, 8, 32, 128, 512):print(json.dumps(benchmark(c), ensure_ascii=False))
Notes:
- The script targets internal self-hosted services. For production load testing, restrict source IP addresses and add access control. Do not expose an unauthenticated inference service directly to the public internet.
- Under high concurrency, a single-threaded client can itself become a bottleneck. For formal testing, use an asynchronous client or a multi-machine distributed load generator, and include client resource specifications in the report.
- Report output-token throughput and input-token throughput separately. Do not combine them into one number and compare it with someone else’s output-token throughput.
3.1 Attribution Experiment: Turning Off Optimizations One by One
When comparing across generations, design four groups:
- Old hardware + old software stack: control group.
- Old hardware + new software stack: group A.
- New hardware + old software stack: group B.
- New hardware + new software stack: experimental group.
Assume the control group is normalized to 1. Measured results are A = 2.5, B = 4, and experimental group = 58. Even if hardware and software contributions were completely independent, 2.5 × 4 = 10, which is still 5.8× short of 58. This indicates that some factors were not measured separately. Common sources include:
- Parallelism changes caused by a larger interconnect domain.
- Differences in the test model architecture.
- The baseline group did not enable key optimizations.
- Resource contention exists in one of the configurations.
Use a unified record table for item-by-item attribution:
| Variable category | Control | A | B | Experimental | Notes |
|---|---|---|---|---|---|
| Hardware, including model, per-GPU bandwidth, and memory | |||||
| Software, including framework, version, and scheduling features | |||||
| Parallelism, including interconnect-domain size, EP/TP/PP | |||||
| Model, including total parameters, activated parameters, and precision | |||||
| Workload, including input distribution, output length, and concurrency | |||||
| Measured output throughput, tokens/s | 1.0 | 2.5 | 4.0 | 58 |
Rule: change one cell at a time and rerun the benchmark. The final multiplier must be explainable by the changes recorded in the table.
3.2 Capacity-Planning Example
Assume a typical latency constraint, for illustration only: TTFT P99 ≤ 2 s and TPOT P99 ≤ 50 ms. A service instance has the following measured results. Note that throughput below counts output tokens only.
| Concurrency | Output throughput, tokens/s | TTFT P99 | TPOT P99 | Meets SLO? |
|---|---|---|---|---|
| 1 | about 200 | 0.3 s | 4.7 ms | Yes |
| 32 | about 6,000 | 0.9 s | 18 ms | Yes |
| 128 | about 15,000 | 1.6 s | 38 ms | Yes |
| 512 | about 18,000 | 4.2 s | 85 ms | No |
Note: At 512 concurrency, throughput is only about 20% higher than at 128 concurrency, while latency more than doubles. Therefore, the effective throughput satisfying the SLO for this instance is 15,000 output tokens/s, not the peak 18,000.
Assume peak business traffic is 300 QPS, average input is 2048 tokens, and output is 512 tokens. Capacity must be calculated under two separate units:
Decode side, output tokens: 300 × 512 = 153,600 output tokens/sPrefill side, input tokens: 300 × 2048 = 614,400 input tokens/s
- Decode resource pool: 153,600 ÷ 15,000 ≈ 10.24 → 11 instances. Add N+2 fault redundancy → 13 instances.
- Prefill resource pool: if one instance can support 40,000 input tokens/s under TTFT P99 ≤ 2 s, then 614,400 ÷ 40,000 ≈ 15.36 → 16 instances. Add redundancy → 18 instances.
The two numbers must not be mixed. Dividing 153,600 by 15,000 gives the number of decode instances. Dividing 614,400 by 15,000 is a unit error. Similarly, if peak 18,000 is mistakenly used instead of effective 15,000, the result becomes 8.5 → 9 instances, underestimating capacity by about 20% and causing widespread timeouts during peak periods.
6. Capacity Planning and Cost: Move From “Peak Throughput” to “Effective Unit Throughput”
- Plan with effective throughput. Estimate instance count using goodput under the SLO, and reserve capacity for failure redundancy and traffic fluctuation. Retest promptly when input or output length distributions change, such as when long-document summarization is introduced.
- Split resource pools by stage. Prefill consumes compute, while decode consumes bandwidth and memory capacity. After separation, each pool can be scaled independently according to its own bottleneck.
- Pay attention to energy efficiency. Rack-scale systems typically have significantly higher power consumption and cooling density than traditional servers. Tokens per megawatt and cost per thousand tokens should be included in decision-making.
- Optimize input structure. Shorten prompts, reuse fixed prefixes, and set context windows reasonably. Reducing unnecessary tokens is the most direct way to lower cost.
- Use general cloud capabilities when appropriate. When deploying on major public clouds, teams can combine neutral capabilities such as:
- Object storage for model weights and datasets.
- Container services for hosting inference instances and elastic scaling.
- Message queues for smoothing traffic spikes.
- Monitoring, alerting, and logging services for collecting runtime metrics and setting threshold alerts.
7. Continuous Observation After Launch
Benchmark results only represent a point-in-time state. After deployment, collect the following continuously:
- Latency metrics: P50/P95/P99 of TTFT and TPOT, with particular attention to tail latency.
- Service status: queue time, preemption and swap-out counts, goodput ratio.
- Cache effects: prefix-cache hit rate and KV-cache block utilization.
- Resource metrics: memory usage, interconnect-link bandwidth utilization, power, and temperature.
It is recommended to establish a shadow-traffic mechanism. New hardware, new software versions, or new models can first receive replicated production traffic in an equivalent environment. Results are recorded but not returned to users. After covering several peak and low-traffic periods, traffic can then be shifted. At the same time, configure threshold alerts on key metrics to prevent performance regressions caused by version upgrades.
8. Frequently Asked Questions
Does “tens of times single-GPU throughput gain” mean that each GPU became tens of times faster?
Not necessarily. This metric is usually aggregate throughput within an interconnect domain divided by GPU count. It depends on interconnect-domain size, parallelism strategy, and model architecture. Under the model in this article, topology typically contributes a few times, and there is an upper bound determined by dense replicas.
What should I do if throughput drops sharply as context length increases?
This happens because KV-cache reads and writes replace weight reads as the bottleneck. Options include KV-cache quantization, improving bandwidth utilization, rate-limiting long-context requests separately, or adopting architectures with lower cache complexity.
Does speculative decoding always improve speed?
Not always. The benefit depends on acceptance rate and draft overhead. A/B testing with real business corpora is recommended.
After a hardware upgrade, throughput improvement is lower than expected. What should I check first?
Check in this order:
- Parallelism strategy and load balancing.
- Communication bottlenecks, especially interconnect bandwidth utilization.
- KV-cache capacity limits on concurrency.
- Whether the client or gateway has become the bottleneck.
Common benchmark mistakes:
- Starting measurement without warmup.
- Failing to fix output length, causing metric drift.
- Failing to disable sampling randomness.
- The client or gateway becomes a bottleneck, but its resource usage is not recorded.
- Sample size is insufficient to support P99.
- Mixing input-token and output-token metrics.
- The baseline group did not enable key optimizations.
- Reporting only peak throughput and not goodput.
- The report does not state hardware model, driver version, and framework version.
Conclusion
Behind a claim such as “tens of times single-GPU throughput gain” are the combined effects of model architecture, parallelism topology, interconnect capability, and software scheduling, rather than a linear mapping of a single hardware parameter.
A more reliable approach is to:
- Align benchmark definitions, including a precise definition of single-GPU throughput.
- Use roofline analysis and ablation experiments to attribute multipliers to individual variables.
- Plan capacity by effective throughput and separate input and output units under SLO constraints.
- Continue observing and validating with canary or shadow traffic after launch.
Only in this way can a multiplier on paper become measurable improvements in responsiveness, stability, and cost efficiency.
