LLM Inference Throughput Benchmarking: Measurement Scope, Bottlenecks, and Hardware Differences
This article explains why large model inference throughput numbers can be misleading when measurement scope, workload shape, numerical precision, and software stacks are not aligned. It covers how output tokens/s is measured, how fixed-concurrency benchmarking works, why short-input and long-input workloads expose different bottlenecks, and how compute capability and memory bandwidth relate to real inference performance. It also summarizes single-card multi-concurrency examples and provides practical recommendations for reproducible benchmarking, accelerator selection, and capacity planning.
1. Start with the measurement scope, then compare numbers
When teams deploy large models in production, they often need to choose among accelerator cards with different architectures, numerical precision modes, and inference-serving frameworks. One common mistake is to place throughput numbers measured at different times, in different environments, and under different software stacks side by side, then conclude that “Card A is X times faster than Card B.” This kind of comparison looks intuitive, but it is easy to introduce systematic bias.
output tokens/s, or output tokens per second, is one of the most commonly used metrics for online inference throughput. Under sustained request pressure, it measures the total number of tokens actually generated by the system per unit of time. To interpret this number correctly, three questions must be answered first:
- How was it measured?
- Under what workload shape was it measured?
- Is it comparable across different tests?
Any cross-card comparison that skips these three steps is unlikely to be reliable.
It is also important to emphasize that inference data collected across different accelerator cards, precisions, and software stacks should be understood as a set of observations under specific conditions, not as a controlled experiment in the same environment. If the environment is not controlled, the result can only be treated as a trend reference, not as a definitive conclusion.
2. How throughput is measured: fixed concurrency and two workload shapes
A reference-quality throughput dataset usually comes from a unified inference-serving interface, such as an online endpoint compatible with common industry API conventions. The service is kept under fixed concurrency and continuously loaded. The server then counts the output tokens of all completed requests and the elapsed time, converting them into output tokens/s.
To cover different inference scenarios, benchmarkers usually prepare two workload shapes:
- Short input, short output, for example
512 input / 512 output. This workload is dominated by the generation phase and is more decode-oriented, meaning tokens are generated one by one. - Long input, short output, for example
8192 input / 1024 output. This workload is dominated by the prefill phase and is more prefill-oriented, meaning a long prompt is processed in one pass.
The following example shows a simplified fixed-concurrency throughput collection loop:
# Example: throughput collection under a fixed-concurrency stream (pseudocode)import timefrom concurrent.futures import ThreadPoolExecutordef one_request(prompt, out_len):t0 = time.perf_counter()resp = client.generate( # common inference-serving interfacemodel="served-model",prompt=prompt,max_tokens=out_len,)dt = time.perf_counter() - t0return resp.output_tokens, dtdef measure(concurrency, prompts, out_len, warmup=50, window_sec=60):for _ in range(warmup): # warmup to exclude cold-start effectsone_request(prompts[0], out_len)total_tokens, start = 0, time.perf_counter()with ThreadPoolExecutor(max_workers=concurrency) as pool:futures = [pool.submit(one_request, prompts[i % len(prompts)], out_len)for i in range(concurrency * 64)]for f in futures:tokens, _ = f.result()total_tokens += tokensif time.perf_counter() - start >= window_sec:breakelapsed = time.perf_counter() - startreturn total_tokens / elapsed # output tokens/s
Several details that are easy to overlook determine whether results are comparable:
- Whether warmup was performed
- Whether concurrency was fixed
- Whether the input/output length distribution was consistent
- Whether the measurement window included cold-start effects
- Whether time to first token (TTFT) was separated from steady-state throughput
If these conditions are inconsistent, the resulting numbers lose comparability.
3. Environment differences: why these numbers are not same-environment controlled experiments
Even when the measurement method is the same, the test environments themselves often differ significantly: server platforms may differ, power limit settings may differ, drivers and GPU compute runtimes may differ, and inference-serving frameworks and container images may also be on different versions.
For example, in one set of public data, Card A and Card C were tested at an actual power limit of 450W, while Card B was tested at 600W, even though all three hardware designs had a rated power limit of 600W. Different power limits mean that the same card can produce different throughput under different settings.
Software-stack differences are equally important. Some tests use newer drivers and compute runtimes, while others use older versions. Different accelerator architectures may also implement low-precision paths such as FP4 differently. Therefore, when all these numbers are placed in the same table, the reasonable approach is to treat them as snapshots under different conditions, useful for observing trends and orders of magnitude, rather than for strict ranking.
4. Compute and memory bandwidth: two different capability dimensions
To understand throughput differences, it is important to distinguish two fundamental capabilities:
- Compute capability, usually measured as achievable matrix-multiply FLOPS, often expressed as MAMF.
- Memory access bandwidth, usually measured by a specified memory-access kernel.
A single-card example is shown below:
| Capability | Card A | Card B | Card C |
|---|---|---|---|
| Measured BF16 compute (TFLOPS) | 652 | 419 | 580 |
| Measured FP8 compute (TFLOPS) | 1,205 | 820 | 1,120 |
| Memory read bandwidth (GB/s) | 4,472 | 1,540 | — |
From the bandwidth numbers, Card A’s read bandwidth is about 2.90 times that of Card B, based on the calculated ratio 4,472 ÷ 1,540. This ratio is meaningful only under the same bandwidth kernel.
Two points should be emphasized:
- TFLOPS values at different precisions should not be directly ranked against each other, because low-precision paths are implemented and accelerated differently across architectures.
- Both compute and bandwidth represent only one side of a capability ceiling. Neither equals the actual online inference throughput of a model.
5. Decode and prefill: where is the bottleneck?
LLM inference is usually divided into two phases.
Prefill processes the entire input prompt in parallel in one pass. It is compute-intensive and more likely to reflect differences in compute capability and attention-path efficiency.
Decode generates one token per step. It repeatedly reads model weights and KV cache, and is usually constrained by memory bandwidth and batching efficiency.
Therefore:
- Short-input workloads, which are more decode-oriented, often reflect the effects of memory traffic and concurrent batching.
- Long-input workloads, which are more prefill-oriented, are more likely to expose differences in compute capability and attention implementations.
However, the final bottleneck depends on the model architecture, batch size, and sequence length. It cannot be generalized without qualification.
6. Single-card multi-concurrency results: rankings change with model, precision, and workload
Meaningful conclusions require comparisons under the same model, same precision, and same workload shape. The following public examples use single-card results at 128 concurrency, measured in output tokens/s. “Short” means 512 input / 512 output, and “long” means 8192 input / 1024 output.
A 30B-class MoE model in FP8:
- Short input: Card A 7,459, Card B 3,995. Card A leads by about 87%.
- Long input: Card A 1,820, Card B 823. Card A is about 2.21 times faster.
- The same cards in BF16:
- Short input: 5,240 / 2,881.
- Long input: 1,592 / 588.
- This shows that the acceleration from FP8 versus BF16 is not fixed. It can be significant under some workloads and limited under others.
An 8B-class dense model in FP8:
- Short input: Card A 11,054, Card B 8,073, Card C 11,045. Card A and Card C are very close.
- Long input: 2,045 / 1,481 / 1,211. The ranking changes.
A 120B-class model using MXFP4 weights:
- Short input: Card C 5,057, Card A 3,376, Card B 3,064. Card C leads.
- Long input: 2,101 / 1,825 / 1,169.
These three groups of results support one core fact: there is no universal “strongest card” ranking. Which card is faster depends on model size and structure, numerical precision, and whether the workload is more decode-oriented or more prefill-oriented. Using the result of a single scenario to summarize overall capability is a typical benchmarking trap.
7. Recommendations for reproducible and explainable inference benchmarking
To make inference benchmarking useful for selection decisions, the following principles are recommended:
- Fix and disclose the measurement scope. Concurrency, input/output length distributions, warmup count, measurement window, and whether TTFT is included must all be clearly recorded.
- Record the full environment. Server platform, power limit, driver and compute runtime, inference framework, and container image version are all necessary. Without them, the numbers cannot be properly attributed.
- Do not directly rank across precisions or models. Different precisions are not comparable, and models with different structures cannot be directly converted into one another.
- Cover multiple workload shapes. At minimum, include both decode-oriented and prefill-oriented workloads. Also cover multiple concurrency levels and observe how throughput changes with concurrency.
- Add metrics closer to user experience. In addition to total throughput, report TTFT, time per output token (TPOT), and goodput under SLA constraints.
When cloud resources are used for this kind of benchmarking, the process may involve general capabilities such as a container platform for inference images, object storage for datasets and results, and monitoring/alerting/logging services for collecting latency, throughput, and resource utilization. These capabilities help make the benchmarking process reproducible and traceable.
8. Conclusion
The value of inference performance benchmarking is not to produce a simple “who is fastest” answer. It is to build an analytical framework with clear measurement scope, controlled variables, and explainable conclusions. By putting output tokens/s back into its measurement context, separating compute from memory bandwidth, distinguishing decode from prefill bottlenecks, and recognizing the comparability limits across environments, precisions, and models, teams can avoid being misled by a single number and make more robust accelerator-selection and capacity-planning decisions.
