GPU Inference Performance Evaluation and Selection: Memory Bandwidth, Throughput, and Benchmark Scripts

    Peak compute numbers on a spec sheet rarely predict real inference throughput. This article lays out a reusable method for evaluating GPU inference performance, using two representative accelerator classes as reference points. It covers what measured matrix-multiplication FLOPS (MAMF) and memory bandwidth can and cannot tell you, how end-to-end throughput shifts with load profile and numerical precision at a fixed concurrency of 128, how to build a reproducible benchmark in three steps (including a ready-to-run Python script), and which mistakes to avoid when interpreting the results. All models and numbers are illustrative and show the analysis path, not any specific product's specifications.

    1. Introduction: Peak Compute Is Not Real Inference Throughput

    When evaluating GPUs for inference, many teams start with the peak compute, memory capacity, and memory bandwidth quoted in a whitepaper, then reason: “This card should run this fast.” After deployment, end-to-end throughput often falls short. The reason is straightforward: online inference is not a single matrix-multiplication benchmark. It is the combined result of prefill, decode, attention computation, KV cache reads and writes, batch scheduling, request queuing, and software stack maturity.

    Using two accelerator classes as reference points, this article abstracts a reusable evaluation methodology for inference performance. Every model and number here is illustrative, meant to demonstrate the analysis path rather than represent any specific product metric. The article answers four questions:

    1. What measured compute and memory bandwidth can and cannot tell you;
    2. How end-to-end throughput changes with load profile and numerical precision at a fixed concurrency;
    3. How to build a reproducible inference benchmark;
    4. Which pitfalls are common when reading the results.

    2. Test Setup, Metrics, and Controlled Variables

    2.1 Accelerators and Inference Interface

    The reference testing covers two accelerator classes:

    • Card A: A data-center-class PCIe add-in card with high-bandwidth memory, tested with a 450 W power limit;
    • Card B: A workstation/server-class high-power add-in card with larger memory capacity, tested with a 600 W power limit.

    All tests were issued through the HTTP interface of an inference service. That interface uses a messages conversation structure with SSE streaming responses and has become the de facto standard shape for inference services. Mainstream inference frameworks, gateways, and assorted coding assistants and chat clients generally support it.

    A unified interface has two benefits:

    • It abstracts away differences in the underlying runtime and stays close to the real application call path;
    • It keeps the measurement consistent across cards: a fixed concurrency, with the total number of output tokens produced by all requests per unit time.

    2.2 Throughput, Latency, and Goodput

    Measuring inference performance by “output tokens/s” alone is incomplete. A practical evaluation looks at three metrics at once:

    • Throughput: total output tokens produced by all requests per unit time; reflects hardware utilization and cost per unit of compute;
    • Time to first token (TTFT): time from request submission to the first received token, dominated by queueing wait plus prefill time;
    • Time per output token (TPOT, also called ITL): the interval between consecutive tokens after the first one, dominated by single-step decode time; this is what determines perceived “streaming speed.”

    What online services actually care about is goodput: the request throughput that satisfies latency constraints. For example, counting successful requests per second under the constraint “P99 TTFT no more than 2 seconds and TPOT no more than 50 milliseconds.” The “output tokens/s” values in the tables below were measured at a fixed concurrency, where the server stays continuously in full-batch operation, so they can be read as an approximation of the throughput ceiling.

    To compute goodput in a script, add per-request TTFT, TPOT, and success status to the existing results, then calculate:

    1. goodput = requests meeting the latency constraints / total test duration

    2.3 Load Profiles: What Each Preset Exercises

    Two preset workloads were tested:

    • Short input, short output: 512 input tokens, 512 output tokens (“short load”). Every decode step reads all weights, so this leans toward decode;
    • Long input, short output: 8192 input tokens, 1024 output tokens (“long load”). Roughly eight times longer prompt to process before the first token, so this leans toward prefill.

    Together these cover the two typical classes of online serving: the former resembles code completion and short question answering, the latter resembles long-document understanding and code base question answering. The two workloads place different demands on hardware and cannot substitute for each other.

    2.4 Variable Control: Aligned Environments Decide Whether Conclusions Hold

    For a cross-card comparison to be valid, variables must be controlled. At minimum:

    • Test method: requests issued at a fixed concurrency through the unified interface, 128 in these examples, with total output tokens across all completed requests divided by total elapsed time;
    • Power limit: Card A and Card B were tested under different power limits, so when comparing absolute values, performance per watt should be considered alongside;
    • Server platform: CPU, memory capacity, PCIe generation, and lane bifurcation all affect scheduling and data movement;
    • Software stack versions: the GPU compute stacks and inference service container versions used on each card were not identical. The conclusions are therefore better understood as “comparison results under a specific hardware and software combination” rather than as conclusions about hardware generations;
    • Measurement conventions: request rate, sampling timing, input/output length distribution, and whether prefix caching was enabled should all be stated in the report.

    3. Compute and Memory Bandwidth: Two Different Capability Ceilings

    3.1 What Measured Matrix-Multiplication FLOPS (MAMF) Means

    Measured attainable matrix-multiplication FLOPS (MAMF below) refers to the “achievable FLOPS” measured by a matrix-multiplication microbenchmark at a specific matrix size and a specific numerical precision. It is closer to reality than the vendor’s quoted peak, but still carries two caveats:

    • Results depend on matrix size, memory layout, and the compute library and driver versions;
    • Different precisions should not be ranked against each other directly. FP8 and BF16 differ in representable range, rounding behavior, and scaling strategy, and low-precision formats such as FP4 also vary between vendors in implementation details like scaling granularity and dequantization.

    Compute numbers across precisions are therefore only suitable for comparing relative relationships within the same implementation.

    The MAMF values from these reference tests: Card A reached 652 TFLOPS in BF16 and 1,205 TFLOPS in FP8; Card B reached 419 TFLOPS in BF16 and 820 TFLOPS in FP8. Card A led clearly in both precisions, with FP8 at roughly 1.85x BF16, consistent with the common pattern of “halve the bytes per element, roughly double the throughput.”

    3.2 The Boundaries of Measured Memory Bandwidth

    Memory bandwidth here means the effective bandwidth measured with a dedicated memory-access microbenchmark. Card A measured 4,472 GB/s and Card B 1,540 GB/s, so the former is about 2.90x the latter. That ratio comes solely from dividing 4,472 by 1,540 and applies only to this bandwidth benchmark; model inference involves many access patterns, including random access, paged KV caches, and collective communication, so real-world ratios usually differ.

    Memory type also matters: HBM delivers high bandwidth through a wide bus and stacked structure, while high-bandwidth GDDR-class memory offers high per-pin data rates but often lower total bandwidth at the same power. Card B offers more memory capacity, which allows larger models or longer contexts — a dimension the bandwidth number cannot express.

    3.3 Why Microbenchmarks and Online Throughput Are Not Equivalent

    MAMF and bandwidth are both “component-level ceilings.” Online inference throughput is also shaped by model architecture, parallelism strategy, batch scheduling, and software stack maturity. A simplified model below illustrates how the two act. It applies only to single-GPU execution with BF16 weights, and ignores attention computation, collective communication, and scheduling overhead.

    Let the model have P parameters, run on one GPU with BF16 weights at 2 bytes per parameter; batch size B; s the number of KV cache bytes added per new token per step. Each generated step involves:

    • reading about 2P bytes of weights;
    • writing and reading about s bytes of KV cache for the new token;
    • producing B tokens.

    So:

    1. single-step time T ≈ (2P + B·s) / effective memory bandwidth
    2. output throughput X ≈ B / T = B × effective memory bandwidth / (2P + B·s)

    If the model is loaded in FP8, weight reads drop to roughly 1P bytes, but scaling factors, dequantization overhead, kernel implementation, and numerical stability all come into play. FP8 therefore does not simply halve weight bytes, and the actual gain must be validated under the target workload.

    Three implications follow from this equation:

    1. At batch size 1, the throughput ceiling is roughly “memory bandwidth ÷ weight bytes,” and compute sits nearly idle;
    2. As batch size grows, the weight-read cost is amortized across more requests, but per-step KV cache traffic grows linearly with B;
    3. Decode is bandwidth-bound, while prefill is closer to compute-bound, because the same weight element is reused across many tokens in the sequence, giving a high compute-to-access ratio.

    This is precisely the rationale for the two load presets. Card A leads Card B by about 56% in BF16 compute (652 divided by 419) and about 190% in bandwidth (4,472 divided by 1,540). If Card A’s advantage differs noticeably between the two workloads, that provides end-to-end support for the reasoning above.

    4. Single-GPU Results at 128 Concurrency: Short and Long Inputs

    The reference tests ran at 128 concurrency, with the following results. Values are illustrative, in output tokens/s.

    Model and precision Short load (512/512) Card A Short load (512/512) Card B Long load (8192/1024) Card A Long load (8192/1024) Card B
    Illustrative model M1 · FP8 7,459 3,995 1,820 823
    Illustrative model M1 · BF16 5,240 2,881 1,592 588

    M1 stands for a mixture-of-experts (MoE) coding model with roughly 30 billion total parameters and about 3 billion activated per token.

    4.1 Bandwidth Gaps Widen with Long Inputs

    • FP8: Card A led by about 87% on the short load and about 2.21x on the long load;
    • BF16: about 82% and about 2.71x respectively.

    Two likely reasons the lead is larger on the long load:

    • An 8K-level prompt brings heavy attention computation and KV cache traffic, where the higher-bandwidth card gains more;
    • With longer inputs, per-step compute grows, and the efficiency difference between the FP8 and BF16 paths on large matrix multiplications is amplified further.

    This observation is one interpretation of the data, not a confirmed cause. Verifying the exact reason requires emitting prefill and decode timings separately, or profiling at the operator level to see whether the bottleneck sits in attention, matrix multiplication, KV cache, or scheduling.

    4.2 The FP8-to-BF16 Speedup Is Not a Constant

    On the same card, the FP8 gain over BF16 was:

    • Card A: about 1.42x on the short load (7,459 divided by 5,240) and about 1.14x on the long load (1,820 divided by 1,592);
    • Card B: about 1.39x on the short load (3,995 divided by 2,881) and about 1.40x on the long load (823 divided by 588).

    Card A’s gain narrowed markedly as the input grew, while Card B’s stayed roughly flat. A reasonable guess: on the short load, Card A is mainly bandwidth-constrained, so halving weight bytes yields nearly linear gains; on the long load the bottleneck shifts toward attention and scheduling, diluting the benefit of quantization. Because vendors implement low precision differently, the assumption that “a given precision always delivers the same gain” does not hold, and any precision conclusion should be measured under the target workload.

    4.3 Other Models and Precision Formats Need Separate Evaluation

    The reference testing also covered:

    • Dense models of different parameter counts, and large-scale MoE models;
    • Large models under multi-GPU tensor parallelism;
    • MXFP4 (a 4-bit floating-point format based on block scaling);
    • Multimodal models.

    Because a MoE model activates far fewer parameters than it holds in total, decode overhead comes more from expert routing and collective communication; for large models, memory capacity directly determines whether single-GPU deployment is possible at all. M1’s results therefore cannot be extrapolated linearly to every model.

    5. A Reproducible Benchmark in Three Steps

    5.1 Step 1: Fix the Input Length

    A tokenizer splits text into tokens, and the “512” and “8192” in public benchmarks are token counts. In English, one token usually corresponds to 3–4 characters, so building prompts by character count introduces meaningful error. Recommended practice:

    • Confirm the actual length through the inference service’s tokenize endpoint;
    • Or compute the length locally with the server’s tokenizer;
    • Use the same text and the same tokenizer for every accelerator being compared.

    The script below builds prompts with a local tokenizer and counts output tokens using the target tokenizer or the server-side usage statistics.

    5.2 Step 2: Fix Concurrency and Count Output Tokens from the Stream

    The script depends on the Python standard library and a tokenizer library; transformers is used in the example. Install it with:

    1. pip install transformers

    Example run:

    1. python bench.py \
    2. --url http://127.0.0.1:8080/v1/chat/completions \
    3. --model demo-model \
    4. --tokenizer /path/to/tokenizer \
    5. --concurrency 128 \
    6. --input-tokens 512 \
    7. --max-tokens 512 \
    8. --requests 1024

    The full script:

    1. # bench.py — fixed-concurrency streaming load test; reports output tokens/s and TTFT
    2. # Requires: pip install transformers
    3. import argparse
    4. import json
    5. import time
    6. import urllib.request
    7. from concurrent.futures import ThreadPoolExecutor
    8. from transformers import AutoTokenizer
    9. FILLER = "The quick brown fox jumps over the lazy dog. "
    10. def build_prompt(tokenizer, n_tokens: int) -> str:
    11. """Build a prompt of the requested token count and return the string to send."""
    12. text = ""
    13. while True:
    14. ids = tokenizer.encode(text, add_special_tokens=False)
    15. if len(ids) >= n_tokens:
    16. break
    17. text += FILLER
    18. ids = tokenizer.encode(text, add_special_tokens=False)[:n_tokens]
    19. return tokenizer.decode(ids, skip_special_tokens=False)
    20. def run_one(url: str, model: str, prompt: str, max_tokens: int,
    21. tokenizer, result: list):
    22. """Issue one streaming request and record TTFT plus the actual output token count."""
    23. payload = json.dumps({
    24. "model": model,
    25. "messages": [{"role": "user", "content": prompt}],
    26. "max_tokens": max_tokens,
    27. "stream": True,
    28. "temperature": 0.0,
    29. "stream_options": {"include_usage": True}
    30. }).encode()
    31. req = urllib.request.Request(
    32. url,
    33. data=payload,
    34. headers={"Content-Type": "application/json"}
    35. )
    36. start = time.perf_counter()
    37. ttft = None
    38. usage_tokens = None
    39. delta_tokens = 0
    40. with urllib.request.urlopen(req, timeout=600) as resp:
    41. for raw in resp: # SSE: one "data: {...}" object per line
    42. line = raw.decode("utf-8", errors="ignore").strip()
    43. if not line.startswith("data:"):
    44. continue
    45. data = line[5:].strip()
    46. if data == "[DONE]":
    47. break
    48. chunk = json.loads(data)
    49. usage = chunk.get("usage")
    50. if usage and usage.get("completion_tokens") is not None:
    51. usage_tokens = int(usage["completion_tokens"])
    52. choices = chunk.get("choices") or []
    53. if not choices:
    54. continue
    55. delta = choices[0].get("delta") or {}
    56. content = delta.get("content") or ""
    57. if ttft is None and content:
    58. ttft = time.perf_counter() - start
    59. # If the server does not return usage, count the deltas with the target
    60. # tokenizer. This is more reliable than approximating by character count;
    61. # for exact numbers, make sure the server returns usage.
    62. if usage_tokens is None and content:
    63. delta_tokens += len(
    64. tokenizer.encode(content, add_special_tokens=False)
    65. )
    66. if choices[0].get("finish_reason"):
    67. break
    68. tokens = usage_tokens if usage_tokens is not None else delta_tokens
    69. result.append({"ttft": ttft, "tokens": tokens})
    70. def main():
    71. ap = argparse.ArgumentParser()
    72. ap.add_argument(
    73. "--url",
    74. default="http://127.0.0.1:8080/v1/chat/completions"
    75. )
    76. ap.add_argument("--model", required=True)
    77. ap.add_argument(
    78. "--tokenizer",
    79. required=True,
    80. help="Local tokenizer path or model name, used to build the prompt and count tokens"
    81. )
    82. ap.add_argument("--concurrency", type=int, default=128)
    83. ap.add_argument("--input-tokens", type=int, default=512)
    84. ap.add_argument("--max-tokens", type=int, default=512)
    85. ap.add_argument("--requests", type=int, default=1024)
    86. args = ap.parse_args()
    87. tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
    88. prompt = build_prompt(tokenizer, args.input_tokens)
    89. results = []
    90. start = time.perf_counter()
    91. # Thread pool size equals the concurrency, which holds the load constant.
    92. with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
    93. futures = []
    94. for _ in range(args.requests):
    95. futures.append(
    96. pool.submit(
    97. run_one,
    98. args.url,
    99. args.model,
    100. prompt,
    101. args.max_tokens,
    102. tokenizer,
    103. results
    104. )
    105. )
    106. for f in futures:
    107. f.result()
    108. elapsed = time.perf_counter() - start
    109. valid = [r for r in results if r["tokens"] is not None and r["tokens"] > 0]
    110. if not valid:
    111. raise SystemExit("No valid output token counts")
    112. total = sum(r["tokens"] for r in valid)
    113. print(f"output tokens/s: {total / elapsed:.1f}")
    114. ttfts = sorted(r["ttft"] for r in valid if r["ttft"] is not None)
    115. if ttfts:
    116. p50 = ttfts[len(ttfts) // 2]
    117. p99 = ttfts[min(len(ttfts) - 1, int(len(ttfts) * 0.99))]
    118. print(f"TTFT p50={p50:.3f}s p99={p99:.3f}s")
    119. else:
    120. print("TTFT: no valid data")
    121. if __name__ == "__main__":
    122. main()

    The script prefers the server-reported usage.completion_tokens for output token counts, and falls back to counting the streamed deltas with the target tokenizer if the server does not return usage. Compared with counting characters, this is much closer to real token throughput. Approximating by character count instead will deviate significantly from the actual token count and must be noted in the report.

    5.3 Step 3: Record the Environment and Configuration

    Every run should record:

    • The accelerator’s power limit, driver and compute stack versions, and inference service container version;
    • Tensor parallelism degree, KV cache precision, maximum batch size, and whether chunked prefill and prefix caching are enabled;
    • Tokenizer version and random seed;
    • Whether the service automatically applies a chat template, and whether the actual input token count matches what was built locally.

    If needed, a “thinking interval” drawn from a Poisson distribution can be inserted after each request to simulate a realistic user arrival pattern.

    6. Common Pitfalls and an Interpretation Checklist

    1. Treating microbenchmarks as end-to-end performance: MAMF and bandwidth are ceilings only; MoE routing, attention, and scheduling overhead all push real throughput away from the theoretical value.
    2. Comparing compute across precisions directly: FP4, FP8, and BF16 differ in representation and scaling strategy, and implementations differ between vendors, so comparisons are only valid within one definition.
    3. Assuming FP8 always delivers a multiple: Card A gained only about 1.14x from FP8 over BF16 on the long load, which shows that quantization gains depend heavily on load profile.
    4. Ignoring the power convention: Card A was tested at 450 W and Card B at 600 W. Using the short-load throughput figures, Card A reaches about 16.6 tokens/s/W and Card B about 6.7 tokens/s/W. This metric counts GPU power only, excludes whole-system overhead, and is indicative at best.
    5. Ignoring the coverage boundaries: the example data covers a single model class at 128 concurrency on one GPU and says nothing directly about multi-GPU parallelism, larger models, multimodal workloads, or higher concurrency.
    6. Using average latency instead of percentiles: a small number of long prompts can push P99 TTFT up sharply, so capacity planning should rely on percentiles.
    7. Not fixing the input length: building prompts by characters rather than tokens makes cross-card comparisons meaningless.

    7. Conclusion

    An accelerator’s inference performance is not determined by any single spec. MAMF reflects compute capability, memory bandwidth reflects access capability, and end-to-end throughput is decided together by load profile, numerical precision, batching and scheduling, and software stack maturity.

    Teams that need GPU capacity quickly can provision GPU instances or bare-metal servers on demand from major cloud GPU platforms, then apply the method in this article to measure results under their target model and typical input/output distribution. A reliable selection decision rests on real business metrics, a fixed measurement convention, reproducible scripts, and percentile latency — not on paper specifications.