High-Compute GPU Inference Performance Comparison: Metrics, Boundaries, and Selection Criteria
This article compares inference performance across three high-compute GPU accelerators and explains why ranking them by a single output tokens/s number can be misleading. It examines testing consistency, workload shape, software stack, and power settings across the BF16, FP8, and MXFP4 precision paths under fixed concurrency. It introduces two kernel-level metrics—matrix-achievable FLOPS (MAMF) and memory bandwidth—then reports single-card throughput at 128 concurrency for short (512/512) and long (8192/1024) loads, shows how load shape can flip rankings, and provides a reproducible load-testing method, a selection framework, and an FAQ. It closes with clear boundaries on how far the data can be generalized.
1. Conclusions first: high-compute GPU inference comparison is not just a throughput ranking
In a high-compute GPU inference comparison, looking at a single output tokens/s figure in isolation is easy to misread. What actually matters is whether the test methodology is consistent, whether the workload shape matches your use case, and whether the software stack and power settings are comparable. This article focuses on online inference testing at fixed concurrency, re-examines how the BF16, FP8, and MXFP4 precision paths perform, and provides a reproducible load-testing method along with selection boundaries.
For the inference capability of three high-compute GPU accelerators, the following five conclusions can be drawn:
- Decode-dominated short loads: On a 30B-class MoE code model, the high-bandwidth-memory card (referred to below as Card A) delivered roughly 87% higher FP8 output throughput than Card B, and the gap widened to about 2.21x under long-input loads. This suggests that decode and memory-access efficiency matter more for short loads, while bandwidth and the prefill path have a larger impact for long loads.
- Dense-model rankings shift with load shape: On an 8B dense model, Card A and a new-generation AI accelerator (referred to below as Card C) had nearly identical FP8 short-input throughput, a difference under 1%. Under long inputs, Card A led Card C by about 69%, and the ranking reversed with load shape.
- Low-precision paths need separate validation: In a supplementary test with an MXFP4-quantized 120B-class MoE model, Card C’s short-input throughput was about 50% higher than Card A’s, but its long-input lead narrowed to about 15%. Because the 120B-model measurement caliber needs a separate explanation, this data is better treated as supplementary reference rather than being folded directly into a single-card ranking.
- Specs are not the same as measured results: Matrix-achievable FLOPS (MAMF) and memory bandwidth define the performance ceiling, but the measured gap is also affected by kernel implementation, quantization-format support, power walls, scheduling policy, sampling parameters, and software stack versions.
- The data is “an empirical reference in a specific environment”: The test environments were not fully identical, so these figures are not a same-environment controlled experiment that can be reproduced as-is, and they cannot be used directly as an absolute performance ranking.
1.1 Test subjects and labeling
This article keeps the hardware scope of the original evaluation and labels the cards consistently as follows, without pointing to any specific vendor:
| Label | Hardware positioning | Typical features |
|---|---|---|
| Card A | Data-center-class add-in card with high-bandwidth memory (PCIe form factor) | High-capacity, high-bandwidth memory, aimed at large-model serving |
| Card B | General-purpose graphics accelerator for professional workstations and servers (supports FP8 and MXFP4) | Large memory capacity, relatively high compute, higher per-card power cap |
| Card C | New-generation AI accelerator card | Single-card and multi-card form factors, covers low-precision compute |
The three belong to different architecture generations, and their software ecosystems, quantization support, and kernel maturity all differ. As a result, test conclusions must be read together with “hardware + software + power settings” rather than by a single peak spec.
1.2 Test method and two load shapes
The original evaluation sent sustained load to the inference service at fixed concurrency over an OpenAI-compatible HTTP interface and aggregated total output tokens/s across all requests. Two input/output length combinations were tested:
- Short load: 512 input / 512 output. The compute in the output phase is roughly half of the end-to-end total, and decode behavior dominates.
- Long load: 8192 input / 1024 output. Prefill must process 8,192 input tokens—more than 8x the work of the output tokens—which better exposes differences in compute, the attention path, and memory-access strategy.
1.3 Software, power, and data caliber
The raw data comes from a limited number of tests in a specific environment. For verifiability, this article keeps the software environment from the original records: the Card A vs. Card B comparison used CUDA 13.4.1 and the corresponding framework container; most single-card models on Card C ran on a combination of ROCm 10.1.0 and inference framework 0.32.0, with some records for large models using ROCm 7.2.3. If the original records lack the test date, image tag, model weight version, or quantization configuration, this article does not fabricate those fields, and the relevant conclusions should be rechecked against the original caliber.
During testing, Card A and Card C were set to 450W and Card B to 600W; the maximum rated power of all three is 600W. This leads to three points:
- Card A and Card C were not run at their rated power ceiling, so their results are relatively conservative.
- Cross-generation, cross-vendor performance multiples only reflect the gap “under a specific environment and power setting.”
- Vendor-claimed peaks are built on specific shapes and ideal conditions, and differ systematically from real online-inference behavior.
On the 120B-MXFP4 data, a special note is needed: Table 2 only includes items confirmed as single-card, 128-concurrency measurements. Because 120B-class models are large, the original records may differ in whether they used a single card, multiple cards on a single machine, or different resource groups. Therefore, this article lists the 120B-MXFP4 data separately in a supplementary table and does not fold it directly into single-card rankings. If it is later confirmed to be a strict single-card caliber, it can be merged into a same-caliber comparison; if it is a multi-card caliber, it serves only as a trend reference.
2. Memory bandwidth and compute: the two underlying paths that determine throughput
2.1 Two kernel-level metrics
Decode-phase throughput is intuitively limited by weight-read speed, so memory bandwidth is often treated as the deciding factor for decode—but that intuition needs to be corrected by the definition of the metric itself. This article uses two kernel-level metrics to describe the base capability of each card:
- MAMF (Matrix-Achievable FLOPS): achievable FLOPS measured on a controlled matrix-multiply kernel, reflecting the compute that can actually be used under a “specific precision, specific shape.”
- Memory bandwidth: bandwidth measured with a specified memory-access kernel, reflecting the actual efficiency of the access path.
The common point of the two is that both are closer to engineering reality than to a paper peak. Their shared boundaries are:
- Both only measure a “single operator” and exclude end-to-end overhead such as scheduling, sampling, attention, and communication;
- MAMF results depend on matrix shape and kernel-tuning quality;
- Bandwidth results depend on access patterns and cache behavior.
As a result, neither metric equals online inference throughput; they can only be used to explain trends and form hypotheses, not to directly convert into end-to-end tokens/s.
2.2 Single-card measured specs
Table 1 lists the single-card measured MAMF and bandwidth:
| Metric | Card A | Card B | Card C |
|---|---|---|---|
| BF16 MAMF (TFLOPS) | 652 | 419 | 580 |
| FP8 MAMF (TFLOPS) | 1,205 | 820 | 1,120 |
| Memory bandwidth (GB/s) | 4,472 | 1,540 | Not provided in raw data |
| Test power (W) | 450 | 600 | 450 |
| Rated power cap (W) | 600 | 600 | 600 |
From the MAMF figures:
- The FP8 uplift over BF16 was 1.85x for Card A, 1.96x for Card B, and 1.93x for Card C—the compute gain from low precision is similar in magnitude across all three cards.
- The gap between cards is clearly smaller at FP8 than at BF16: at BF16, Card A was 1.56x Card B and 1.12x Card C; at FP8, these ratios dropped to 1.47x and 1.08x.
- This suggests that low precision alleviates “under-utilization of compute units,” bringing the gaps between vendors back more to memory access and software.
On bandwidth, Card A is 4,472 GB/s and Card B is 1,540 GB/s, a ratio of 4,472 ÷ 1,540 ≈ 2.90x. This ratio has two usage boundaries:
- It is a value computed on a bandwidth kernel and applies only to this kind of access pattern;
- Card C’s bandwidth was not provided in the raw data, so its bandwidth cannot be back-derived from MAMF.
2.3 Two rules for precision comparability
First, MAMF across different precisions should not be ranked directly against each other. Memory bandwidth of 4,472 GB/s and FP8 MAMF of 1,205 TFLOPS belong to the memory-access and compute paths respectively and cannot be directly divided or mixed; there is also a structural difference between BF16 and FP8, for example:
- FP8 typically requires block-wise scaling of weights;
- Some hardware implements an FP8-like path as “two low-precision operations plus one high-precision accumulation,” whose effective throughput differs from pure FP8.
Therefore, multiples between BF16 and FP8 can only be compared within the same precision family, and bandwidth metrics cannot be mixed directly with compute metrics into a single conclusion.
Second, the implementation path for FP4-class formats is not uniform across vendors. Common approaches include:
- Micro-scaling block formats: extending low-precision representation with a shared scale factor per small block;
- Online conversion of FP8 or higher precision;
- Applying low precision only to weights while keeping activations at higher precision.
These three paths differ in kernel efficiency, numerical behavior, and supported model structures, which causes the actual performance of the same “MXFP4” label to vary widely across platforms.
2.4 From kernel to end-to-end: three easily confused layers
A simplified loop-ceiling model can be used to set expectations, across three layers in sequence: theoretical specs → kernel measurements → end-to-end throughput.
- Theoretical specs: vendor-claimed peak compute and peak bandwidth.
- Kernel measurements: results from the MAMF and bandwidth kernels; Table 1 belongs to this layer.
- End-to-end throughput: the output tokens/s in Table 2 and Table 3, jointly constrained by batching efficiency, the scheduler, attention implementation, sampling policy, and memory capacity.
The basic form of the loop-ceiling model is:
- decode: per-step loop ceiling ≈ min(compute ceiling / compute per step, bandwidth ceiling / bytes moved per step)
- prefill: theoretical single-request ceiling ≈ min(compute ceiling / prefill compute, bandwidth ceiling / input bytes moved)
Here the “bytes moved per step” or “input bytes moved” is not a fixed value; it must be estimated jointly with batch size, KV-cache sharing, weight sharing, quantization format, and attention implementation. This model is only used to form a trend judgment and cannot directly replace real load testing.
Take the 30B-A3B MoE model at FP8 precision for an illustrative estimate. For clarity, assume that about 5 GB of weights and KV must be moved per token:
| Item | Calculation | Card A | Card B |
|---|---|---|---|
| Per-step loop ceiling (128 concurrency) | bandwidth ÷ 5 GB | ≈894 steps/s | ≈308 steps/s |
| Corresponding throughput ceiling | ceiling × 128 | ≈114,000 tokens/s | ≈39,400 tokens/s |
| Measured short-load throughput | — | 7,459 tokens/s | 3,995 tokens/s |
| Measured long-load throughput | — | 1,820 tokens/s | 823 tokens/s |
The assumed value is only for demonstrating the reasoning method; the actual bytes moved per step should be calculated from model weight size, the number of active parameters, and the KV-cache strategy. The point of this estimate is that measured throughput is generally far below the theoretical ceiling, which shows that a real system is simultaneously constrained by scheduling, attention, sampling, and batching efficiency—bandwidth is only one of many constraints.
3. Single-card measurements and supplementary caliber: from spec gaps to measured gaps
3.1 Table 2: single-card, 128-concurrency measured throughput
Table 2 is the single-card, 128-concurrency measured output tokens/s. When comparing within the same row, the model, precision, and input load are held constant. Because the 120B-MXFP4 caliber needs separate confirmation, its data goes into Table 3 as supplementary and does not directly take part in the single-card comparison in this table.
Table 2 Single-card, 128-concurrency measured throughput (output tokens/s)
| Model/Precision | Load | Card A | Card B | Card C |
|---|---|---|---|---|
| MoE 30B-A3B (FP8) | Short (512/512) | 7,459 | 3,995 | — |
| MoE 30B-A3B (FP8) | Long (8192/1024) | 1,820 | 823 | — |
| MoE 30B-A3B (BF16) | Short (512/512) | 5,240 | 2,881 | — |
| MoE 30B-A3B (BF16) | Long (8192/1024) | 1,592 | 588 | — |
| Dense 8B (FP8) | Short (512/512) | 11,054 | 8,073 | 11,045 |
| Dense 8B (FP8) | Long (8192/1024) | 2,045 | 1,481 | 1,211 |
3.2 MoE 30B-A3B: two profiles for FP8 and BF16
Card A vs. Card B at FP8:
- Short input: 7,459 vs. 3,995, Card A leads by about 87%.
- Long input: 1,820 vs. 823, Card A is about 2.21x Card B.
Against the spec gaps in Table 1: at FP8 MAMF, Card A is 1.47x Card B; on bandwidth, it is 2.90x. The measured 2.21x under long input falls between the two and is closer to the bandwidth ratio, which suggests that under long inputs the contribution of memory access and the bandwidth path may rise; at the same time, kernel implementation, scheduling efficiency, sampling policy, and the attention path also jointly affect the final result.
Card A vs. Card B at BF16:
- Short input: 5,240 vs. 2,881, Card A leads by about 82%;
- Long input: 1,592 vs. 588, Card A is about 2.71x Card B, higher than the 2.21x at FP8.
FP8 speedup over BF16 within the same model:
- Card A: 1.42x on short input, 1.14x on long input;
- Card B: 1.39x on short input, 1.40x on long input.
For the same model, the speedup is not fixed: FP8 gains are clear on short inputs but get diluted by prefill and the attention path on long inputs. This means the conclusion “FP8 is always faster than BF16” needs to be re-evaluated for long-input scenarios.
3.3 Dense 8B: how load changes the ranking
In the FP8 tests on the 8B dense model:
- Short input: Card A 11,054, Card B 8,073, Card C 11,045—Card A and Card C are nearly tied, and Card B trails Card A by about 27%.
- Long input: Card A 2,045, Card B 1,481, Card C 1,211—the ordering becomes Card A > Card B > Card C, and Card A leads Card C by about 69%.
This suggests that single-card conclusions are very sensitive to load shape: under short inputs the ranking is more influenced by scheduling, memory-access efficiency, and batching behavior; under long inputs, the influence of compute, attention implementation, and KV-cache strategy becomes more pronounced. No single test conclusion automatically covers another load shape.
3.4 Table 3: 120B-MXFP4 supplementary data
The 120B-class MoE model uses MXFP4 weight quantization. Because this model is large and the original records may involve single-card, multi-card-on-single-machine, or different-resource-group differences, Table 3 lists it separately and does not fold it directly into single-card rankings.
Table 3 120B-MXFP4 supplementary throughput (output tokens/s)
| Model/Precision | Load | Card A | Card B | Card C |
|---|---|---|---|---|
| MoE 120B (MXFP4) | Short (512/512) | 3,376 | 3,064 | 5,057 |
| MoE 120B (MXFP4) | Long (8192/1024) | 1,825 | 1,169 | 2,101 |
From the supplementary data:
- Short input: Card C 5,057, Card A 3,376, Card B 3,064—Card C leads Card A by about 50%;
- Long input: Card C 2,101, Card A 1,825, Card B 1,169—Card C’s lead over Card A narrows to about 15%, while it still holds about 1.80x over Card B.
This suggests that on the MXFP4 path, Card C’s kernel, software stack, or scheduling policy may be better suited to short-input scenarios; the narrowing under long inputs may relate to the prefill compute path, attention implementation, KV-cache strategy, and quantization-format support. Because this table is a supplementary caliber, procurement and capacity planning should re-test in the target environment first.
3.5 Summary and cross-validation
Putting Table 1 and Table 2 together yields four cross-validation rules:
- Spec gaps and measured gaps are not linear. For example, the 2.71x of Card A vs. Card B under BF16 long input is closer to the 2.90x bandwidth ratio than to the 1.56x compute ratio, suggesting that prefill and the attention path may amplify bandwidth differences—though the actual result is still affected by software stack and scheduling.
- The precision dividend shrinks under long inputs. For 30B-A3B, moving from FP8 to BF16, Card A’s long-input throughput drops from 1,820 to 1,592, about 12.5%; Card B drops from 823 to 588, about 28.5%—Card B is more sensitive to precision.
- Cross-card comparison must use the same model, precision, and load. The inconsistent rankings for Dense 8B and 120B-MXFP4 show that model structure itself changes the relationship between cards.
- Low-precision paths need extra validation. Card C’s advantage on MXFP4 varies with load, suggesting that the implementation path and optimization level for this format differ across platforms and that BF16/FP8 conclusions cannot be simply applied.
4. A reproducible load-testing method and selection guidance
4.1 Making load tests reproducible: five dimensions
To turn “reference data” into a “reproducible experiment,” you need to align at least these five dimensions:
- Hardware and power: standardize the add-in form factor, power cap, and cooling conditions. If the power walls differ, mark it explicitly in the report.
- Software stack: record the driver, CUDA/ROCm version, inference framework image, model weight version, quantization format, and quantization algorithm.
- Load definition: standardize input/output lengths, concurrency, sampling parameters (temperature, top_p, max_tokens), and whether streaming is used.
- Statistical caliber: define clearly how “total output tokens/s” is computed. It is recommended to use: valid output tokens from requests that completed successfully and were not truncated within the steady-state window, divided by the steady-state window seconds. Failed, timed-out, and connection-error requests are excluded by default; if included, they must be listed separately. Truncated requests (finish_reason=length) are excluded or listed separately by default.
- Environment isolation: close unrelated processes, fix CPU affinity and NUMA binding, and avoid container resource contention.
On the output tokens/s caliber, two more points: first, prefer the server-returned usage field, such as completion_tokens; only if usage is unavailable should you approximate the output token count from non-empty delta events in the streaming response, and clearly state the approximation error. Second, distinguish between “total throughput,” “steady-state throughput,” “valid throughput,” and “request success rate.” Reporting only a single throughput number without failure rate, truncation rate, and window length easily causes misreading.
4.2 Illustrative load-testing script
Below is an illustrative script based on the OpenAI-compatible interface. It relies on the standard library, drives load at fixed concurrency, and reports steady-state output tokens/s and average TTFT. The script excludes failed, timed-out, and truncated requests by default; in production, use a dedicated load-testing tool and keep full logs.
# Illustrative script: drive load at fixed concurrency; report steady-state output tokens/s and average TTFT# Metric caliber: valid output tokens of successful, non-truncated requests / steady-state window seconds# Failed/timeout/truncated requests are counted separately and kept out of the main throughputimport jsonimport timeimport threadingfrom concurrent.futures import ThreadPoolExecutorfrom urllib.request import Request, urlopenBASE_URL = "http://<inference-endpoint>:8000/v1/chat/completions"MODEL = "<model-name>"PROMPT = "<input text validated by the tokenizer to be 512 or 8192 tokens>"TARGET_OUT = 512 # short load 512, long load 1024CONCURRENCY = 128WINDOW_SEC = 120 # steady-state window after warm-uplock = threading.Lock()agg = {"out_tokens": 0,"reqs": 0,"ttft_sum": 0.0,"ttft_cnt": 0,"failed": 0,"truncated": 0,}def one_request():"""Send one streaming request; return output token count, first-token latency, and finish reason."""payload = {"model": MODEL,"messages": [{"role": "user", "content": PROMPT}],"max_tokens": TARGET_OUT,"temperature": 0.0,"stream": True,}req = Request(BASE_URL,data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json"},)t0 = time.perf_counter()ttft = Nonen = 0usage_tokens = Nonefinish_reason = Nonewith urlopen(req, timeout=600) as resp:for raw_line in resp:line = raw_line.decode("utf-8").strip()if not line.startswith("data:"):continuechunk = line[len("data:"):].strip()if chunk == "[DONE]":breakobj = json.loads(chunk)# If the server returns usage, prefer completion_tokensusage = obj.get("usage")if usage and usage.get("completion_tokens") is not None:usage_tokens = usage["completion_tokens"]choices = obj.get("choices") or []if not choices:continuechoice = choices[0]delta = choice.get("delta", {})if delta.get("content"):if ttft is None:ttft = time.perf_counter() - t0# If usage is not yet available, approximate the count with non-empty deltasif usage_tokens is None:n += 1if choice.get("finish_reason"):finish_reason = choice["finish_reason"]if usage_tokens is not None:n = usage_tokensreturn n, ttft, finish_reasondef worker(deadline):while time.perf_counter() < deadline:try:n, ttft, finish_reason = one_request()with lock:if n <= 0:agg["failed"] += 1elif finish_reason == "length":agg["truncated"] += 1else:agg["out_tokens"] += nagg["reqs"] += 1if ttft is not None:agg["ttft_sum"] += ttftagg["ttft_cnt"] += 1except Exception:with lock:agg["failed"] += 1if __name__ == "__main__":one_request() # warm-up: trigger kernel compilation and cache warm-upstart = time.perf_counter()deadline = start + WINDOW_SECwith ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:for _ in range(CONCURRENCY):pool.submit(worker, deadline)elapsed = time.perf_counter() - startvalid_tokens_per_sec = agg["out_tokens"] / elapsed if elapsed > 0 else 0avg_ttft_ms = 1000 * agg["ttft_sum"] / agg["ttft_cnt"] if agg["ttft_cnt"] else 0print(f"Steady-state output tokens/s = {valid_tokens_per_sec:.0f}")print(f"Valid requests = {agg['reqs']}")print(f"Failed/timeout requests = {agg['failed']}")print(f"Truncated requests = {agg['truncated']}")print(f"Average TTFT = {avg_ttft_ms:.0f} ms")
The “non-empty delta event count” in the script is only an approximation of output tokens. A more rigorous approach is to use the server-returned usage field or switch to an offline dataset for exact counting. If the business needs to measure “total attempt throughput including failed requests,” compute it separately; it must not be mixed with steady-state valid throughput.
4.3 A selection decision framework
Based on the data and methods in this article, a three-layer selection framework can be given:
- Start with the workload profile. For short inputs, long outputs, and high concurrency, prioritize bandwidth and decode efficiency; for long inputs with a high prefill share, prioritize compute and attention implementation. Short loads test decode, long loads test prefill—one set of conclusions cannot cover all workloads.
- Then look at the precision path. With BF16/FP8 as the baseline, compare MAMF and bandwidth first; if low precision like MXFP4 is needed, you must validate the actual speedup, numerical stability, and inference-framework support for the target model on that platform, rather than looking only at a format-support list.
- Finally, look at the software stack and power. The driver, CUDA/ROCm version, framework image, quantization implementation, and power wall all significantly affect measured values and should be re-tested in the target environment.
In a general-purpose GPU cloud environment, inference services typically run on compute resources such as GPU cloud servers. An organization can turn a “fixed concurrency + fixed load” load-testing method into a pre-launch benchmark, then archive the test scripts and results using object storage, and use cloud monitoring and logging services to track online TTFT, throughput, and error rates—forming a “load-test → launch → observe → tune” loop. This cloud description is only a general example and does not point to any specific vendor.
4.4 Frequently asked questions
How is output tokens/s computed?
Output tokens/s usually means valid output tokens within the steady-state window divided by the measurement time. A rigorous caliber needs to specify: whether failed, timed-out, and connection-error requests are excluded; whether requests truncated by max_tokens are excluded; whether the server usage field is used; whether warm-up is performed; and how long the measurement window is. Reporting only throughput without failure rate, truncation rate, and window conditions easily distorts the number.
How do I choose short vs. long loads?
If the workload has short inputs and long outputs, the main pressure is on decode, so focus on memory bandwidth, KV-cache efficiency, and the decoding kernel; if the workload has long inputs and relatively short outputs, the main pressure is on prefill, so focus on matrix-achievable FLOPS, attention implementation, and long-context handling. Short loads suit evaluating interactive generation; long loads suit evaluating long-document understanding, code completion, and large-context summarization.
How do I choose among BF16 / FP8 / MXFP4?
BF16 usually has better compatibility and works well as a baseline; FP8 is more balanced between throughput and memory usage, but you need to validate model accuracy loss and kernel support; low-precision formats like MXFP4 suit memory-sensitive scenarios or specific MoE models, but you must confirm that the inference framework, quantization format, activation precision, and model structure are all compatible. Selection cannot be based on the format name alone—it depends on measured throughput, first-token latency, output quality, and stability of the target model on the target platform.
4.5 Limitations
The data in this article comes from a limited number of tests in a specific environment and cannot serve as an absolute performance ranking. Any use of these conclusions for procurement or capacity planning should first be reproduced and validated in your own target environment. If the original test environment, power settings, driver version, framework image, or quantization configuration changes, the measured results may differ significantly.
