A Guide to Evaluating LLM Inference Performance on Heterogeneous Accelerators
Evaluating hardware for LLM inference is often oversimplified by comparing peak compute or single-card bandwidth. However, real-world throughput is determined by a combination of model architecture, precision formats, batching strategies, memory capacity, interconnect topology, inference engine implementations, and software stack versions. Based on an anonymized single-card and multi-card testing round, this article discusses how to establish a more credible inference performance evaluation system, covering micro-benchmarks, model-level benchmarks, and reproducible test matrix designs.
In large language model (LLM) inference scenarios, accelerator selection is often oversimplified to comparing peak compute or single-card bandwidth. However, actual online throughput is determined by a combination of model architecture, precision formats, batching strategies, memory capacity, interconnect topology, inference engine implementations, and software stack versions. This article uses an anonymized single-card and multi-card testing round as a thread to discuss how to establish a more credible inference performance evaluation system. The identifiers A, B, and C in this article are solely anonymized labels for this round of testing and do not represent any specific products. All data should be understood in the context of the testing conditions and should not be cited separately from the model, precision, input/output length, and software stack.
1. Why Peak Compute Does Not Directly Represent Inference Throughput
Inference services are typically divided into two phases: Prefill and Decode. The Prefill phase processes the input context and is more compute-intensive; with long inputs, it is more likely to reflect differences in matrix computation, attention paths, and operator fusion. The Decode phase generates tokens one by one and is often more influenced by memory traffic, KV Cache management, batching efficiency, and scheduling strategies. In short input/short output scenarios, the Decode phase accounts for a larger proportion, whereas in long input scenarios, the weight of the Prefill phase increases. However, the final bottleneck still depends on the model architecture, concurrency level, and batch size.
Therefore, achievable compute in matrix multiplication kernels, memory bandwidth in memory access kernels, and online model throughput represent three different levels of metrics. The first measures the achievable floating-point throughput of a specific matrix multiplication kernel at a given precision, and the second measures the read bandwidth of a specific memory access kernel. Neither equates to end-to-end inference throughput. Cross-precision comparisons require particular caution: BF16, FP8, and FP4 differ in numerical representation, scaling strategies, kernel implementations, and precision calibration requirements. You cannot directly rank the peak values of different precisions. The FP4 path exhibits more significant differences across different vendors and software stacks, and must be validated alongside actual model precision loss, kernel coverage, and business throughput.
2. Anonymized Comparison of Single-Card Basic Capabilities
In a certain round of single-card testing, the basic capabilities of three types of accelerators are shown in the table below. The data in the table is only used to illustrate the metric definitions and does not serve as a basis for cross-precision ranking.
| Metric | A | B | C |
|---|---|---|---|
| BF16 Matrix Multiplication Achievable Compute (TFLOPS) | 652 | 419 | 580 |
| FP8 Matrix Multiplication Achievable Compute (TFLOPS) | 1205 | 820 | 1120 |
| Memory Read Bandwidth (GB/s) | 4472 | 1540 | Undisclosed |
Looking at the same precision data in the table, A’s BF16 and FP8 matrix multiplication achievable compute is higher than B and C. A’s memory read bandwidth is approximately 2.90 times that of B. This ratio comes from 4472 ÷ 1540 and applies only to that specific bandwidth kernel; it cannot be directly extrapolated to a throughput multiplier for all models. C did not disclose some metrics, and missing value inferences should not be made. More importantly, matrix compute and bandwidth are only basic capabilities; online throughput requires looking at model-level benchmarks.
3. Throughput Interpretation Under 128 Concurrency on a Single Card
When comparing the same model precision and input load within the same row, short input is defined as 512/512, and long input is defined as 8192/1024, with units in output tokens/s. The results of a single-card test with 128 concurrent requests can be abstracted as follows.
| Scenario | Precision | Input/Output | A | B | C | Observation |
|---|---|---|---|---|---|---|
| Medium-scale sparse activated model | FP8 | 512/512 | 7459 | 3995 | — | A is about 87% higher than B |
| Medium-scale sparse activated model | FP8 | 8192/1024 | 1820 | 823 | — | A is about 2.21 times B |
| Medium-scale sparse activated model | BF16 | 512/512 | 5240 | 2881 | — | A is higher than B |
| Medium-scale sparse activated model | BF16 | 8192/1024 | 1592 | 588 | — | Long input gap widens |
| Small parameter general model | FP8 | 512/512 | 11054 | 8073 | 11045 | A and C are close |
| Small parameter general model | FP8 | 8192/1024 | 2045 | 1481 | 1211 | Ranking changes |
These results illustrate three points. First, the acceleration ratio of lower precision relative to higher precision is not fixed across different models and input lengths for the same accelerator. The improvement of FP8 over BF16 is affected by quantization kernels, matrix shapes, KV Cache precision, and operator fusion; a fixed multiplier cannot be assumed. Second, short input scenarios are closer to Decode-dominated, where memory traffic and batching efficiency have a noticeable impact; in long input scenarios, the proportion of Prefill increases, amplifying differences in compute paths and attention implementations. Third, on the small parameter model, A and C are close under short input, but the ranking changes for long input, indicating that selection must cover the input/output distribution of the target business rather than just looking at a single short input benchmark.
4. Comparability of Software Stack Versions and Multi-Card Setups
In a certain round of testing, most single-card model scenarios used a newer combination of runtime and inference framework, while the multi-card large model scenario used a different version combination, and the comparison platform used its corresponding software stack. The software stack version must be treated as an experimental variable: different versions may exhibit significant differences in operator scheduling, memory management, parallelization strategies, quantization kernels, and communication overlap. If multi-card large model data uses a different software stack version than single-card data, they should not be directly compared horizontally.
Multi-card scenarios also require attention to interconnect bandwidth, topology, tensor/pipeline/expert parallelism strategies, communication overhead, and load balancing. Single-card 128 concurrency mainly examines single-card memory capacity, KV Cache management, and batching efficiency; multi-card large models add inter-card communication and parallel partitioning costs. During evaluation, single-card micro-benchmarks, single-card model-level benchmarks, and multi-card business-level benchmarks should be established separately, and the software stack version, driver version, inference engine configuration, parallelism degree, and quantization scheme should be recorded.
5. Recommendations for Reproducible Benchmark Design
It is recommended to organize tests in a matrixed manner, fixing sampling parameters and input distributions, covering at least four combinations: short input, long input, low concurrency, and high concurrency. A pseudo-configuration is as follows:
test_matrix:models: [medium-scale sparse activated model, small parameter general model, multi-card large model]precisions: [BF16, FP8, MXFP4]input_output: [[512,512], [8192,1024]]concurrency: [1,32,128]metrics: [output_tokens_per_s, ttft_ms, tpot_ms, p99_latency_ms, peak_mem_gb]controls:temperature: 0top_p: 1warmup_rounds: 3repeat_rounds: 5
Regarding metrics, in addition to output throughput, you should record Time To First Token (TTFT), Time Per Output Token (TPOT), P99 latency, peak memory, and error rate. For fairness, ensure the same model weights, same precision, same input/output length, same concurrency, same sampling parameters, and same parallelism strategy; if the software stacks differ, tune each to a stable state before comparing. For FP4 and MXFP4 paths, precision evaluation should also be added to confirm that the output quality is acceptable for the business. Cost evaluation can be normalized using “cost per unit token = card-hour cost ÷ throughput,” but this should not be detached from latency SLA and memory capacity constraints.
6. Tuning and Selection Recommendations
First, establish a three-tier evaluation: micro-benchmarks to look at matrix compute and bandwidth limits, model-level benchmarks to look at actual throughput and latency, and business-level benchmarks to look at P99 and cost under mixed request distributions. Second, do not directly rank across precisions, and do not treat bandwidth kernel results as model throughput. Third, in short input scenarios, focus on Decode efficiency, KV Cache, and batching; in long input scenarios, focus on Prefill compute paths, attention implementations, and memory capacity. Fourth, in multi-card scenarios, lock the software stack version, record parallelism strategies and communication overhead, and avoid misjudging version differences as hardware differences. Fifth, conduct joint precision and throughput acceptance for low-precision paths, especially for FP4/MXFP4.
7. Conclusion
The evaluation of LLM inference performance on heterogeneous accelerators should shift from “single-metric comparison” to “scenario-based benchmarking.” Matrix compute, memory bandwidth, precision formats, concurrency, input length, and software stack versions jointly determine online throughput. Anonymized data shows that the ranking of different accelerators may change under different models and loads, and the low-precision acceleration ratio of the same card is not fixed. Only by establishing a reproducible, consistently defined benchmark system that covers the target business distribution can reliable evidence be provided for inference service selection and tuning.
