Benchmarking Inference Accelerator Cards for Large Language Models: A Unified Testing Methodology
This article provides a practical framework for comparing the inference performance of accelerator cards for large language models (LLMs). It emphasizes the critical need to standardize hardware, software stacks, test scripts, and metric definitions before drawing conclusions. Using data from three platform types (A, B, C), the analysis reveals how short-input and long-input workloads produce different performance rankings, why FP8 speedups over BF16 vary with context length, and how memory bandwidth and compute ceiling interact in decode-bound scenarios. The article also covers multi-card parallelism considerations, KV cache memory budgeting, and a reproducible benchmarking template, concluding with scenario-specific selection advice and common pitfalls to avoid.
Test Methodology: Standardize Before Comparing
In large model inference selection, claims like “Card X is faster” often fall apart under scrutiny. The problem is rarely the data itself but the measurement context: hardware platforms, software stack versions, test scripts, concurrency parameters, and quantization implementations all differ. To compare inference performance across platforms fairly, the following baselines should first be unified.
- Hardware baseline: Prefer comparing products from the same generation and the same VRAM tier. Different memory capacities change the achievable concurrency ceiling, which in turn alters the conclusion.
- Software baseline: Use the same mainstream open-source inference framework, the same model weights, and the vendor’s unified container or base image. Framework versions significantly affect scheduling, paged KV cache management (e.g., PagedAttention), graph capture, and other optimization paths.
- Drivers and toolchain: Versions of the same vendor’s general-purpose compute stack vary noticeably. Older versions often lack support for newer data formats and insufficient operator fusion, making cross-version comparisons meaningless.
- Test scripts and metric definitions:
- Input and output lengths are recorded as “tokens / input tokens / output tokens.”
- Throughput is uniformly reported as output tokens/s, i.e., per-request output length multiplied by the number of completed requests, divided by total elapsed time.
- Concurrency, warm-up rounds, and repetition counts are also recorded.
- Runtime environment: Fix the node and CPU model, disable energy-saving frequency scaling that may introduce jitter, and avoid sharing the device with other tasks.
The test data referenced in this article is as follows:
- Three types of accelerator cards are labeled Platform A, Platform B, and Platform C.
- Platforms A and B are data-center general-purpose compute cards; Platform C is a workstation-class professional accelerator card.
- Single-card tests use each vendor’s latest-generation general-purpose compute stack and framework version, with the corresponding newer container image by default.
- In multi-card tests of large models, Platform C uses a relatively older software stack version.
- NVIDIA baseline measurements are based on CUDA 13.4.1 and the corresponding framework container. Per the naming convention in this article, “Platform A” refers to this NVIDIA baseline group.
- The baseline data comes from public technical community single-card test results and represent only specific hardware/software combinations, not a definitive product ranking. Actual selection should be based on re-testing in your own environment.
Key conclusions in advance:
- Under short input (512/512), Platforms A and C achieve nearly identical FP8 throughput. Platform B is about half of Platform A, consistent with the bandwidth ratio of 4,472 ÷ 1,540 ≈ 2.90×.
- Under long input (8192/1024), the ranking flips: Platform A’s advantage over Platform C grows from ~1.00× at short input to ~1.69× at long input.
- The FP8 speedup over BF16 for the same model is not constant: roughly 1.4× for short input, and between 1.1× and 1.4× for long input depending on the platform.
- In multi-card scenarios, Platform C used an older software stack; its results should be treated as reference under specific conditions and not compared directly with newer single-card data.
1. Two Different Benchmarks: Matrix Compute vs. Memory Bandwidth
The most common mistake in evaluation is treating peak or achievable matrix compute (often quoted in marketing materials) as a predictor of online throughput. These two types of benchmarks measure different things and must be clearly distinguished.
Achievable matrix compute (e.g., MAMF-type metrics) measures “how fast can it compute.” It is obtained from pure matrix multiplication kernels with fixed shapes and fully reusable data, reflecting the upper bound of FLOPS achievable through matrix units and compiler optimizations. It does not include:
- Overhead of transferring weights and KV cache;
- Memory-bound operations such as attention and normalization;
- Scheduling and communication overhead.
Memory bandwidth measures “how fast can it fetch data.” It comes from pure memory-access kernels and represents the theoretical access capability of high-bandwidth memory (HBM) or graphics memory (GDDR).
| Platform | Achievable matrix compute (BF16 / FP8, TFLOPS) | Memory bandwidth (GB/s) |
|---|---|---|
| Platform A (data-center card, HBM) | 652 / 1,205 | 4,472 |
| Platform B (workstation-class card, GDDR) | 419 / 820 | 1,540 |
| Platform C (data-center card) | 580 / 1,120 | Not in the listed data range |
Three points:
- The bandwidth ratio holds only for that bandwidth kernel. 4,472 ÷ 1,540 ≈ 2.90 is the ratio of the bandwidth kernel. Platform A’s achievable FP8 compute is about 1,205 ÷ 820 ≈ 1.47× of Platform B. Thus Platform A has a higher “bandwidth/compute ratio”. Intuitively, it provisions more bandwidth per unit of matrix compute. In the memory-bound decode stage, this structure more easily translates into a throughput advantage.
- Different precisions cannot be directly ranked. BF16 and FP8 achievable compute are two independent measurements. If a low-precision kernel still needs to dequantize weights back to higher precision for accumulation, or is limited by shared memory capacity, register pressure, or compilation strategies, the actual gain will be significantly reduced.
- 4-bit floating point paths vary substantially across vendors. Block scaling granularity, scaling factor storage format, and conversion overhead all differ. The same 4-bit format may achieve vastly different compute on different software stacks, making cross-vendor 4-bit comparisons largely meaningless.
Neither of these two benchmarks equals model online inference throughput. Online throughput is determined by the combination of model architecture, parallelism strategy, and framework implementation.
2. Single-Card, 128 Concurrent Requests: Short vs. Long Inputs
Two representative model types are tested:
- A sparse Mixture-of-Experts (MoE) encoder model with ~30B total parameters, activating only ~3B parameters per token;
- A dense general-purpose model with ~8B parameters.
Two workload profiles are used:
- Short input: 512/512 — 512 input tokens and 512 output tokens per request;
- Long input: 8192/1024 — close to code completion and long-document understanding scenarios.
Concurrency is fixed at 128, and the metric is output tokens/s. Rows compare the same model and precision under different workloads.
| Model & precision | Workload | Platform A | Platform B | Platform C | Key difference |
|---|---|---|---|---|---|
| ~30B total / 3B active MoE encoder, FP8 | 512/512 | 7,459 | 3,995 | — | B is ~54% of A (A ~87% higher) |
| Same, FP8 | 8192/1024 | 1,820 | 823 | — | A is ~2.21× of B |
| Same, BF16 | 512/512 | 5,240 | 2,881 | — | A is ~1.82× of B |
| Same, BF16 | 8192/1024 | 1,592 | 588 | — | A is ~2.71× of B |
| ~8B dense model, FP8 | 512/512 | 11,054 | 8,073 | 11,045 | A≈C, B is ~73% of A |
| ~8B dense model, FP8 | 8192/1024 | 2,045 | 1,481 | 1,211 | Ranking flips: A≈1.69C≈1.38B |
Observation 1: For short inputs, the bandwidth ratio maps almost linearly to throughput ratio.
- For the MoE model with FP8 short input, Platform A is about 1.87× of Platform B, close to the bandwidth ratio of 2.90× (within the lower-bound relationship).
- Platform B is about 54% of A, near the midpoint between the theoretical reciprocal of bandwidth ratio (1/2.90 ≈ 34%) and the full ratio.
- For the 8B model with FP8 short input, A and C are nearly identical (11,054 vs 11,045). This is consistent with both being data-center cards with likely similar memory bandwidth tiers.
Observation 2: With long inputs, the ranking flips, and Platform C shows the largest drop.
- When input length grows from 512 to 8192, throughput for A, B, and C drops to about 24%, 21%, and 11% of their short-input values, respectively.
- The ranking for the 8B model changes from A≈C>B to A>B>C.
- With identical compute precision, parameter count, and framework version, the ranking shift is rooted in the amplified cost of prefill and attention computation for 8192-token sequences, which magnifies differences in attention operator efficiency and long-sequence compute paths across platforms.
Observation 3: FP8 speedup over BF16 is not constant.
| Platform | Short input 512/512 | Long input 8192/1024 |
|---|---|---|
| Platform A | 7,459 ÷ 5,240 ≈ 1.42× | 1,820 ÷ 1,592 ≈ 1.14× |
| Platform B | 3,995 ÷ 2,881 ≈ 1.39× | 823 ÷ 588 ≈ 1.40× |
- Platform A’s lower long-input speedup can be explained as follows: The MoE model activates only ~3B parameters per token, so the per-step weight read volume is inherently limited. When input length grows, KV cache reads and attention computation become the main incremental costs. If these stages still run in BF16, FP8 only compresses the weights, diluting overall benefit.
- Platform B’s slightly higher long-input speedup suggests that attention and KV access are even stronger bottlenecks under long sequences, so FP8 benefits extend to more hotspots.
Conclusion: The “FP8 vs. BF16 speedup” of a given card is not a fixed constant; it must be measured on the actual model and actual input distribution.
3. A Roofline Perspective: Why Decode Is Bandwidth-Bound First
During the decode phase, each generated token requires reading all activated weights, the current sequence’s KV cache, and framework metadata, while each weight element participates in only one multiply-accumulate. Arithmetic intensity (FLOPs/Byte) is very low. Many matrix operations degrade into matrix-vector operations, placing the workload firmly in the memory-bound region.
Based on this, we can construct a simplified lower-bound execution time model:
# Symbols: W = activated weight bytes, C = KV cache bytes per token# B = concurrent requests, L = current context length# BW = memory bandwidth, FLOPS_eff = effective compute at the given precision (typically well below achievable)bytes_per_step = W + B * C * Lt_step ≈ max(bytes_per_step / BW, flops_per_step / FLOPS_eff)
This lower-bound model gives at least three engineering insights:
- The weight cost is amortized over longer contexts. When B×C×L exceeds W, the bandwidth demand is dominated by the KV cache. In that regime, increasing memory capacity or direct bandwidth is more effective than increasing compute.
- Decode is highly bandwidth-sensitive. For the MoE model with FP8, a single request (B=1) must read roughly 3 GB of activated weights per decode step. At 1,540 GB/s, an ideal step would take about 2 ms. In reality, attention, scheduling, and communication overhead make measured latency much longer.
- Prefill is compute-bound. Under long inputs, computation grows quadratically with sequence length, dominated by matrix compute and attention kernel efficiency. Since decode and prefill alternate within a request, the performance gap between cards differs across workload types.
Framework and Operator-Level Optimizations Also Affect Results
# Illustrative pseudocode: execution path of a decode step in an inference framework# A request = (prompt_tokens, sampling_params, priority)def decode_step(running_requests, kv_cache, model, engine):# 1) Scheduling: choose requests to process this step based on priority and KV cache spacebatch = scheduler.select(running_requests, kv_cache.free_blocks())if not batch:return {"computed": 0, "reason": "kv_cache_or_queue_empty"}# 2) Paged KV cache: allocate memory blocks on demand to reduce fragmentation and reserved waste# Smaller block sizes reduce fragmentation but increase metadata and addressing overheadblocks = kv_cache.allocate(batch, block_size=16)if blocks is None:return {"computed": 0, "reason": "preempt"} # preempt/swap out lower-priority requests# 3) Forward pass: weight reads + paged attention + sampling# - Fused attention kernels avoid materializing the full attention matrix# - Graph capture turns fixed-shape operator sequences into replayable execution graphs, reducing per-step launch overhead# - Speculative decoding uses a small draft model to propose multiple candidate tokens, verified in parallel by the main model,# trading extra compute for fewer serial steps (especially effective for bandwidth-bound decode)logits = model.forward(batch, blocks, kv_cache.page_table)next_tokens = sampler.sample(logits, batch.sampling_params)# 4) Request lifecycle managementfor req, tok in zip(batch, next_tokens):req.append(tok)if req.is_finished(): # hit stop token or reached max output lengthrunning_requests.remove(req)kv_cache.release(req)return {"computed": len(batch)}
The implementation differences of an inference framework directly show up in long-input tests. Paged KV cache block granularity, attention kernel chunking strategies for long sequences, graph capture coverage, and support for speculative decoding all affect real-world performance at different input lengths. This further underscores that horizontal comparisons must lock in the framework version and key configuration; otherwise, you are comparing software, not hardware.
4. From Single-Card to Multi-Card: Version and Topology Change the Conclusion
When model size exceeds single-card memory, the evaluation target shifts from “card” to “system.” The multi-card data in this article uses a relatively older software stack, and two points must be noted when interpreting it.
Version alignment cannot be mixed. Older software stacks typically lack:
- Native support for new data formats;
- Overlapping optimization of multi-card communication and computation;
- Newer fused attention and normalization kernels.
Therefore, multi-card results should be treated as a “lower-bound reference under a specific version” and not compared horizontally with newer single-card data, nor used to infer the relative position of newer multi-card platforms.
Parallelism strategy determines communication overhead. Common strategies differ in what they split and where the bottleneck lies:
| Parallelism | What is split | Main communication per step | Primary bottleneck |
|---|---|---|---|
| Tensor Parallelism (TP) | Matrix rows/columns within a layer | AllReduce / AllGather | Inter-card bandwidth and interconnect topology |
| Pipeline Parallelism (PP) | Layers divided into stages | P2P transfer between adjacent stages | Pipeline bubble, first-token latency on long inputs |
| Expert Parallelism (EP) | Different experts distributed across cards | All-to-All routing | Tail latency due to routing imbalance |
- TP: At least two collective communications per step; communication volume grows with hidden size and batch size. On a single node with high-speed private interconnect, efficiency is high, but scaling across nodes is severely limited.
- EP for MoE models: Each token activates only a few experts, so inter-card communication consists of fine-grained, frequent small packets, extremely sensitive to latency and routing balance. When the older software stack’s communication scheduling is inefficient, measured results may be far below the hardware interconnect’s theoretical ceiling.
A reasonable multi-card evaluation process:
- Test different TP/EP combinations on a single-node, multi-card topology (e.g., NVLink-class).
- Record scaling efficiency, i.e., ratio of N-card throughput to single-card throughput.
- Compare against a linear estimate based on pure bandwidth or compute.
- Finally, evaluate cross-node scaling options.
5. Long-Context Memory Budget: KV Cache Often Limits Concurrency
Memory capacity and KV cache efficiency determine how many concurrent requests a deployment can support. KV cache size can be approximated as:
# In GQA models, the number of KV heads is typically much smaller than query headskv_bytes ≈ 2 × layers × kv_heads × head_dim × dtype_bytes × seq_len × batch# Factor 2 accounts for K and V; seq_len is the current context length; prefill requires reserving for input length
For a typical configuration: 32 layers, 8 KV heads, head dimension 128, FP16 precision, the KV cache per token is about 64 KiB. A context length of 8192 would consume about 512 MiB per request, and 128 concurrent requests would need about 64 GiB just for KV cache. This directly reduces the space available for model weights and framework reserves, causing the actual concurrency at long input to be much lower than the configured 128.
This mechanism explains a common observation: long-input throughput degradation is not entirely “slow compute” but an increased proportion of queueing, preemption, and recomputation under high concurrency. Practical engineering responses include:
- KV cache quantization to compress cache size with acceptable precision loss;
- Hierarchical caching, swapping cold requests’ KV cache to host memory or even remote storage, and restoring on demand;
- P/D separation, deploying compute-intensive prefill and bandwidth-sensitive decode on separate instances with matched hardware;
- Prefix cache reuse for common prefixes such as system prompts or long documents, avoiding repeated prefill.
The relationship between higher and lower precision can be understood as:
- Higher compute precision simplifies operator implementation and tuning, reducing rounding errors and certain kernel performance penalties, but increases memory overhead and lowers the concurrency ceiling;
- FP8 uses E4M3 / E5M2 encodings to balance dynamic range and precision, typically applied to weights, activations, and KV cache;
- 4-bit floating point formats rely on block-level scaling factors to restore dynamic
