How to Choose a Large-VRAM Workstation GPU: VRAM Capacity, Memory Bandwidth, and Compute Explained
Selecting a workstation GPU for local LLM inference requires looking beyond raw compute figures. This guide explains why VRAM capacity determines whether a model can run at all, how memory bandwidth governs token generation speed, and how to interpret conflicting compute specifications like INT8 TOPS and FP4 ratings. It also covers cooling design for multi-GPU setups, PCIe considerations, software stack compatibility, and a step-by-step selection process to help you match hardware to your actual workloads.
1. Why Local Inference Deserves Careful GPU Selection
More development teams and enterprises are moving large model inference onto local servers or workstations. The benefits are clear: data never leaves the internal network, response latency is predictable, and long-term costs are easy to forecast. Workloads with high and steady request volumes are especially well suited to this approach.
Local deployment does not rule out the cloud. A common pattern is a hybrid architecture: model training and bursty peak inference go to elastic cloud compute, while steady day-to-day inference runs on local workstations. This keeps costs under control while keeping sensitive data in-house.
2. Start with VRAM Capacity — It Decides Whether the Model Runs at All
Many people look at compute numbers first when choosing a GPU. For local inference, VRAM capacity is usually the first gate. The model weights must fit entirely in one GPU’s VRAM, or be distributed across the VRAM of multiple GPUs, before the model can run.
Take a 30-billion-parameter model as an example. After INT8 quantization, each parameter takes 1 byte, so the weights need roughly 30 GB. That is only the starting point. Inference also requires storing the KV Cache, which grows with context length and concurrent request count. The framework itself consumes additional VRAM as well.
The KV Cache can be roughly estimated with this formula:
KV Cache bytes per token ≈ 2 × number of layers × number of KV heads × head dimension × bytes per element
For a model with 60 layers, 8 KV heads, a head dimension of 128, and FP16 caching, each token takes about 245 KB. With a context length of 4096 and 4 concurrent requests, the KV Cache totals roughly 4 GB. Stretch the context to 32K, and a single request’s KV Cache reaches around 8 GB.
This shows that whether a 32 GB GPU can host a 30B-parameter INT8 model depends on how it is used. With short contexts and low concurrency, it may be just enough. As concurrency rises or contexts lengthen, VRAM runs out before compute does. INT4 quantization significantly reduces weight footprint, but it introduces some accuracy loss that you need to validate against your own tasks.
Multi-GPU tensor parallelism can shard weights and the KV Cache across multiple cards, breaking through single-GPU capacity limits. The trade-offs are increased inter-GPU communication overhead and more complex deployment and debugging.
3. Memory Bandwidth Determines Generation Speed
In autoregressive generation, producing each token requires reading the full model weights from VRAM once. Therefore, in single-request scenarios, decoding speed is primarily limited by memory bandwidth.
A rough theoretical calculation: with 600 GB/s of bandwidth and 30 GB of weights, the weights can be fully read at most about 20 times per second, so the theoretical decoding ceiling is around 20 tokens/s. Actual speed will fall below this number due to framework efficiency, how the KV Cache is read, and batching strategy. Among GPUs of the same architecture generation, bandwidth differences often reflect real-world inference experience better than compute differences do.
4. Interpreting Compute Specifications Correctly
Compute figures on the market are not standardized, and this is where selection mistakes happen most easily. Common specifications include:
- INT8 TOPS: peak compute at 8-bit integer precision, currently the most common precision for local deployment;
- FP16/BF16 FLOPS: half-precision floating-point compute, relevant to training and some high-precision inference;
- FP4 and other ultra-low-precision ratings: often very high numbers, but whether they can be realized in practice depends on the model, framework, and operator support.
Two points deserve attention. First, numbers at different precisions cannot be directly converted or compared. An INT8 figure from one product and an FP4 figure from another cannot be ranked side by side. Second, some vendors’ TOPS figures are based on structured sparsity and are only achievable when the model can exploit sparsity. Confirm the test conditions behind the rating before purchasing.
The correct approach: first determine the quantization precision your business model will use, compare compute at the same precision, and then make a combined judgment together with VRAM capacity and bandwidth.
5. Cooling and Chassis Design
Workstation GPUs often run at full load for long periods, and multiple cards may be densely installed. Blower-style coolers exhaust hot air directly out of the chassis, which matters especially when cards sit side by side. Open-fan coolers in a compact layout will ingest hot air exhausted by neighboring cards, causing temperatures to rise noticeably. A vapor chamber covering both the core and the memory spreads heat more evenly across the heatsink fins.
When evaluating cooling, check three data points: core temperature under full load, memory temperature (easily overlooked, but equally critical), and the fan noise curve. Also verify that the chassis airflow path is unobstructed.
6. Interfaces and Multi-GPU Expansion
For pure inference, PCIe 4.0 is usually sufficient even though PCIe 5.0 x16 interfaces exist. Higher bandwidth becomes more valuable in multi-GPU tensor parallelism and when loading model shards.
Before deploying multiple GPUs, confirm three things: whether slot spacing allows adequate cooling, whether the power supply meets the demand, and whether the motherboard can sustain x16 lanes per card when all cards are installed — some motherboards drop to x8 in multi-GPU configurations.
7. Validate the Software Stack with Real Tests, Not Datasheets
Hardware is only the foundation; the software stack determines whether the GPU can actually be used. When selecting, confirm the following:
- Whether native Linux drivers are provided, since the vast majority of server and inference environments are Linux-based;
- Compatibility with the inference framework and version you plan to use;
- Whether operator coverage for your target quantization format is complete;
- Whether the multi-GPU parallelism modes meet your business needs.
Before purchasing, run small-scale tests with your target model and record: model loading time, peak VRAM usage including long contexts, tokens generated per second at target concurrency, and stability over several hours of continuous operation. Lock the driver and framework versions before going live. Upgrades can cause operator fallbacks or performance changes, so keep a rollback plan.
8. Selection Steps
- List your models and record each one’s parameter count, quantization precision, target concurrency, and context length.
- Estimate VRAM. Weight VRAM ≈ parameter count × bytes per parameter (1 for INT8, 2 for FP16, 0.5 for INT4). Calculate the KV Cache with the formula above. Reserve headroom for runtime overhead, and use the peak VRAM measured in stress tests as the final reference.
- Compare compute at the same precision, and check memory bandwidth at the same time.
- Choose the cooling form factor based on chassis conditions and deployment density. For dense multi-GPU setups, prefer blower-style coolers.
- Run compatibility and stress tests with the real model and inference framework.
- Plan a hybrid architecture: assign elastic and peak workloads to the cloud, and keep stable baseline workloads local.
9. Conclusion
There is no universally optimal GPU choice. Whether an option is cost-effective depends on how well its VRAM, bandwidth, cooling, and software ecosystem match your actual workload. Calculate your VRAM requirements first, compare compute and bandwidth at the same precision, and verify with real-world testing — this process avoids most selection mistakes.
