FP8 + INT4 Mixed-Precision Quantization for Large Language Model Inference
This article explains how to reduce large language model inference costs through FP8 and INT4 mixed quantization. It covers the trade-offs between low-bit formats, techniques for preserving model accuracy, hardware-aware deployment considerations, implementation patterns, and practical guidelines for turning theoretical compression into measurable memory, bandwidth, and cost savings.
1. Why LLM Inference Costs Need to Be Rethought
The cost of large language model inference is driven mainly by three factors: memory capacity, memory bandwidth, and compute throughput. In autoregressive generation, the system must repeatedly move the model’s weights from memory to the compute units as each token is generated. As the model grows to tens of billions of parameters or more, this data movement becomes increasingly expensive.
Even with FP16 or BF16, the weights alone can consume several gigabytes or tens of gigabytes of memory. KV cache, activations, and framework runtime overhead add further pressure. A single accelerator may no longer be sufficient, forcing the deployment to use model sharding across multiple devices and introducing additional communication and scheduling costs.
The core idea behind inference cost reduction is therefore not simply to add more hardware. It is to reduce the number of bits required to represent each parameter, thereby lowering memory usage and data movement while preserving generation quality as much as possible. This is the fundamental reason low-bit and mixed-precision quantization have attracted so much attention.
2. From Uniform Quantization to FP8 + INT4 Mixed Quantization
Quantization maps high-precision floating-point weights to lower-bit representations and uses scale factors to reconstruct their numerical values. Different low-bit formats involve different trade-offs:
- FP8: Retains the dynamic-range advantages of floating-point representation. It is more tolerant of outliers, generally provides good numerical stability, and usually causes limited accuracy degradation. It is suitable for layers that are sensitive to quantization error.
- INT4: Uses only 4 bits per parameter, providing a very high compression ratio and significant bandwidth savings. However, its narrow dynamic range can cause noticeable accuracy degradation if it is applied uniformly across the entire model.
The idea behind mixed quantization is to assign precision according to the characteristics of each layer or tensor. Sensitive components, such as attention projections and output projections, can use FP8 to preserve accuracy. More error-tolerant linear layers can be quantized to INT4 to reduce memory and bandwidth requirements. Embedding layers and LayerNorm are typically kept at higher precision rather than quantized.
This approach seeks the best balance between accuracy and cost under a fixed overall bit budget.
3. Key Techniques for Preserving Accuracy
Whether low-bit quantization can maintain nearly unchanged accuracy depends less on the format itself than on the engineering details of the quantization process.
- Per-channel or grouped quantization: Calculating a scale for each channel or small group represents distribution differences more accurately than using one scale for an entire tensor. This can substantially reduce INT4 quantization error.
- Outlier handling: Extreme values in activations and weights can be addressed through techniques such as channel scaling and smooth migration. These methods move difficult-to-quantize dynamic ranges into tensors that can represent them more effectively.
- Calibration datasets: A small but representative dataset can be used for forward-pass statistics to estimate activation distributions and determine scale factors. This helps avoid calibration bias caused by data that does not reflect the actual inference distribution.
- Sensitive-layer fallback: If a small number of layers show significant accuracy degradation after quantization, they can be restored to a higher precision, such as FP8 or FP16. This recovers critical quality with limited additional cost.
A typical workflow begins with small-scale calibration and evaluation. The output difference before and after quantization is then compared layer by layer to identify sensitive layers. The deployment can subsequently balance the overall compression ratio against the need to preserve precision in critical parts of the model.
4. The Value of Model–Hardware Co-Design
Quantized weights alone are not enough. The actual cost reduction depends on whether the underlying hardware and operators are properly aligned with the quantized model.
General-purpose GPU ecosystems are mature, but on specialized accelerator hardware, the benefits of quantized inference often depend on three factors:
- Low-bit operator support: The target hardware should provide native INT4 and FP8 matrix multiplication and dequantization operators. If low-bit data must be converted back to higher precision for computation, the resulting overhead may offset the benefits of compression.
- Compute and memory-movement coordination: The primary value of low-bit formats is lower memory bandwidth pressure. Operators must be optimized for the accelerator’s memory hierarchy and data layout to make effective use of the available bandwidth.
- An integrated quantization, graph optimization, and scheduling pipeline: The workflow should cover quantized model export, graph compilation, operator fusion, and batch scheduling. This reduces runtime format conversions and unnecessary data copies.
A model–hardware co-designed solution aligns the quantized model with the accelerator’s operators, instruction set, and memory characteristics. This is what converts a theoretical compression ratio into measurable throughput and cost improvements. It is also one of the most important considerations when deploying low-bit LLM inference on specialized accelerator platforms.
5. Implementation Example
The following pseudocode illustrates the configuration and execution model for mixed quantization. It is framework-independent and is intended only to demonstrate the overall logic:
# Mixed-quantization configuration: assign a bit budget based on layer sensitivityquant_config = {"attn_proj": "fp8", # Attention projections: FP8 for dynamic range"mlp_up_proj": "int4", # Selected MLP layers: INT4 for bandwidth savings"mlp_down_proj": "int4","lm_head": "fp8", # Output projection: FP8 for generation quality"embedding": "fp16", # Embedding layer: keep unquantized}def quantize(weight, mode, group_size=128):if mode == "fp16":return {"q": weight, "scale": 1.0}# Compute scales by group for a balance between compression and stabilityscales, q = [], []for g in chunk(weight, group_size):s = amax(g) / quant_max(mode) # Set the scale according to the format rangescales.append(s)q.append(quantize_group(g, s, mode))return {"q": concat(q), "scale": stack(scales)}def forward(x, layer):qw = quantize(layer.weight, quant_config[layer.name]) # Quantize offline and cacheacc = low_bit_matmul(x, qw["q"], qw["scale"]) # Execute with a low-bit operatorreturn acc + layer.bias
A general deployment checklist includes:
- Confirm that the target accelerator provides native operators for the selected low-bit formats. If it does not, run benchmarks first to verify whether quantization actually improves performance.
- Use a calibration dataset to collect scale statistics, then evaluate consistency between the original and quantized models.
- Restore higher precision for layers that show significant quality degradation.
- Optimize batching and KV cache usage to amortize the cost of moving weights for individual requests.
6. Best Practices and Key Takeaways
Low-bit mixed-precision inference is not a single optimization technique. It is an end-to-end process that combines format selection, accuracy preservation, operator optimization, and scheduling.
- Measure before compressing: Establish an evaluation baseline before applying aggressive quantization, then compare the original and quantized models layer by layer.
- Mixed precision is usually better than a uniform format: Combining FP8 and INT4 is generally more stable than applying INT4 to the entire model, while requiring less memory and bandwidth than using FP8 everywhere.
- Operators determine the practical ceiling: Whether low-bit quantization translates into real cost savings depends on native hardware support, operator efficiency, and memory-access optimization.
- Integration is essential: Coordinating the model with the target hardware is the prerequisite for turning a theoretical compression ratio into actual production savings.
For teams running LLMs on specialized accelerator hardware, a practical understanding of mixed quantization and model–hardware co-design can significantly reduce memory consumption and per-inference cost without sacrificing output quality. This provides a solid engineering foundation for sustainable large-scale deployment.
