LLM Precision Format Selection: A Complete Guide from BF16 to MXFP4
As large language models scale into the hundreds of billions of parameters, numeric precision stops being an academic detail and becomes a hard engineering constraint. This guide walks through why precision matters, traces the quantization path from FP32 to FP16/BF16, FP8/INT8, and sub-8-bit MXFP4, breaks down microscaling and the E2M1 format, and offers a practical decision framework for choosing the right format based on training versus inference, memory and bandwidth limits, precision tolerance, and hardware support.
1. Why Precision Format Is a Core Engineering Problem for LLMs
When a model’s parameter count climbs into the hundreds of billions, compute precision is no longer just an academic question of “numerical representation.” It becomes an engineering constraint that directly determines whether a project can ship at all. Take a 671B-parameter model: if the weights are stored in 16-bit floating point, the weights alone need roughly 1.4TB of VRAM, which even an eight-GPU high-end server cannot fully accommodate. This exposes three bottlenecks that cannot be ignored:
| Bottleneck | Underlying Problem | Typical Scenario |
|---|---|---|
| VRAM capacity | Weight storage demand far exceeds single-card capacity | A 100B+ model in 16-bit needs about 1.4TB; multi-card deployment is still difficult |
| Memory bandwidth | During autoregressive decoding, every generated token requires a full read of the weights | Inference speed is constrained by VRAM bandwidth rather than compute |
| Communication bandwidth | Once a very large model is sharded across cards, activations must be repeatedly broadcast between devices | All-Reduce and communication during distributed training significantly increase latency |
This is exactly where precision formats matter: every bit removed reduces weight size, VRAM usage, and bandwidth pressure proportionally, letting a larger model fit into less hardware. But the lower the precision, the worse the numerical distortion, and that tension is the core of the entire selection logic.
2. The Quantization Roadmap: From FP32 to Sub-8-Bit
The industry’s precision evolution roughly follows a clear ladder:
FP32 → FP16 / BF16 → FP8 / INT8 → FP4 (MXFP4, etc.)32-bit 16-bit 8-bit 4-bitGEMM cost halved distortion grows sub-8-bit is very hard(training +70%) (hard to hold (depends on fine-grainedprecision) quantization strategies)
This is not a simple “lower is always better” progression. Each step requires a matching numerical compensation strategy. Let’s break it down segment by segment.
3. Mixed Precision: The First Leap with BF16 and FP16
Moving from FP32 to 16-bit is the first key leap in training. It directly halves the compute cost of matrix multiplication (GEMM) and can improve end-to-end training throughput by more than 70%, and it is now the default configuration for large-model training.
Within the 16-bit camp, the trade-off between BF16 and FP16 is worth unpacking:
- FP16 (1 sign + 5 exponent + 10 mantissa): high mantissa precision, suitable for scenarios with a narrow numeric range, but a narrow dynamic range means large gradients overflow easily, so it must rely on Loss Scaling to protect small gradients.
- BF16 (1 sign + 8 exponent + 7 mantissa): the exponent width is identical to FP32, giving it a very large dynamic range and almost no overflow, with good numerical stability. It is the safer choice when training large models, at the cost of slightly lower mantissa precision.
The common practice is: keep master weights and optimizer states in FP32 to preserve update precision, and use BF16 to accelerate the matrix multiplications in the forward and backward passes. This is the typical mixed-precision training mode.
4. Moving to 8-Bit: The Art of Scaling with FP8 and INT8
Once you reach 8-bit, the numeric representation space is heavily compressed and quantization distortion begins to grow sharply. At this point, “scaling accurately” matters more than “compressing aggressively.” The industry has settled on two mainstream paths:
- FP8: usually subdivided into E4M3 (4 exponent + 3 mantissa) and E5M2 (5 exponent + 2 mantissa). The former offers higher precision and a narrower range, suitable for forward-pass weights and activations; the latter offers a wider range and lower precision, suitable for backward-pass gradients. In recent years, several widely followed open-source large models have adopted FP8 for native training and inference, demonstrating that low-precision training at scale is entirely feasible with fine-grained strategies.
- INT8: integer quantization, hardware-friendly with a mature ecosystem, commonly used for inference deployment. The core difficulty is again how to choose a suitable scale factor for each tensor so that outlier-induced precision collapse is suppressed within a limited dynamic range.
The granularity of the scale factor is the deciding factor in 8-bit quantization. There are three main approaches:
| Scaling Method | Granularity | Pros | Cons |
|---|---|---|---|
| Per-Tensor | One scale factor shared across the entire tensor | Simple to implement, low overhead | Too coarse; struggles to balance large and small values; nearly infeasible at sub-8-bit |
| Per-Channel | One scale factor per output channel | Finer granularity, better accuracy than per-tensor | Still not robust enough to extreme outliers in activations |
| Microscaling | One scale factor shared by every 32 elements | Matches local dynamic range, effectively suppresses outliers, balances accuracy and hardware efficiency | Requires extra storage for scale factors, a slight overhead |
A useful rule of thumb: when the model weights show a typical outlier distribution of “a few very large values plus many values near zero,” the finer the scaling granularity, the better it protects numerical precision. This is precisely why the microscaling approach has become widely adopted in the sub-8-bit era.
5. Exploring 4-Bit: MXFP4 and Microscaling Formats
When precision drops to 4-bit, per-tensor and per-channel scaling largely break down, and microscaling becomes the only viable practical solution. The idea is to split the tensor into blocks of a fixed size (usually 32), give each block its own high-precision shared scale factor, and then represent the elements within the block using a low-bit format.
Building on the publicly available microscaling format specifications, the MXFP4 element format is E2M1, structured as follows:
- 1 sign bit: indicates positive or negative;
- 2 exponent bits: with an exponent bias of 1, providing a limited dynamic range;
- 1 mantissa bit: extremely low precision, so the mantissa can take only a few values.
E2M1 does not encode any special values (no NaN or Infinity), but when the shared scale factor (typically in E8M0 format, i.e., an 8-bit pure exponent) reaches all ones (0xFF), it can mark the entire block as NaN for exception propagation.
E2M1 has 16 encodings (4 bits), 8 positive and 8 negative. Taking the positive values as an example, all representable numbers are as follows:
| Encoding (binary) | S | E | M | Decode Formula | Value |
|---|---|---|---|---|---|
| 0000 | 0 | 00 | 0 | — | 0 |
| 0001 | 0 | 00 | 1 | 2^(1-1) × (1/2) | 0.5 |
| 0010 | 0 | 01 | 0 | 2^(1-1) × 1.0 | 1.0 |
| 0011 | 0 | 01 | 1 | 2^(1-1) × 1.5 | 1.5 |
| 0100 | 0 | 10 | 0 | 2^(2-1) × 1.0 | 2.0 |
| 0101 | 0 | 10 | 1 | 2^(2-1) × 1.5 | 3.0 |
| 0110 | 0 | 11 | 0 | 2^(3-1) × 1.0 | 4.0 |
| 0111 | 0 | 11 | 1 | 2^(3-1) × 1.5 | 6.0 |
As you can see, the representable values of a 4-bit float are extremely sparse (only 8 positive levels, with a maximum of 6.0), and the spacing between adjacent values is uneven: dense near small values and sparse near large values. This “logarithmic” distribution happens to match the weight characteristics of neural networks, where most values are small with a few large ones. Combined with a per-block scale factor, it can keep overall distortion within an acceptable range even at extremely high compression ratios.
Different hardware ecosystems vary slightly in the specific encoding for FP4 microscaling (for example, the scale-factor bit width and block size), but the core paradigm is the same: back it up with a high-precision shared scale factor and compress with a low-bit element format. The following illustrative code shows the quantization idea behind microscaling:
import torch# All representable positive levels of E2M1 (FP4)E2M1_GRID = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])def quantize_microscaling(x: torch.Tensor, block_size: int = 32):"""Microscaling quantization: every block_size elements share one scale factor.Steps: split into blocks -> find block max -> compute scale -> normalize -> map to E2M1 levels"""flat = x.reshape(-1, block_size)block_max = flat.abs().amax(dim=-1, keepdim=True) # max absolute value within the blockscale = block_max / 6.0 # map to the max E2M1 level, 6.0scale = torch.where(scale == 0, torch.ones_like(scale), scale) # avoid division by zeronormalized = (flat / scale).unsqueeze(-1) # normalize to [0, 6]# Nearest-neighbor rounding: find the closest representable E2M1 leveldist = (normalized - E2M1_GRID).abs()q_abs = E2M1_GRID[dist.argmin(dim=-1)].reshape_as(flat)q = torch.sign(flat) * q_abs # restore the signdequant = q * scale # dequantize, for accuracy evaluationreturn dequant.reshape_as(x), scale
6. A Decision Framework for Selecting Precision Formats
Back to the original question: when facing BF16, FP8, INT4, and MXFP4, how should you decide? Judge layer by layer across the following four dimensions:
- Training or inference? Training is extremely sensitive to backward gradients and numerical stability, usually starting with BF16 and advancing to FP8; inference is more sensitive to latency and VRAM, where the benefits of INT8/INT4/FP4 are more direct.
- Are VRAM and bandwidth hitting their ceiling? If a single card cannot hold the model or inference is severely constrained by bandwidth, lowering precision is almost the only cost-effective option, and sub-8-bit solutions should be prioritized.
- What is your precision-loss tolerance? Compare key metrics on a validation set (such as perplexity and downstream task accuracy) before and after quantization, set an acceptable threshold, and then choose the lowest bit width that still meets it.
- Does the hardware natively support it? Formats like microscaling and FP8 only deliver acceleration value on accelerator cards with the corresponding instruction sets. Otherwise, quantization saves memory but not necessarily compute, so it needs to be weighed carefully.
7. Practical Implementation Recommendations
- Build on mixed precision: keep master weights and optimizer states at high precision and use low precision on the compute path, balancing stability and speed.
- Prioritize granularity over bits: below 4-bit, favor fine-grained scaling like microscaling rather than chasing a more extreme representation.
- Handle outliers specifically: separately preserving or smoothing extreme outlier channels in activations is a common and critical technique for low-precision quantization.
- Quantization-Aware Training (QAT): for accuracy-sensitive scenarios, introduce fake-quantization nodes during training so the model adapts to the low-precision distribution. This is significantly better than post-training quantization (PTQ).
- Close the end-to-end evaluation loop: always judge whether a quantization scheme meets the bar using real business metrics, not just raw numerical error.
8. Summary
Selecting a precision format is essentially a three-way trade-off among “capacity, bandwidth, and accuracy.” From the mixed precision of BF16, to the scaling-granularity debate of FP8/INT8, to the sub-8-bit exploration of MXFP4 microscaling, every drop in bits comes with an upgrade in compensation strategies. Truly professional selection is not about blindly chasing the lowest bit width, but about building a measurable, rollback-friendly decision process based on the training/inference scenario, hardware capability, and precision tolerance. Master this framework, and developers can make well-grounded engineering trade-offs between cost and quality.
