Why Extreme Low-Bit Ternary Quantization Makes LLMs Less Prone to Nonsense: Principles and Practical Analysis
This article explains how ternary quantization, which stores each weight as -1, 0, or +1, can shrink a large language model to under 2 bits per parameter while preserving much of its capability. It covers why outliers make extreme low-bit quantization fail, how rotated weight-basis projection and group-wise FP16 scaling address that problem, and how to evaluate whether a quantized model is more reliable, including tests for factual accuracy, long-context retrieval, and instruction following.
1. Background: The Size Problem in Model Deployment
A 27B-parameter open-source model needs roughly 27 GB or more of GPU or system memory when stored in FP8. In BF16, the full-precision format, it needs close to 54 GB. For developers who want to run models locally, on edge devices, or on low-cost inference servers, this is a hard barrier.
The usual remedy is quantization, which compresses weights from 16 bits to 8 bits, 4 bits, or even 2 bits. In practice, 4-bit models often remain usable. At 2 bits, however, models frequently start producing incoherent output: their logic breaks down, facts get mixed up, and instruction following falls apart. This raises a central question:
Why does extreme low-bit quantization make models worse, and is it possible to reduce weights to fewer than 2 bits per parameter without losing reasoning ability?
Recent work on ternary quantization offers a possible answer. The approach restricts weights to three values, {-1, 0, +1}, and combines group-wise scaling with a rotated basis transformation. Reportedly, this retains more than 98% of capability on a 27B-class model while reducing its size from about 54 GB to under 6 GB. This article does not evaluate or recommend any specific product. Instead, it takes a general technical view, breaks down the core principles of this class of methods, and discusses how to scientifically assess whether a quantized model produces less nonsense.
2. Why Extreme Low-Bit Quantization Hurts Model Quality
To understand the difficulty of ternary quantization, start with a key property of LLM weight distributions: outliers.
1. Outliers: The First Cause of Quantization Failure
Most neural network weights cluster in a narrow range around zero. However, deep attention networks often contain a small number of dimensions with very large activation values. These outliers are rare, but they play a critical role in determining the model’s logic. They control how information is routed through attention matrices. If outliers are flattened, the deep reasoning chain becomes disordered.
Traditional 2-bit quantization uses linear mapping, which rounds weights to a small set of discrete levels. This creates a dilemma:
- If the representation range is widened to accommodate outliers, the quantization error for ordinary weights increases sharply.
- If the scale is set to fit ordinary weights, outliers saturate and get clipped, and the critical signals they carry are lost.
This is the mathematical root of the “dumb model” problem.
2. Two Approaches to the Information Bottleneck
Researchers have developed two main approaches to the outlier problem:
- Mixed-precision approach: Keep sensitive layers at high precision and quantize the rest to low bit widths. Its drawbacks are a limited compression ratio and more complex engineering.
- Global transformation approach: Apply a mathematical transformation to the weights before quantization to remove outliers at the source. Ternary quantization methods generally belong to this category.
3. The Two Core Techniques of Ternary Quantization
1. Rotated Weight Basis Projection
This technique targets outliers directly. Before quantization, an orthogonal transformation, usually a Hadamard-type transform, is applied to the weight matrix. This projects the weights into a rotated basis space.
Orthogonal transformations have two key mathematical properties:
- They preserve mathematical equivalence. If the weight matrix W is transformed to W’ = WQ, the corresponding transformation is applied to the input activations during the forward pass. The result is mathematically equivalent to the original model. The transformation itself adds no precision loss.
- They smooth the distribution. Extreme spikes concentrated in individual dimensions are spread evenly across the feature space. After the transformation, the weight distribution is closer to a Gaussian shape, which removes the outlier problem at its mathematical source.
This step is a prerequisite for quantizing to ternary values. Without rotated projection, direct ternarization almost always disrupts attention matrices.
2. Ternary Representation with FP16 Group-Wise Scaling
In its basic form, ternary quantization allows each weight to take only the values {-1, 0, +1}. However, ternary values alone cannot express weight magnitude. Practical methods therefore use group-wise scaling:
- Weights are divided into groups by channel or block, for example 64 or 128 weights per group.
- Each group stores one FP16 scale factor, s.
- Weights within a group are stored as ternary indices. During dequantization, each index is multiplied by s to recover the magnitude.
The effective bit width works out as follows. Ternary indices need about 1.58 bits, which is log2(3). Adding the amortized overhead of the scale factors brings the average to about 1.76 bits per weight. Compared with 16 bits for FP16, this gives a compression ratio of more than 9x.
Inference also gains an additional benefit. Ternary weights can be processed with lookup tables and integer additions and subtractions instead of multiplications. With dedicated kernels, this can substantially reduce memory bandwidth pressure, which matters most on devices with limited memory.
4. How to Evaluate Whether a Quantized Model Produces More Nonsense
Return to the original question: which is more reliable, a low-bit quantized model or a higher-precision model of medium size? This cannot be answered by intuition. It requires a systematic evaluation.
1. Recommended Evaluation Dimensions
- General capability benchmarks: Use standard benchmark suites for knowledge question answering, mathematical reasoning, and code generation. Compare scores before and after quantization and calculate the retention rate.
- Hallucination tests: Build a factual question set and measure the proportion of false statements. Quantization often damages factual recall more than reasoning ability.
- Long-context consistency: Test retrieval accuracy in “needle in a haystack” tasks with very long context windows, such as 128K tokens or more. This checks whether scaling and quantization have damaged long-range attention.
- Instruction following and multi-turn stability: Observe whether formatting drifts or the model fails to answer the question after several turns of dialogue.
2. Practical Selection Guidance
- For tasks centered on factual question answering and knowledge density, validate memory-related capabilities carefully. Keep key layers at high precision if necessary.
- For tasks centered on logical reasoning and code, quantization methods that use rotated basis transformations often retain more capability and are worth testing first.
- Always compare against the original-precision version of the same base model. Comparing models from different base models mixes base-model differences into the results and makes the conclusions unreliable.
5. Conclusion
At its core, extreme low-bit quantization is a fight against outlier distributions at the limits of information theory. Rotated basis projection smooths the weight distribution mathematically, and group-wise scaling preserves magnitude information. Together, they move ternary quantization from near-certain failure toward near-lossless results. For developers, understanding these principles supports better model selection. It also points to a simple lesson: judging whether a model “produces nonsense” depends on a systematic evaluation process, not on intuitions about parameter count or file size.
