How Extremely Low-Bit Quantization Reduces LLM Hallucinations

1. The Core Problem: Why Low-Bit Models Are Prone to Hallucinations

When Large Language Models (LLMs) generate factual errors, contradictions, or fabricated content, it is rarely due to a single cause. For extremely low-bit quantized models, this issue is amplified, primarily stemming from three key areas.

1. Blurred Knowledge Boundaries Caused by Weight Information Loss

Original models store weights at high precision, such as FP16, BF16, or FP32. Each weight value carries specific knowledge association strength. When a model is compressed to an average of less than 2 bits per weight, the number of representable numerical states drops drastically.

Take ternary quantization as an example: each weight can only take one of three states. This means the originally continuous or semi-continuous weight space is discretized into very few values. If the original weights contained many values close to zero but not exactly zero, ternarization flattens these subtle differences. When generating text, the model originally relied on these nuances to determine “which fact is more credible” or “which transition is more logical.” Once this information is lost, the output tends to become ambiguous, random, or incorrect.

More critically, weight quantization errors accumulate layer by layer in the Transformer architecture. Minor perturbations in the early layers are amplified through attention mechanisms, residual connections, and feed-forward networks, eventually evolving into noticeable semantic drift in the deeper layers. This explains why many low-bit models perform acceptably on simple Q&A but suffer logical breakdowns during long-text reasoning, multi-hop questioning, or complex code generation.

2. Aggressive Truncation of Outliers

The weight distribution in deep neural networks is rarely uniform. While most weights cluster near zero, there are a small number of “outliers” with very large absolute values. Though few in number, these outliers play a critical role in attention calculations.

For instance, in a specific attention head’s weight matrix, less than 1% of the values might have an absolute value greater than 0.5, but these exact values control key behaviors like “introducing a transition” or “establishing long-range dependencies.” If the quantization strategy does not protect them and directly truncates these large values or maps them to the nearest ternary state, the attention distribution will shift dramatically.

In this scenario, the model hasn’t become “less intelligent”; rather, certain critical logical switches have been erroneously turned on or off. This manifests in the generated output as more frequent factual errors, logical leaps, and contextual contradictions—what users perceive as hallucinations.

3. Significantly Weakened Long-Context Capabilities

Modern LLMs generally support long context windows, such as 128K, 256K, or more. Long-context capability relies heavily on maintaining sufficient resolution in attention scores between distant positions. However, attention scores are computed via the dot product of Query and Key vectors, making them sensitive to weight precision.

In extremely low-bit scenarios, the weights for Query, Key, and Value are all quantized. If a Key vector’s similarity to a distant Query is incorrectly reduced after quantization, the model might ignore crucial preceding information. Conversely, if the similarity is artificially inflated, the model might incorrectly focus on irrelevant content. Issues in long-text generation like “forgetting earlier statements” or “losing key facts midway” largely originate here.

Therefore, determining the reliability of an extremely low-bit quantized model requires looking beyond aggregate benchmark scores. It is crucial to evaluate its performance specifically on long-text tasks, multi-hop reasoning, and factual consistency.

---

2. Key Technical Approaches to Reduce Hallucinations

Reducing output distortion from extreme low-bit quantization requires more than a single quantization strategy. The current mainstream approach in the industry can be categorized into three layers: transforming the weight space pre-quantization, protecting critical information during quantization, and enhancing fault tolerance during inference.

1. Rotation-Based Space Projection: Mitigating Outlier Impact at the Root

Traditional quantization operates directly on the original weight space. For example, applying uniform quantization or simple binarization/ternarization to a weight matrix. The drawback is that the original space may contain numerous outliers, and direct quantization leads to severe information loss.

The idea behind rotation-based space projection is to apply an orthogonal transformation to the weight matrix before quantization. Orthogonal transformations are norm-preserving—they do not change the matrix’s Frobenius norm, but they do alter the distribution shape of the values in that space.

Simply put, if a small fraction of elements in the original weight matrix are exceptionally large, an orthogonal rotation disperses these extreme values across a broader range of dimensions. The overall weight distribution becomes smoother and closer to a “quantization-friendly” form. Performing ternarization or binarization at this stage significantly reduces information loss.

It is important to note that this rotation is not arbitrary. To ensure correct inference calculations, the rotation must be combined with corresponding transformations for the activation values. The standard practice is to left- and right-multiply the weight matrix by different orthogonal matrices while adjusting the activations accordingly. This way, although both weights and activations are represented in the new basis, the final computation result can be mapped back to the original space.

The core value of this technical path is that it mathematically changes the distribution characteristics of quantization errors. In traditional methods, errors concentrate in the few dimensions containing outliers, which often heavily impact semantics. After rotation, errors are uniformly dispersed across all dimensions, reducing single-point error and stabilizing the model’s overall output.

2. FP16 Group Scaling: Preserving Magnitude Information During Discretization

Pure ternarization has an obvious flaw: it only retains “direction” and “zero,” almost entirely losing the “magnitude” information of the values. Even if rotation-based projection mitigates outlier effects, numerical range differences across different layers and channels still exist.

The core concept of group scaling is to divide the weight matrix into several groups, calculating a separate scaling coefficient for each group and storing it at higher precision. The specific workflow is as follows:

1. Group weights by row or block;
2. Ternarize the weights within each group to obtain discrete values (-1, 0, +1);
3. Calculate the scaling factor for each group so that the scaled quantized result closely approximates the original weights;
4. During inference, multiply the ternary weights by the scaling factor before computing with the activations.

This way, while individual weights remain ternary, different groups maintain distinct magnitude ranges via their scaling factors. These scaling factors are typically preserved in FP16. They account for a minuscule fraction of the parameter count but largely compensate for the magnitude loss caused by ternarization.

From an output stability perspective, group scaling preserves the relative importance of different feature channels. Without scaling factors, certain channels with originally small magnitudes but critical roles in generation quality could be excessively amplified or diminished, causing attention calculation biases. With scaling, the model can more accurately judge which information is primary and which is secondary during generation, thereby reducing erroneous focus and contextual loss.

3. Visual Module Decoupling and Multimodal Integration

The model discussed also involves multimodal input capabilities. In multimodal scenarios, the vision encoder and the language model backbone are usually independent modules. Visual features enter the language model after projection. If the vision module itself is not overly compressed, the language backbone can retain strong image-text understanding capabilities even after extreme low-bit quantization.

Decoupling and encapsulating the vision module means visual feature extraction can maintain high precision, while the language backbone employs ternary quantization. The advantage of this design is that visual feature quality is not noticeably degraded by the language backbone’s quantization. The model retains robust information acquisition capabilities in vision-language tasks. Whether the final output hallucinates depends more on whether the language backbone can correctly utilize the visual features. If the visual features themselves are intact, and the critical attention pathways are not severely damaged, the generation quality is generally acceptable, even if the language backbone has some quantization errors.

---

3. Comparative Concepts and Evaluation Methods in Practice

In actual engineering, developers often need to choose between multiple candidate models. To determine if an extremely low-bit quantized model is less prone to hallucinations, evaluation should be conducted across several dimensions.

1. Factual Consistency Evaluation

Factual consistency primarily examines whether the model’s answers contradict known facts. You can prepare a set of Q&A pairs containing precise facts, such as historical event dates, geographical information, or technical parameters. Input the same questions into two models and compare:

• The number of entirely correct answers;
• The number of answers with factual errors but coherent logic;
• The number of answers with obvious fabrications or contextual conflicts.

Factual errors are often linked to over-quantization of knowledge weights. If a model frequently makes mistakes on a specific category of facts, it indicates that the corresponding weight information in that domain was severely damaged during quantization.

2. Long-Text Logic Tracking Capability

For long-text tasks, design multi-hop reasoning or contextually linked Q&A. For example:

• Embed several interconnected facts within a long document;
• Ask the model to reference preceding text when answering follow-up questions;
• Check if the answer exhibits “forgetfulness,” “contradictions,” or “repetitive fabrication.”

These tests demand highly accurate attention chains. If the attention scores deviate significantly post-quantization, the model will struggle to maintain stable information transmission over long distances, and performance will degrade noticeably.

3. Generation Stability Testing

Generation stability primarily observes whether the model produces a high volume of low-quality outputs under different sampling parameters. With temperature and top-p fixed, sample multiple times and calculate:

• Output repetition rate;
• Frequency of logical breakdowns;
• Proportion of factual errors;
• Proportion of contradictory expressions.

If a model is highly sensitive to sampling parameters—experiencing massive quality collapse with a slight temperature adjustment—it usually indicates insufficient robustness in its internal representations. An extremely low-bit quantized model, if processed correctly, should maintain stable output quality across reasonable sampling ranges.

4. Discrepancy Between Benchmark Scores and Real-World Experience

Aggregate benchmark scores serve as an initial filter but cannot fully replace real-world scenario evaluation. Many benchmark tests are short and straightforward; a model might arrive at the correct answer using remaining information even with localized attention deviations. In real-world scenarios, however, user prompts are often longer, more complex, and demand higher contextual consistency.

Therefore, when selecting a model, it is recommended to comprehensively evaluate:

• Short Q&A and single-turn dialogue performance;
• Long-text and multi-turn dialogue performance;
• Code generation and logical deduction performance;
• Factual Q&A and open-ended generation performance.

If a model scores high on short Q&A but frequently goes off-track in long-text tasks, its actual business viability may be suboptimal.

---

4. Additional Considerations for Engineering Deployment

1. Inference Framework Support for Quantization Kernels

Whether an extremely low-bit model runs stably depends heavily on how well the inference framework adapts to the quantization compute kernels. Even if the model’s weights are well-preserved, errors in the implementation of de-quantization, scaling calculations, or attention operators during inference can introduce additional numerical instability.

When evaluating a model, conduct end-to-end testing in the target inference environment rather than relying solely on offline metrics provided by the model publisher. The same quantized model can exhibit visible differences in generation quality across different inference backends.

2. Memory Bandwidth and Edge Deployment Advantages

From an engineering perspective, compressing a model from tens of gigabytes to a few gigabytes allows more edge devices to load it within limited memory. This offers practical value in latency-sensitive, network-restricted, or highly privacy-conscious scenarios. However, it must be clear that the size advantage brought by compression should not come at the cost of a severe degradation in output quality. When selecting a model for critical business operations, prioritize output reliability.

3. Continuous Monitoring of Generation Quality

After a model goes live, establish an output quality monitoring mechanism. For example:

• Sample and track the proportion of factual errors in user requests;
• Monitor context loss rates in multi-turn dialogues;
• Implement manual sampling or automated validation for high-risk scenarios.

The degradation patterns of quantized models sometimes differ from those of full-precision models. Continuous monitoring enables early detection of output quality anomalies, allowing timely adjustments to model versions or inference parameters.

---

5. Conclusion

Whether an extremely low-bit quantized model is prone to hallucinations fundamentally depends on whether the quantization strategy effectively protects the critical information governing semantics and logic. Rotation-based space projection, group scaling, visual module decoupling, and inference chain stability optimization are vital tools for reducing output distortion.

When choosing between two low-bit models, one should not look at isolated metrics. A comprehensive assessment combining factual consistency, long-text logic, multi-turn dialogue, and generation stability is required. For demanding business scenarios, the size advantages of compression must be weighed against output reliability to make decisions suitable for real-world applications.