Low-Bit Quantized LLM Selection: Which Model Hallucinates Less?

In the practice of deploying large language models (LLMs), a recurring question arises: how should we choose between a compact, runnable model and a reliable one that minimizes hallucinations? When comparing ultra-low bit ternary models with high-precision FP8 quantized models, many developers instinctively assume the former is inherently worse. However, the reality depends on how the quantization algorithm handles the internal structure of the neural network, rather than just the bit-width number.

Rather than ranking specific products, this article frames the issue as a general technical question: Given a similar parameter count, which is less likely to produce factual errors and logical confusion—an average 1.7~2 bit ternary model or an 8-bit floating-point quantized model? We will start with the mechanics of quantization failure and then provide actionable selection and validation methods.

1. Why Ultra-Low Bit Quantization Causes Hallucinations

To determine which model hallucinates less, we must first understand the technical origins of “hallucinations” in the context of quantization. The factual and reasoning capabilities of LLMs rely heavily on the distribution of activation values in attention matrices and feed-forward layers. Deep networks contain abnormal outliers: values that are highly concentrated in certain dimensions, numerically far exceeding the mean, yet decisive for logical flow.

If we apply naive uniform quantization and forcefully round these weights to a few discrete values (e.g., the ternary set {-1, 0, +1}), the outliers are directly flattened. The consequence is not a “slight overall accuracy drop” but structural damage: the sharp peaks of the attention distribution are clipped, and the ability to model long-range dependencies degrades. The model then begins to exhibit typical hallucination symptoms, such as answering irrelevantly, fabricating details, and repeating boilerplate text.

Therefore, the key to evaluating the reliability of a low-bit model is whether it handles outliers before quantization, not how much it compresses the size.

2. Three Key Technical Stages of the Ternary Approach

Taking common ternary quantization schemes as an example, their reliability usually depends on three critical stages, all of which are indispensable.

1. Rotational Basis Space Projection

Before quantization, an orthogonal transformation is applied to the weight matrix, projecting the weights into a “rotational basis” space. Orthogonal transformations preserve vector lengths and inner products, so they do not alter the model’s mathematical expressive power. However, they can uniformly disperse the spiky energy concentrated in a few dimensions across the entire feature space. This step mitigates the outlier problem at its mathematical root, ensuring that subsequent ternary quantization does not indiscriminately destroy critical activations.

2. Group-wise Scaling

Ternary quantization retains only sign information, while the numerical magnitude is carried by scaling factors. In practice, weights are grouped by channel or block, with each group maintaining a high-precision (e.g., FP16) scaling coefficient. This way, the ternary pattern is shared within the same group, while different dynamic ranges are preserved across groups. The effective bit-width can be reduced to below 2 bits while avoiding the systematic bias caused by global uniform scaling.

3. Residual Compensation and Calibration

Quantization error is inevitable. In engineering, it is common practice to introduce low-rank residuals or a small number of high-precision compensation terms, and use a small batch of calibration data to regress the scaling coefficients. This ensures that the output distribution of the quantized model on critical layers closely matches the original model. This step determines whether the “retention rate” sits at the 95% or 98% level—note that this is an order-of-magnitude description, and exact figures must be based on official evaluations.

A conceptual pseudocode for ternary quantization is shown below, illustrating the process only and not representing any specific implementation:

# Conceptual illustration: Rotational basis projection + group-wise ternary quantization
W = weight_matrix              # Original weights
W_rot = orthogonal_transform(W)  # Orthogonal rotation to disperse outliers
for group in split_into_groups(W_rot, group_size=128):
    scale = group.abs().mean()        # FP16 group-wise scaling factor
    ternary = sign(group)             # Map to {-1, 0, +1}
    dequant = ternary * scale
    residual = group - dequant        # Optional residual: low-rank compensation

It must be emphasized that these benefits come with a cost: dequantization during inference requires additional computation and memory access, meaning actual throughput may not scale proportionally with the reduction in model size.

3. Comparison Dimensions: Ternary vs. FP8

To make “which hallucinates less” meaningful, we must break it down into observable dimensions.

Dimension	Ternary (~1.7~2 bit)	FP8 Quantization (8 bit)
Size	Extremely small, suitable for memory-constrained devices	About half the size of the original
Precision Loss	Algorithm-dependent; leading solutions minimize this	Usually very small, close to the original
Factual Accuracy / Anti-hallucination	Depends on effective outlier handling	Generally more stable, smoother degradation
Inference Speed	Dequantization overhead may offset size benefits	Fast when well-supported by hardware
Long Context	Largely orthogonal to the quantization scheme	Same as left
Ecosystem Maturity	Relatively new, toolchains still maturing	Relatively mature

The conclusive judgment is: Given the same base model and generation of training, FP8 generally has a higher precision ceiling and more predictable anti-hallucination performance. However, an excellent ternary solution can narrow this gap significantly, making it a reasonable choice when memory is tight. Moreover, “hallucinating less” is not a strictly monotonic function of bit-width—a coarsely algorithmic 4-bit model can easily perform worse than a finely tuned 2-bit model.

4. Engineering Selection: Calculate Memory First, Then Determine Precision

Model selection should start from constraints, not parameters.

1. Memory Budget Priority: First confirm the available VRAM/RAM limit, and then reverse-calculate the model size it can accommodate. If the budget only allows for ultra-low bit models, debating “which is more accurate” is moot; focus on the model’s performance within that bit-width.
2. Task Tolerance Tiering: Factual QA, code generation, and structured extraction are sensitive to hallucinations and should prioritize high-precision quantization. Creative writing and summarization have higher tolerance, making ultra-low bit more acceptable.
3. Context Length: Long contexts significantly increase KV cache usage. The space saved by quantizing weights might be consumed by the cache, requiring an overall system assessment.
4. Hardware Operator Support: Ensure that the target inference framework natively supports ternary/FP8 operators. Otherwise, dequantization overhead will prevent the “small size” from translating into “fast speed”.

5. How to Validate “Which Hallucinates Less”

Do not rely on single benchmark scores from marketing materials. It is recommended to build your own small evaluation set:

• Factual Consistency: Construct 100~200 questions with definitive answers and track the error and fabrication rates.
• Long-range Logic: Use multi-step reasoning and multi-turn follow-ups to observe contextual contradictions.
• Structured Output: Measure field completeness and format compliance for JSON/table extraction.
• Refusal Capability: Test whether the model dares to say “I don’t know” to unanswerable questions, a crucial anti-hallucination metric.
• Consistency Sampling: Sample the same question multiple times and calculate answer variance.

During evaluation, fix decoding parameters (temperature, top-p) and record quantization configurations to ensure reproducibility. If conditions permit, run a baseline comparison against the full-precision version of the same base model to quantify exactly “how much was lost”.

6. Conclusion

The debate of “Ternary vs. FP8: which hallucinates less” is fundamentally not about bit-width, but about whether the quantization algorithm respects the numerical structure of the network. Rotational basis projection, group-wise scaling, and residual compensation make 2-bit level compression viable for the first time. Meanwhile, FP8 remains the prudent choice for hallucination-sensitive scenarios due to its smoother degradation curve.

The rational approach is to use memory constraints to shortlist candidates, validate factual accuracy with a custom evaluation set, and make the final decision based on hardware operator support. When conducting containerized deployments on general cloud platforms like Baidu AI Cloud, it is similarly recommended to include quantization configurations in version control. Pair this with monitoring and alerting to track anomaly rates in production outputs, turning “hallucinating less” into a measurable, regressive engineering metric rather than a subjective feeling.