Extreme Quantization in Practice: Precision and Deployment Trade-offs After Shrinking an LLM by 60%

1. Background: When Model Scale Collides with Deployment Cost

In recent years, the parameter scale of open-source large language models has continued to rise. Mixture-of-experts (MoE) architectures with hundreds of billions or even trillions of parameters are no longer unusual. These models perform well across a range of benchmarks, with some capabilities approaching those of closed-source commercial models. However, the improvement in capability comes with rapidly growing deployment costs.

Take a trillion-parameter MoE model that natively uses INT4 quantization as an example. Its model file is roughly 594GB. Although this is already much smaller than the FP16 equivalent, smoothly starting inference still requires eight high-end accelerators with 141GB of VRAM each. The hardware barrier is extremely high for most developers and small-to-medium-sized teams.

A natural question follows: since the model is already natively INT4, can quantization be pushed even further? From an information-theory perspective, INT4 already compresses each weight to 4 bits, so the remaining compression space would seem limited. However, engineering practice shows that more aggressive quantization strategies can still achieve substantial file size reduction.

2. The Technical Path to Re-Quantization: From INT4 to 1-Bit-Level Precision

In the field of model quantization, extremely low-bit-width quantization has long been an active research area. For very large MoE models, existing approaches have achieved an effective quantization level of around 1.8-bit, compressing the model file from roughly 594GB to about 247GB, a reduction of nearly 60%. This result overturns the intuition that “there is little room left after INT4.”

2.1 Quantization Versions and File Size Comparison

Quantization Version	Effective Bit Width	Approximate Model Size	Accuracy
Native INT4	4-bit	594GB	Baseline
Recommended Balanced Version	About 2-bit	360GB	Close to baseline
Extreme Compression Version	About 1.8-bit	247GB	About 85%

As shown in the table, the extreme compression version has a significant size advantage but lower accuracy. The medium-bit-width version achieves a better balance between size and accuracy and is the recommended choice in most scenarios.

2.2 Core Ideas Behind the Quantization Method

This kind of ultra-low-bit-width quantization typically combines several techniques:

• Grouped quantization: Weights are divided into groups by channel or block, and each group has an independent scaling factor, reducing quantization error.
• Mixed-precision allocation: Sensitive layers keep a higher bit width, while redundant layers use an extremely low bit width, achieving overall compression.
• Calibration set optimization: Representative calibration data is used to adjust quantization parameters and reduce accuracy loss caused by shifts in activation distribution.
• Differentiated handling of MoE experts: Different experts have different levels of importance, so experts activated more frequently retain more information.

Together, these methods allow the model to maintain usable generation quality even at extremely low bit widths.

3. Deployment Resource Estimation: Disk, Memory, and VRAM

To understand the deployment requirements of a quantized model, start with a basic constraint:

Disk space + system memory (RAM) + GPU memory (VRAM) ≥ quantized model size

Inference frameworks generally support memory-mapped loading, meaning model weights can be loaded from disk into memory or VRAM on demand. This means that even if the combined memory and VRAM are insufficient, the model can still run, but speed will drop significantly. In extreme cases, throughput may fall below 1 token/s.

3.1 Expected Experience Under Different Configurations

• Minimum requirement: As long as the sum of the three resources exceeds the model size, the model can start, but speed will be extremely slow and only suitable for feasibility validation.
• Recommended configuration: When the combined memory and VRAM roughly equal the model size, a relatively smooth experience can be achieved, such as 5+ tokens/s.
• GPU-accelerated setup: A single 24GB GPU combined with 256GB of system memory and MoE offloading can deliver roughly 1–2 tokens/s.
• Multi-GPU high-VRAM setup: More VRAM leads to faster generation and significantly reduces reliance on system memory.

3.2 Cost and Speed Trade-offs

Shrinking the model from 594GB to 360GB and then to 247GB substantially lowers hardware costs. But it is important to recognize that extremely low-bit quantization provides possibility rather than production-grade usability. For stable, high-speed online services, adding GPU memory is still preferable to relying solely on disk offloading.

4. Practical Steps: Building the Runtime and Obtaining the Model

The following workflow uses a general-purpose inference framework as an example to show how to prepare a runtime that supports the latest MoE architectures.

4.1 Building the Latest Inference Framework

First, obtain and compile the latest source code of an inference framework that supports the target model architecture.

# Update the system and install base dependencies
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
# CPU-only build configuration when no GPU is available
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=OFF \
  -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first
# Copy the generated executables
cp llama.cpp/build/bin/llama-* llama.cpp

If a GPU environment is available, change -DGGML_CUDA=OFF to ON and make sure the CUDA toolchain is properly installed.

4.2 Downloading Quantized Models

A client script that supports resumable downloads and parallel transfer is recommended so you can flexibly select a specific quantized version.

# Install dependencies
pip install huggingface_hub hf_transfer
# Set environment variable to enable high-speed transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
# Download a specified quantized version (medium-bit-width version in this example)
huggingface-cli download <repo_id> \
  --include "*Q2_K_XL*" \
  --local-dir ./model

After downloading, load the model through the inference framework and start the service. It is advisable to first verify that the model can generate normally with short prompts before gradually increasing concurrency and context length.

5. Choosing the Right Quantized Version

With several quantized versions available, how should you choose? Consider the following dimensions:

1. Hardware constraints: The combined VRAM and system memory determine the largest model size you can run.
2. Accuracy requirements: If the task is sensitive to output quality, such as code generation or logical reasoning, prefer a higher-bit-width version.
3. Speed requirements: For online serving, ensure sufficient VRAM to avoid frequent disk offloading.
4. Budget: Extremely low-bit versions significantly lower the hardware barrier, but you must accept trade-offs in speed and accuracy.

Overall, the medium-bit-width balanced version at roughly 360GB is a safer choice in most scenarios. It offers a better compromise among size, accuracy, and speed.

6. Summary and Outlook

Quantization experiments with extremely large MoE models show that even on top of native low precision, more aggressive quantization strategies can reduce model size by about 60%. This creates possibilities for developers with limited resources to explore frontier models.

However, quantization is not a free lunch. Extremely low-bit quantization brings accuracy loss and slower inference. In production environments, VRAM remains the core resource that determines the inference experience. In the future, as quantization algorithms and inference frameworks continue to evolve, we can expect to maintain higher accuracy at even lower bit widths, further lowering the deployment threshold for large models.

For developers, understanding quantization principles, mastering resource estimation, and selecting the right version for each scenario are key skills for using these models efficiently.