Open-Source LLM Selection and Deployment: MoE and 4-Bit Quantization in Practice
This article compares two open-source LLM deployment approaches: massive Mixture-of-Experts models, which raise the capability ceiling at the cost of substantial memory requirements, and efficient dense models using 4-bit quantization, which can run on a single server or GPU. It explains how to estimate compute, VRAM, attention, and KV cache costs, and highlights the key considerations for evaluating benchmarks and planning deployment.
Open-source large language models have been released at a rapid pace recently, with two technical approaches attracting particular attention. The first is the massive Mixture-of-Experts (MoE) model, which uses a much larger total parameter count to achieve a higher capability ceiling. The second is the high-efficiency dense model, combined with 4-bit quantization to reduce deployment requirements to a single server or even a single GPU.
Each approach involves trade-offs. Model selection should start with the available hardware budget, latency requirements, and business workload. This article does not evaluate specific products. Instead, it focuses on the underlying technologies, estimation methods, and key deployment considerations.
1. Massive MoE Models: The Trade-Off Between Capability and VRAM
A vendor has disclosed an MoE model with approximately 2.8 trillion total parameters, around 104 billion activated parameters per token, and native support for context windows of up to 1 million tokens. These figures should be verified against the vendor’s official technical report.
1.1 Sparse Activation: Calculate Compute and Memory Separately
MoE architectures replace the feed-forward network layers with multiple expert networks. A gating network selects only a small number of experts for each token, so the compute required for an individual token depends primarily on the activated parameters.
However, the weights of all experts must remain resident in GPU memory so that the router can select any of them at runtime. As a result, activated parameters determine the computational workload, while total parameters determine the minimum memory requirement.
For a model with 2.8 trillion parameters, the weights alone require approximately:
- FP16: 5.6 TB of VRAM
- 4-bit: 1.4 TB of VRAM
These estimates exclude the KV cache, activations, and framework overhead. The deployment cost of an MoE model is therefore driven mainly by total memory capacity, not by the number of activated parameters.
1.2 A 1M-Token Context: Two Types of Attention and KV Cache Costs
The computational cost of standard self-attention grows quadratically with sequence length. If an implementation materializes the full (n \times n) attention matrix in memory, memory usage also grows quadratically.
Implementations such as FlashAttention use tiled computation to avoid storing the entire attention matrix, significantly reducing peak memory usage. The computational complexity, however, remains quadratic. Ring Attention distributes long sequences across multiple GPUs for processing.
Another major source of memory usage is the KV cache, which grows linearly with sequence length. Its size can be estimated using the following formula:
KV cache bytes ≈ 2 × number of layers × number of KV heads × head dimension × sequence length × batch size × bytes per element
Even with a batch size of one, the KV cache for a 1-million-token context can consume a substantial amount of VRAM. The exact figure depends on the model architecture and must be calculated using the actual model configuration.
1.3 How to Interpret Benchmark Results
The vendor reports that the model performs strongly in logical reasoning, code generation, and multimodal understanding, and suggests that open-source models are approaching the capability of closed-source models.
When reading claims like these, check three aspects:
- The name and version of each evaluation set.
- Whether the models used the same prompts and decoding parameters.
- The versions of the comparison models.
If a model approaches or is locally comparable to other models on several public benchmarks, that conclusion should be recorded as stated. It should not be generalized into a claim of overall superiority.
2. Dense Models and 4-Bit Quantization: Single-GPU Deployment
A dense model activates all of its parameters for every generated token. Its inference path is simpler: it has no expert routing and requires no cross-device expert communication. As a result, single-machine deployment is generally easier than deploying an MoE model.
A hardware vendor has disclosed a 65-billion-parameter dense model pretrained on 12 trillion tokens. The vendor claims that the model can run on a single GPU with 128 GB of VRAM when using 4-bit quantization. It also reports a score above 80 on a benchmark for end-user automation tasks.
However, the benchmark result does not disclose the test-set version, prompts, or hardware conditions, so it cannot be directly compared with results from other models. The result should be reproduced on the target workload before being used for deployment decisions.
2.1 VRAM Estimation
4-bit quantization compresses model weights from 16 bits to 4 bits. For a 65-billion-parameter model, the theoretical weight size is approximately:
65 × 10⁹ × 0.5 bytes ≈ 32.5 GB
This calculation uses decimal gigabytes. In practice, quantization metadata, layers that remain unquantized—such as embedding layers—and framework overhead will increase memory consumption. The measured usage will therefore be higher than the theoretical weight size.
The vendor reports approximately 40 GB under conditions that were not fully disclosed. When estimating VRAM requirements, memory should be divided into three categories:
- Model weights
- KV cache
- Runtime and framework overhead
