Deploying a 2.8T Open-Weight Model: How to Calculate VRAM and Licensing
Deploying a 2.8T parameter open-weight Mixture-of-Experts (MoE) model involves more than just downloading 1.56TB of weights. This guide walks through the critical steps for successful deployment: understanding MoE memory requirements, navigating quantization formats like MXFP4, accurately calculating VRAM (including the crucial TB to GiB unit conversion and its impact on hardware selection), reviewing essential licensing clauses, and comparing the costs of self-hosting versus renting cloud infrastructure.
Introduction
A 2.8T parameter open-weight MoE (Mixture-of-Experts) model typically has a weight file of approximately 1.56TB. Downloading the file is only the first step. Whether you can actually run the model depends on four factors: sufficient VRAM, efficient execution of the quantization format by the inference framework, licensing permissions, and a cost comparison between self-hosting and renting.
This article does not target any specific vendor’s product. Instead, it uses this class of models as a generic case study to clarify how to read the architecture, calculate VRAM, understand quantization, check licensing terms, and determine whether to build or rent. The numbers used here come from publicly available model card configurations. If your model version differs, recalculate using your specific numbers.
1. Understanding MoE: High Storage, Low Computation
MoE models distribute a vast number of parameters across many expert networks. When a token enters the router, the router selects only a few experts to participate in the calculation; the remaining experts remain idle for that step.
According to typical model card configurations, this class of models has about 2.8T total parameters and roughly 104B activated parameters per step. The single-step computation volume is close to that of a 100-billion-level dense model, but with a much larger knowledge capacity.
This has two direct implications. First, since computation is tied to activated parameters, single-step inference speed does not degrade linearly with total parameters. Second, because any expert might be called for a given token, all expert weights must be accessible at all times. Therefore, VRAM must be provisioned based on the total parameter volume, not the activated parameters. Many people miscalculate here, assuming 104B’s VRAM requirement is sufficient.
These models support context lengths in the million-token range. The primary cost of long context comes from the KV Cache, which grows linearly with sequence length. Some models use sparse attention to reduce overhead; whether this is used and how it is implemented depends on the model configuration file and inference framework support.
If the repository includes a vision encoder, you must also reserve VRAM and inference paths for the vision components during deployment. Whether vision weights are included should be verified against the repository’s file manifest, not just marketing materials.
2. Quantization Format: The Trade-offs of 4-bit Weights
Model cards indicate that these weights use the MXFP4 format. Weights are 4-bit floating-point, activations are 8-bit floating-point, and the model is trained using Quantization-Aware Training (QAT).
In MXFP4, every 32 elements share an 8-bit scale factor (E8M0), and each element itself is a 4-bit float (E2M1). On average, each parameter occupies about 4.25 bits.
The calculation is as follows: 2.8T × 4.25 bits ÷ 8 ≈ 1.49TB. The model card lists the size as 1.56TB, a difference of about 0.07TB. This discrepancy likely comes from unquantized tensors (like embedding and normalization layers) and file metadata, which should be verified item by item using the file manifest.
Because QAT simulates quantization errors during training, the precision loss at the same bit-width is usually smaller than with Post-Training Quantization (PTQ). This is a common reason vendors release only quantized versions. Specific precision performance depends on the task and evaluation set; start with the vendor’s published benchmarks and then retest with your own business data.
Loading the model is just the first step after quantization. The inference framework must have corresponding dequantization or low-precision computation kernels, or you will end up with a model that loads but suffers from low throughput. Before deployment, verify the following checklist:
- Inference framework version: Check release notes to confirm explicit support for the weight format and note the minimum required version. Common frameworks include vLLM, SGLang, and TensorRT-LLM; refer to their official documentation for specific ranges.
- GPU architecture: Ensure the target GPU has efficient low-precision kernels. Older architectures may fall back to slow paths, significantly reducing throughput.
- CUDA and driver versions: Verify minimum versions per the framework’s documentation.
- File integrity: Check the file manifest, version numbers, and checksums (e.g., sha256) to confirm they match the model card release.
- Feature combinations: Confirm if expert parallelism and long context can be enabled simultaneously.
- Performance baseline: Compare throughput and Time-to-First-Token (TTFT) using fixed inputs, rather than just checking if the service starts.
3. How to Calculate VRAM
The following three formulas cover most estimation needs:
- Weight VRAM ≈ Parameter count × bits per parameter ÷ 8. For this model, we directly use the 1.56TB stated on the model card.
- KV Cache ≈ 2 × number of layers × KV heads × head_dim × sequence length × batch size × bytes per element.
- Activations and communication buffers: Reserve 20% to 30% of the total.
It is crucial to unify the unit of measurement first. File sizes are calculated in decimal, where 1TB equals 1000GB. Thus, 1.56TB equals 1560GB, which is approximately 1453GiB. GPUs usually list VRAM in GiB (powers of 2), for example, an 80GiB card is about 85.9GB. This article uses GiB uniformly for calculations. If your GPU spec sheet uses decimal GB, the conclusions will differ, as demonstrated in the examples below.
Example 1: 80GiB GPUs. To hold just the weights: 1453 ÷ 80 ≈ 18.2, rounded up to 19 cards. This provides 1520GiB total, leaving only 67GiB. This remaining space must hold the KV Cache, activations, and communication buffers, which is clearly insufficient. Factoring in the 20% to 30% reservation, the total requirement is about 1744 to 1889GiB, requiring 22 to 24 cards. In practice, deployments scale by whole machines, so three 8-GPU servers (24 cards) is a common starting point.
Example 2: 192GiB GPUs. Eight cards provide 1536GiB total. The weights take 1453GiB, leaving 83GiB, which cannot even cover the 20% reservation (~291GiB). Based on the total requirement of 1744 to 1889GiB, you would need 10 cards (1920GiB total). Since standard servers usually have 8 GPUs per node, this typically means deploying two 8-GPU servers, totaling 16 cards.
If “192GB” is interpreted as a decimal unit, it equates to only about 178.8GiB. Eight cards would provide about 1430GiB, which cannot even hold the weights. Therefore, whether “8 cards can run it” depends entirely on the unit of measurement. Always read the spec sheet carefully before selecting hardware.
The KV Cache is the primary variable in long-context scenarios. Let’s demonstrate with a set of hypothetical parameters: 60 layers, 8 KV heads, head_dim of 128, sequence length of 131072 (~128K), batch size of 1, and BF16 (2 bytes per element). Plugging into the formula: 2 × 60 × 8 × 128 × 131072 × 1 × 2 ≈ 3.2×10^10 bytes, or about 30GiB. If the batch size increases to 8, the KV Cache jumps to about 240GiB. These parameters only demonstrate the calculation method and do not represent any specific model. Always use the actual values from your model’s configuration file for estimation.
4. What to Check in the License
The license dictates whether you are allowed to do this; download access does not equal usage rights. Review the following clauses carefully:
- Scope of use: Whether commercial use is allowed, if there are user scale or revenue thresholds, and whether exceeding them requires separate authorization.
- Redistribution: Whether you can package the weights into your own products, images, or installers, and whether you can offer them for download externally.
- External services: Whether charging third parties via an API falls within permitted use.
- Derivative models: Whether fine-tuned, distilled, or quantized models must retain the same license, and whether attribution or noting the original model in the name is required.
- Acceptable Use Policy (AUP): Prohibited uses listed, and restrictions on applications in specific domains.
- Output usage: Whether you can use the model’s output to train other models.
- Patents and trademarks: Whether the license grants patent rights, and whether using the publisher’s name or trademarks is allowed.
- Export controls and geography: Export control regulations in your country/region, and whether the model restricts usage in certain areas.
- Termination and updates: Termination conditions upon violation, and whether new versions automatically apply the same terms.
Always refer to the LICENSE file in the model repository and the official model card for the original text. For commercial rollouts, have legal counsel review the terms. This article does not constitute legal advice.
5. Costs Beyond Hardware
- Interconnect bandwidth: Expert parallelism requires distributing tokens to cards hosting different experts at each layer and gathering the results back. Communication volume scales with batch size and the number of activated experts. Insufficient inter-card bandwidth forces GPUs to spend significant time waiting for communication, dropping throughput. Intra-node usually relies on high-speed interconnects (like NVLink), while inter-node relies on networking. Before deployment, you can check GPU connections in NVIDIA environments using
nvidia-smi topo -m. - Storage and loading: 1.56TB of weights requires high-speed disks of equivalent capacity. Loading time depends on disk and network speed, and cold starts may take a considerable amount of time.
- Host memory: The loading process requires CPU memory to store temporary data; reserve this according to the framework’s requirements.
- Power and cooling: An 8-GPU server has a very high power draw under full load. Data center power supply and cooling must be accounted for in advance.
- Operations: Driver, framework, and kernel upgrades, faulty card replacement, and monitoring/alerting all require dedicated personnel.
6. Build vs. Rent
Calculate two figures first.
Monthly fixed cost of building = Hardware purchase price ÷ depreciation months + electricity + data center & network + ops personnel.
Monthly cost of renting = Hourly rate × actual running hours per month.
The break-even point is: Monthly running hours = Monthly fixed cost of building ÷ Hourly rate. If your actual usage hours are below this value, renting is more cost-effective. If you exceed it, building is cheaper.
Beyond cost, consider the following:
- Data compliance: Whether data is allowed to leave your local environment or intranet. For sensitive data, self-hosting is usually preferred.
- Load stability: Long-term, high-utilization workloads are better suited for self-hosting, while evaluation phases or businesses with high traffic fluctuations are better suited for renting.
- Team capability: Whether your team can maintain a multi-GPU inference cluster, including upgrades, troubleshooting, and monitoring. Without this capability, the hidden costs of self-hosting are high.
- License constraints: Whether your deployment method is permitted by the license, such as hosting on the cloud or offering external services.
Conclusion
First calculate the weights and VRAM headroom, then check the license, and finally compare the costs of building versus renting. By following these three steps, the hardware barriers and strategic direction will become clear.
