Deploying a 2.8T Open-Weight Model: VRAM, Licensing, and Deployment Options
With the release of open weights for a 2.8-trillion parameter Mixture of Experts (MoE) model, teams face critical questions regarding hardware requirements, licensing, and deployment strategies. This article provides a practical guide to estimating VRAM requirements, auditing license terms, and choosing the right deployment method—from managed APIs to self-hosted clusters. It offers a structured approach for engineering teams to navigate the complexities of deploying large-scale MoE models without incurring unnecessary costs or compliance risks.
Executive Summary
Following the public release of full weights for a 2.8T-parameter Mixture of Experts (MoE) model, most teams immediately ask three core questions. Here are the short answers, with details provided below.
- Local VRAM Requirements: Weights alone require approximately 1.5TB of VRAM (estimated using MXFP4). Once you factor in the KV cache, activations, and framework overhead, a single machine cannot run it locally. Full deployment requires multiple GPUs or multiple nodes.
- Commercial Licensing: You cannot assume commercial viability simply because a model is labeled “open weight.” You must read the original license text and any additional terms in the model card line by line.
- Cost-Effective Deployment: Most businesses should first validate their use case using managed services or distilled smaller models. Only consider self-hosting once the return on investment is confirmed.
This article does not provide specific pricing, performance benchmarks, or legal conclusions. Architecture and performance data are sourced from the vendor’s model card and release materials; we have not independently reproduced these results. Always refer to the official documentation for accurate references.
1. Background
The model was initially launched as an API and web service, with the vendor promising to release the full weights about ten days later. The weights subsequently appeared on public model hosting repositories, available for download without application or review, with mirror sites available globally.
The repository contains over a hundred files, including nearly a hundred safetensors shards, totaling approximately 1.56TB. This scale highlights a significant engineering gap between downloading a model and running it. Downloading is only the first step; teams must subsequently address hardware, inference frameworks, licensing, and operational maintenance.
2. Architecture Breakdown
According to the vendor’s model card, the model utilizes a sparsely activated Mixture of Experts (MoE) architecture.
- Parameter Scale: Total parameters are around 2.8T, with approximately 104B parameters activated per token. Sparse activation allows the model to maintain a large capacity while keeping the computational load for each inference relatively small.
- Expert Configuration: There are roughly 900 routing experts, with the model card indicating that each token is routed to a dozen or so experts. A small number of shared experts handle general knowledge, reducing redundant learning across individual experts. Exact top-k values should be verified against the model card configuration.
- Network Structure: Approximately 90+ layers, a context window in the millions of tokens, and an integrated vision encoder for simultaneous text and image processing. Exact context length should be confirmed via the model card.
The model card also notes that some layers replace full attention with sparse attention, allowing each query to focus only on a critical subset of keys and values. While standard attention computation and KV cache grow rapidly with context length, sparse attention reduces costs for long documents, extended conversations, and codebase analysis. However, the KV cache will still scale with context length and concurrency, requiring separate calculations during capacity planning.
Regarding precision, the model card states that weights are stored in MXFP4, activations use MXFP8, and this was achieved through Quantization-Aware Training (QAT). QAT allows the model to adapt to low-precision computation during training, typically resulting in less accuracy loss than post-training quantization. The official release only includes this quantized version; there are no BF16 or FP8 versions available. This means no higher-precision baseline is provided, so teams must evaluate the quantized version’s performance on their specific tasks.
It is important to note that the MoE architecture saves computation per token, not weight storage. All expert weights must be accessible. Unless using an offloading strategy, they must reside entirely in VRAM or fast-access memory.
3. VRAM Estimation Methodology
3.1 Formula
VRAM requirements can be broken down into four components:
Total VRAM Requirement ≈ Weights VRAM + KV Cache + Activations & Temporary Buffers + Framework Overhead
The calculation methods are as follows:
- Weights VRAM = Total Parameters × Average Bytes per Parameter
- KV Cache = 2 × Number of Layers × Number of KV Heads × Head Dimension × Context Length × Concurrent Requests × Bytes per Element (The 2 represents storing K and V separately)
- Activations & Temporary Buffers: Dependent on batch size, sequence length, and inference framework; empirical testing is recommended.
- Framework Overhead: Inference engine, communication buffers, and fragmentation. Usually reserved as a percentage of the total, which must be measured in the target environment.
The number of layers, KV heads, and head dimensions must be read from the model card configuration. This article does not provide these specific values to avoid speculation.
MXFP4 is a microscaling format where every 32 elements share an 8-bit scaling factor. Calculating 4 bits per element plus the shared scaling factor yields approximately 4.25 bits per element, or roughly 0.53 bytes. This is an estimate, not an exact figure.
3.2 Example
Calculating weights VRAM using 2.8T total parameters:
2.8 × 10¹² × 0.53125 bytes ≈ 1.49 × 10¹² bytes ≈ 1.49 TB
The actual repository size is about 1.56TB, showing a discrepancy with this estimate. The difference likely comes from the vision encoder, unquantized layers, and other files. Always rely on the actual files in the repository.
Assuming a single GPU has 80GB of VRAM, storing just the weights requires at least:
1490 GB ÷ 80 GB ≈ 18.6, meaning at least 19 GPUs.
This is the absolute lower bound. Actual deployment requires adding KV cache, activations, and framework overhead, increasing the GPU count. Furthermore, while activated parameters are only around 104B, this affects computation and per-token memory access, not the VRAM occupied by the stored weights.
Estimates only serve as a starting point for planning. Before going live, peak VRAM should be empirically tested on target hardware, using the target inference framework, with real request lengths, while leaving sufficient buffer.
4. License Audit Checklist
The following checklist helps identify clauses requiring careful review. This article does not constitute legal advice; final conclusions depend on the original license text. Consult legal counsel for commercial and compliance decisions.
- Weight License Itself: Identify the license file in the repository; record the license name and version.
- Model Card Additional Terms: Check for extra usage policies, prohibited use cases, or user scale limitations. These may differ from standard licenses.
- Commercial Terms: Confirm if use in paid products or services is allowed, and whether there are limits on user numbers or revenue.
- Redistribution Terms: Confirm whether packaging, reselling, or hosting the weights on third-party platforms is permitted.
- Derivative Model Terms: Verify if newly generated models from fine-tuning, merging, or distillation can be published, and whether they must use the same license.
- Training on Outputs: Confirm whether model outputs can be used to train other models. This is crucial for distillation strategies.
- Attribution & Notices: Check if retaining copyright notices, crediting sources, or disclosing usage in products is required.
- Patent & Litigation Terms: Look for patent grants, termination conditions, or litigation-related agreements.
- Export Controls: Large model weights and related technologies may be subject to export controls. Assess cross-border transfers or providing services to specific regions.
- Data Compliance: If using cloud services, evaluate requirements for cross-border data transfer and storage of user data.
5. Deployment Options Comparison
The table below compares four deployment options across several dimensions. It provides qualitative assessments without specific pricing.
| Option | VRAM & Hardware | Latency & Throughput | Cost Structure | Data Compliance | Ops Overhead |
|---|---|---|---|---|---|
| Managed Cloud API | No hardware required | Depends on provider resources and queuing | Pay-per-use; usually cheaper at low volumes | Must assess if data leaves your environment or is processed by a third party | Low |
| Self-hosted Cluster | Requires multi-GPU or multi-node; estimate capacity per Section 3 | Controllable, depends on parallelization and networking | Fixed investment plus power, facilities, and maintenance; utilization drives unit cost | Data can remain on-premises | High |
| Distilled Small Model | VRAM requirements much lower than the original | Usually faster; requires testing | One-time training cost, followed by lower inference costs | Must verify if license allows training on outputs | Medium |
| Offloading | Weights on CPU memory or SSD; GPU keeps only partial data | Significantly slower than full VRAM residency; requires testing | Lower hardware investment, but high time cost | Data remains local | Medium to High |
Applicable scenarios for each option:
- Managed Cloud API: Suitable for validating requirements, handling fluctuating traffic, or for teams lacking GPU operations experience.
- Self-hosted Cluster: Suitable when data must remain on-premises, and traffic is stable and sufficiently high.
- Distilled Small Model: Suitable for latency- and cost-sensitive scenarios where some quality degradation is acceptable. Always verify license terms before distilling.
- Offloading: Suitable for offline batch processing, latency-insensitive tasks, or small-scale experiments.
6. Selection Checklist
Before deciding on a deployment strategy, confirm the following questions:
- Does the target task genuinely require a model of this scale? Compare against smaller models and baselines first.
- Have you finished reading the original license and model card additional terms, and recorded conclusions regarding commercial use, redistribution, and derivative models?
- Have you consulted legal counsel to confirm export control and data compliance requirements?
- Have you estimated weights and KV cache using the formula in Section 3, and reserved budget for framework overhead?
- Have you empirically tested peak VRAM and latency on the target hardware?
- Have you evaluated the performance of the quantized version on your specific tasks?
- Have you calculated the total cost of all four options, including hardware, power, labor, and time?
- If choosing to self-host, do you have sufficient traffic to amortize the fixed investment?
- If choosing distillation, have you confirmed the licensing terms for training on outputs?
Only commit to self-hosting once you have clear answers to most of these questions. For most teams, validating business value first with managed services or distilled models is generally the lower-risk path.
