Deploying a 2.8T Open-Weight MoE Model: License, Memory Budget, and Production Checklist
This article examines the practical implications of deploying a 2.8-trillion-parameter open-weight mixture-of-experts (MoE) model with a 1.56TB MXFP4 weight package. It covers the model architecture, quantization approach, custom license restrictions, GPU memory requirements, inference configuration choices, compliance checks, and deployment paths for teams considering local or cloud deployment.
Deploying a 2.8T Open-Weight MoE Model: License, Memory Budget, and Production Checklist
1. Background: From API Availability to Downloadable Weights
On November 4, a model with 2.8 trillion total parameters released its complete weights in a public repository. The weight package is about 1.56TB, split into roughly 100 shards, and includes index files, configuration files, and a model card. The weights can be downloaded without applying for access.
The model is positioned as one of the largest open-weight models currently available. Its release marks a new stage for open-weight models: the trillion-parameter era.
For developers, “open weights” and “hosted API” are fundamentally different.
When using a hosted API, the service provider bears the model scale, infrastructure, and operating cost. Developers pay for usage, while the model architecture, hardware stack, and deployment cost remain a black box.
Open weights hand three things to the user at the same time:
- The weight files themselves.
- A custom license.
- A deployment path with strict hardware and operations requirements.
This article does not focus on release milestones. Instead, it answers three practical questions:
- What is inside the weights?
- What does the license allow you to do?
- What does it really cost to run the model locally?
2. Model Architecture: How 2.8T Is Structured
2.1 Total Parameters vs. Active Parameters
According to the model card, this is a natively multimodal, agent-oriented model that was continued from a previous generation. The key metrics are:
| Metric | Value | Engineering meaning |
|---|---|---|
| Total parameters | 2.8T | Determines weight file size and GPU memory usage |
| Active parameters per token | 104B | Determines compute per token and which weights are read during inference |
| Total experts | 896 | Model capacity is distributed across experts |
| Active experts | 16 routed + 2 shared | Affects routing and expert-parallel implementation complexity |
| Layers | 93 | Affects KV cache size and pipeline partition granularity |
| Context length | 1M tokens | Affects attention implementation and KV cache pressure |
| Vision encoder | Proprietary ViT-based design | Enables image-and-text input and supports native multimodality |
The ratio of total parameters to active parameters is roughly 27:1. In other words, only about 3.7% of the parameters participate in computation for each generated token.
This is the core characteristic of sparse MoE: model capacity and per-token compute are decoupled. Total parameters determine how much knowledge the model can store. Active parameters determine how much compute is required to process one token.
From an engineering perspective, this creates two different types of load:
- Memory capacity bottleneck: Regardless of what is being generated, all expert weights must remain resident in memory.
- Bandwidth and compute bottleneck: Only the active portion of the weights needs to be read for each token. If memory bandwidth is sufficient and routing is balanced, single-GPU token throughput is theoretically determined mainly by the ratio of memory bandwidth to active parameter size.
As a result, “fitting the model” is often harder than “running the model.” Whether inference can run smoothly depends heavily on low-bit quantization.
2.2 Sparse Attention: Reducing the Cost of Long Context
Compared with the previous generation, the architecture has one major change: 24 layers that previously used dense full attention now use sparse attention, referred to as DSA.
The motivation is straightforward. Standard attention compute and KV cache usage grow quadratically with context length. At the million-token scale, attention overhead can overwhelm the MoE computation itself.
Sparse attention works by allowing each query to attend only to a selected subset of key/value pairs rather than the entire history. This changes attention cost from quadratic growth with length to approximately linear growth, making ultra-long contexts more feasible.
For deployers, this has two practical implications:
- The cost of long-context inference drops significantly, making scenarios such as long-document analysis and whole-repository code understanding more viable.
- The attention implementation required by the deployment stack also changes. The inference engine must support the corresponding sparse attention operators and cannot simply reuse standard FlashAttention-like kernels.
2.3 What “Native Multimodal” Means
“Native multimodal” means that the vision encoder was jointly trained with the language model starting from the pre-training stage, rather than being attached afterward as a separate visual module.
This has several engineering implications:
- Image and text tokens share the same context and positional system, allowing the model to continue a conversation after seeing an image.
- Visual input consumes context budget. High-resolution images can expand into a substantial number of visual tokens.
- Multimodal inference can affect memory usage and latency differently from text-only inference, so it should be benchmarked separately.
3. Weight Format: MXFP4 and the Trade-off of Shipping Only One Version
3.1 Low-Bit Quantization Is Not Just “Cutting Precision”
The official release provides only one version: MXFP4 weights and MXFP8 activations. No higher-precision version is offered.
To understand this choice, it helps to distinguish between two types of quantization:
- Post-training quantization (PTQ): After training is complete, the weights are compressed to lower precision using a small calibration dataset.
- Quantization-aware training (QAT): Quantization error is simulated during training, allowing the model to learn to operate under lower precision. QAT usually preserves accuracy better than PTQ under similar conditions, but it requires additional training cost.
The released weights are the product of QAT.
MXFP4 is a micro-scaling floating-point format. Each 4-bit weight has a limited number of exponent and mantissa bits and shares a scaling factor with the small block in which it resides. Compared with fixed-point INT4, it better accommodates differences in weight distributions across experts while maintaining the same storage overhead.
Low-bit quantization is especially suitable for MoE models for three reasons:
- When parameter count is very large, memory savings from quantization scale up significantly.
- The parameter count of each individual expert is relatively manageable, making quantization error easier to control.
- During inference, experts are activated in batches. Low-bit weights can benefit from wider memory bandwidth.
3.2 How the 1.56TB Weight Size Is Calculated
The weight size can be checked with a simple calculation. If all 2.8 trillion parameters were stored at 4 bits, the theoretical size would be approximately:
2.8 × 10^12 parameters × 4 bits ÷ 8 = 1.4 × 10^12 bytes ≈ 1.4 TB
The actual package is about 1.56TB, roughly 0.16TB larger than the pure 4-bit theoretical size. This difference mainly comes from three sources:
- Some weights stored at MXFP8, such as certain layers saved at 8 bits.
- Metadata such as scaling factors for each quantized block.
- Components that are usually kept at higher precision, such as embedding layers, output heads, and normalization parameters.
Therefore, “1.56TB” should not be interpreted as simply “2.8 trillion parameters × 4 bits.” It is the result of mixed-precision quantization. Memory planning should use the officially published weight size.
3.3 What “Only One Version” Means
Releasing only one quantized version is a pragmatic but constrained choice.
- Pragmatic: A higher-precision version would multiply both storage and memory requirements. At 8 bits, the same parameter count would require about 2.8TB just for weights. Higher precision would be even larger. Shipping a single version helps control hosting cost and makes hardware requirements more predictable.
- Constraint: Deployers lose the ability to choose. To run a BF16 version, they must either dequantize the released weights or retrain the model. Dequantization cannot restore accuracy lost during training. Retraining is expensive.
This creates two risks:
- Ecosystem adaptation risk: Whether inference engines and hardware drivers natively support this format directly affects time to deployment.
- Single-version risk: If the license is later tightened or the files are removed, users have no alternative source.
A practical recommendation is to treat “private mirror plus checksum archive” as a standard practice. After downloading, verify the integrity of each shard, and include both the weights and the license text in the enterprise asset inventory. The weights and license should be version-bound and archived together.
4. License: Open Weights Are Not Open Source
This is one of the most important aspects to review before deployment.
4.1 Two Documents Must Be Read Together
The repository effectively contains two “contracts”:
- Weight files: Determine what the model can do and what is required to run it.
- License terms: Determine what you are allowed to do.
The license is a custom text labeled as “other” in the public repository. It is not a standard permissive license such as MIT or Apache, nor is it a standard open-source license.
Its main terms resemble an MIT-style grant. It allows free use, copying, modification, distribution, fine-tuning, and commercial use, but it adds two restrictions specifically aimed at large-scale external businesses:
- If the model is offered as a model-as-a-service and the entity’s cumulative revenue over a 12-month period exceeds USD 20 million, a separate agreement must be signed before commercial use.
- If a commercial product’s monthly active users exceed a threshold defined in the license, a separate license is also required.
A useful analogy is that this is a license with “permissive defaults, but two gates.” Small teams and individual developers are largely unaffected. Platform-level commercial users must negotiate an agreement first.
4.2 Three Differences from the Open Source Definition
The Open Source Definition includes core principles such as free redistribution, permission to create derivative works, no discrimination against specific fields of use, and license terms that are not restricted by distribution method.
Compared with that definition, this license has at least two points that need attention:
- Field-of-use limitation: Setting revenue or user-scale thresholds effectively differentiates authorization based on commercial scale. This resembles a field-of-use restriction.
- Scale limitation: Using revenue and monthly active users as trigger conditions means that the same weights impose different legal obligations on users of different sizes.
Therefore, “open weights” or “downloadable weights” is more accurate than “open source.” Marketing materials inside enterprises should avoid calling this model an “open-source model,” to reduce the risk of misrepresenting the actual authorization.
4.3 Enterprise Compliance Checklist
The license mentions revenue, monthly active users, and model-as-a-service, but it does not provide a machine-readable legal determination. Before deployment, teams should clarify the following items:
| Check | Questions to clarify |
|---|---|
| Revenue definition | Is it revenue from a single product or the entire legal entity? Are affiliated entities included? Is it calculated by calendar year or rolling 12 months? |
| Scope of “offering a service externally” | Does calling by external customers count? What about internal employee use? What about embedding the model into a company’s SaaS product? |
| Monthly active user definition | How are users deduplicated? What is the measurement period? Are internal users included? |
| Obligations for derivative models | Do fine-tuned, distilled, or merged models inherit the original license restrictions? What notices must be included during distribution? |
| Redistribution restrictions | Can the weights be embedded into commercial images or appliances? Do end customers need to accept the license individually? |
| Termination and notice | What are the consequences of breach? How does the licensor provide notice or change terms? |
To remain neutral, this article does not cite the original license text in detail. The above questions should be confirmed against the license text and legal advice.
On the engineering side, teams can instrument service gateways to continuously record revenue and monthly active user metrics, providing data support for compliance review:
def need_separate_license(revenue_last_12m_usd: float,monthly_active_users: int,*,revenue_threshold: float, # Confirm the exact value in the license textmau_threshold: int, # Confirm the exact value in the license textoffers_model_as_a_service: bool,) -> bool:"""This function is for continuous monitoring, not a legal determination.The final conclusion should be based on the license text and legal advice."""revenue_hit = revenue_last_12m_usd > revenue_thresholdmau_hit = monthly_active_users > mau_thresholdreturn offers_model_as_a_service and (revenue_hit or mau_hit)
4.4 Other Terms Beyond the License
In addition to the license, the model card usually includes usage-policy terms. Examples include prohibiting illegal use, prohibiting harmful content generation, and requiring human review in high-risk scenarios.
These terms operate alongside the license and are also part of compliance review. However, they are closer to “usage norms” than to “copyright authorization.”
5. Local Deployment: What an 8-GPU Starting Point Really Means
5.1 First, Calculate the Memory Budget
The weights are about 1.56TB. Across an 8-GPU configuration, the average weight size per GPU is:
1.56 TB ÷ 8 ≈ 0.195 TB ≈ 195 GB per GPU
This is a hard threshold. An 8-GPU server with any single GPU having less than about 195GB of memory cannot hold the complete weights.
Consumer-grade 24GB GPUs are impractical if relying on GPU memory alone, no matter how many are used.
At the same time, 195GB is only the weight requirement. A complete memory budget must also include:
- KV cache: Grows with the number of concurrent requests and context length. At million-token context lengths, this can become the second major memory consumer and may determine concurrency capacity.
- Runtime overhead: Includes activation buffers, communication buffers, and scheduling fragmentation. A reserve of 10%–20% is usually needed.
- Optional quantization or cache copies: Some implementations quantize KV cache separately or use layered caching, which consumes additional space.
The real per-GPU requirement is the sum of weights, KV cache, and headroom. Many teams discover after “fitting the parameters” that throughput is still limited. The reason is often here.
5.2 Key Inference Configuration Choices
# Example configuration. Parameter names depend on the inference framework.config = {"model_path": "/data/models/moe-2.8t-mxfp4","quant_format": "mxfp4", # 4-bit micro-scaling floating-point weights"activation_quant": "mxfp8", # 8-bit activations"tensor_parallel_size": 8, # Split in-layer matrix operations across 8 GPUs"expert_parallel_size": 8, # Distribute experts across 8 GPUs"max_model_len": 32768, # Set according to real workload needs"kv_cache_dtype": "fp8", # KV cache quantization for long-context scenarios"enable_chunked_prefill": True, # Chunked prefill for long inputs}
Several parameters are worth explaining:
- Tensor parallelism and expert parallelism: Tensor parallelism splits matrix multiplications within a layer across multiple GPUs. It is communication-intensive and should preferably run within a high-speed interconnect domain. Expert parallelism distributes experts across GPUs. It fits the MoE structure better but introduces all-to-all-style token routing communication. The two are often combined.
- max_model_len: Every doubling of context length roughly doubles KV cache demand. Configuring context length according to actual business needs, instead of always setting it to the maximum of 1M tokens, is one of the most effective ways to reduce cost.
- KV quantization and chunked prefill: KV quantization reduces cache usage. Chunked prefill splits long-input prefill into smaller chunks and interleaves them with decoding, preventing long requests from blocking short requests.
5.3 Are There Alternatives to 8 GPUs?
In theory, there are three ways to bypass the 8-GPU threshold, each with a cost:
- Weight offloading: Place some expert weights in CPU memory or SSD and load them on demand. This can work, but token generation speed drops significantly. It is more suitable for offline batch processing.
- Further quantization: For example, compressing weights to 2 bits. Accuracy loss is usually difficult to accept, and it requires the ability to modify weights, which conflicts with the reality of a single official version.
- Waiting for lower-precision versions or specialized inference optimizations: The benefit is uncertain, and the timeline is uncontrollable.
The conclusion is clear: 8 GPUs are the starting point for smooth online serving, not the minimum requirement for getting the model to run at all.
5.4 Deployment Path Selection
| Deployment path | Suitable scenarios | Key considerations |
|---|---|---|
| Public cloud GPU instances | Rapid validation, elastic concurrency | Total memory per instance, interconnect bandwidth, weight download and image prewarming time |
| Self-built or rented GPU servers | Long-term stable workloads, high data compliance requirements | Data center power and cooling, multi-node interconnect, operations staffing |
| Hosted API | Teams that do not want to manage infrastructure and whose business scale does not trigger license thresholds | Unit price, SLA, data export and privacy requirements |
| Distilled or smaller models | Clear task boundaries, low-latency requirements | Accuracy loss and migration cost |
For cloud deployment, several general capabilities usually need to be combined:
- Use GPU cloud servers to host inference processes, focusing on total memory per instance, inter-GPU interconnect, and network bandwidth.
- Use object storage to store weight packages at the 1.5TB level, combined with image or cache prewarming to shorten scaling time.
- Use a container platform for orchestration and elastic scaling to handle traffic peaks.
- Use logging services and monitoring and alerting to observe throughput, latency, memory usage, and request queue status.
On Baidu AI Cloud, these capabilities have corresponding general product categories that can be combined. For specific specifications and availability, refer to the official documentation. This article does not cover commercial or billing details.
6. Benchmarks: Vendor-Reported Results and Third-Party Evaluation
Benchmark results should be interpreted carefully.
For coding and agent-related benchmarks, the model performs close to leading closed-source models on terminal-oriented tasks and long-horizon software engineering tasks, and leads on some metrics. However, it still shows a clear gap on scientific benchmarks and some high-difficulty reasoning benchmarks.
The overall pattern is: strong in coding and agent tasks, weaker in science and some reasoning tasks. Across 37 benchmarks, the model outperforms the comparison model on 16, resulting in a mixed outcome.
For enterprise adoption, a more meaningful approach is to build a private evaluation set:
- Task fit: Use real business samples, such as code repositories, tickets, and documents, for end-to-end evaluation rather than relying only on public leaderboards.
- Long-context effectiveness: Construct retrieval and reasoning tasks that span multiple documents to verify whether million-token context is truly “remembered and usable.”
- Multimodal quality: Scenarios such as field extraction from mixed image-text inputs and chart understanding should be tested separately.
- Performance baseline: Record first-token latency, per-token latency, concurrent throughput, and memory usage as a basis for capacity planning.
- License compliance upfront: Include license terms in the release review checklist rather than discovering issues only after scaling.
7. Action Checklist
If you are considering bringing this type of ultra-large open-weight model into production, proceed in the following order:
- Define the business boundary: Are tasks concentrated in coding, agents, long documents, or multimodality? If so, this type of model is worth serious evaluation.
- Review the license: Assess revenue, monthly active users, and external service models to determine whether an additional agreement is triggered. If necessary, contact the model provider in advance.
- Calculate the memory budget: Combine weights, KV cache, and headroom. Confirm whether 8 GPUs are sufficient, or whether trade-offs are needed based on business context length.
- Verify inference stack support: Check whether existing inference engines support low-bit MoE kernels, sparse attention, and KV quantization.
- Build a private evaluation set: Validate quality with real samples before deciding whether to replace or run in parallel with existing models.
- Archive assets and licenses: Bind and archive weight checksums, license text, and version numbers to prepare for audits and renewals.
- Plan cost and elasticity: Combine cloud capabilities to design scaling and degradation strategies, avoiding insufficient capacity during peak periods.
Trillion-parameter open-weight models push both model capability and deployment thresholds to a new level. They create an opportunity for broader access to advanced models, but they also pose a comprehensive test of license understanding, hardware planning, and engineering capability. Calculating the costs and constraints first is the safest path to production adoption.
