2.8T MoE Model Deployment: GPU Memory and Licensing Checks First
Deploying a 2.8-trillion-parameter MoE model requires more than checking its active parameter count. This guide examines the reported architecture, MXFP4 quantization, task suitability, and custom license, then walks through GPU memory sizing, KV cache estimates, download verification, storage planning, and cloud infrastructure checks before deployment.
1. Architecture: 2.8T Total Parameters, About 104B Active per Token
The model is further trained from its predecessor’s base model and positioned as a native multimodal agent model. It uses a mixture-of-experts (MoE) architecture with 2.8 trillion total parameters, but only about 104 billion parameters are activated for each token. This provides substantial model capacity while keeping per-token computation more manageable than the total parameter count might suggest.
According to the publisher, the model has 896 expert networks. Its router selects 16 experts for each token, while two shared experts always participate to support consistent extraction of foundational features. The model reportedly has 93 layers, a one-million-token context window, and a separate vision encoder.
These specifications come from the publisher’s public materials and have not been independently verified item by item. Check the model repository’s configuration files for the expert count, layer count, maximum context length, and related settings.
One of the most significant changes from the predecessor is in the attention layers: 24 layers reportedly switch from full attention to sparse attention. Full attention has quadratic computational complexity with respect to context length. Sparse attention restricts each token to a local region or tokens at selected intervals, reducing computation and making million-token inference more practical.
However, sparse attention has limits:
- It primarily reduces computation, not necessarily KV cache memory. Cache savings depend on whether the inference engine retains only the KV data needed by the attention pattern.
- Speedups depend on kernel implementation, batch size, and sequence length.
- Evaluation should record output tokens per second, time to first token, and peak GPU memory, using a full-attention baseline at the same input length.
2. Quantization: MXFP4 Weights and MXFP8 Activations
Storing 2.8 trillion parameters in BF16 would require approximately 5.6 TB for weights alone. The publisher therefore uses low-bit quantization: MXFP4 for weights and MXFP8 for activations.
MXFP4 uses 4 bits per element, with one shared 8-bit scale for every 32 elements. This averages approximately 4.25 bits per parameter:
Average bits per parameter = 4 + 8 / 32 = 4.25
At that rate, 2.8 trillion parameters would occupy roughly 1.49 TB, close to the reported 1.56 TB download size. The difference may reflect unquantized components or metadata, but this has not been verified.
The publisher describes this version as the result of quantization-aware training (QAT), which introduces quantization noise during training. Post-training quantization (PTQ), by contrast, compresses a model after training. Their accuracy characteristics can differ; consult the model card for the specific method used.
At the time described in the source materials, the official release includes only this quantized version, with no BF16 or FP8 alternative. A direct comparison against a higher-precision release is therefore unavailable. Alternative reference points include:
- The predecessor’s base model on the same tasks.
- Human-labeled reference answers.
- Variation across repeated samples from the same model, as a baseline for output noise.
Check Hardware and Framework Support
MXFP4 support varies by chip. GPUs or ASICs with native support can compute directly in the format. Other hardware may need to dequantize weights to a higher precision, reducing the expected speed and memory benefits.
Before deployment, confirm:
- Whether the target hardware and inference framework provide MXFP4 compute kernels.
- Whether dequantization is required, and whether it occurs during loading or at runtime.
- Accuracy, throughput, and peak GPU memory on your own evaluation set.
The evaluation set should include both representative inputs and edge cases. Fix and record the prompt template, decoding settings—including temperature, top_p, and max_tokens—and the number of repetitions. Run each test case at least three times to observe output variability.
3. Capabilities: Stronger in Coding and Agent Tasks, Weaker in Scientific Reasoning
The publisher’s materials indicate stronger performance in coding and automated agent tasks. Expert specialization in an MoE architecture may help support capabilities such as code generation, tool use, and task orchestration.
The reported performance is weaker than that of leading closed-weight models on scientific reasoning and multi-step mathematical reasoning.
This article does not reproduce benchmark scores. Benchmark names, versions, evaluation protocols, and test dates can materially affect results, so unverified numbers should not drive model selection.
| Task category | Relative performance reported by the publisher | Selection guidance |
|---|---|---|
| Code generation and repository analysis | Stronger | Include it as a candidate and validate it on internal code tasks |
| Tool use and task orchestration | Stronger | Consider it for agent projects, with particular attention to tool-call reliability |
| Long-document information extraction | Potential advantage, subject to long-context testing | Test recall and accuracy on real long documents first |
| Scientific computation and multi-step mathematical reasoning | Relatively weaker | Do not use it as the sole model; require human review |
For reproducible evaluation, fix the dataset version, prompt template, decoding parameters, and random seed. Document the human-review sample size—for example, 20 cases per task category.
Use the same scoring criteria across all candidate models. Otherwise, the results will not be directly comparable.
4. Licensing: Custom Open Weights, Not OSI-Standard Open Source
The model uses a custom license rather than a standard license such as MIT or Apache 2.0. Its core permissions resemble MIT-style terms, allowing use, copying, modification, distribution, fine-tuning, and commercial use.
However, it imposes additional conditions on two categories of large commercial entities:
- Companies providing the model as a service whose revenue exceeds a specified threshold over 12 consecutive months.
- Entities whose commercial products exceed a specified monthly active user threshold.
Commercial entities meeting either condition must enter into a separate agreement with the publisher before commercial use. The specific thresholds are not reproduced here; verify them in the applicable license.
License Review Checklist
- Locate the threshold clauses. Find the revenue and monthly active user thresholds in
LICENSE, and record the relevant clause numbers. - Check the definition of model as a service. Determine whether your offering falls within it, including whether usage-based external access is covered.
- Confirm measurement periods and scope. Check whether revenue is measured over 12 consecutive months or another period, whether monthly active users use calendar months or rolling windows, and whether affiliates are included.
- Archive the license version. Record the publication date, version number, and SHA-256 hash of the downloaded license file.
- Obtain an agreement if required. If the additional conditions apply, contact the publisher and complete the agreement before commercial use.
Small and midsize teams may not reach these thresholds, but they still need to review every applicable clause. Large internet companies and cloud providers should obtain legal approval before committing to the project.
5. Deployment Sizing: Can Eight GPUs Run It?
The answer depends on memory per GPU. The reported weight files total approximately 1,560 GB in decimal units. Fewer GPUs mean each device must hold a larger share of those weights.
5.1 A Quick Weight-Memory Check
The following table compares aggregate capacity with the reported weight size. It excludes KV cache, activations, and framework overhead.
| Memory per GPU | Total with 8 GPUs | Total with 16 GPUs | Can the weights fit? |
|---|---|---|---|
| 80 GB | 640 GB | 1,280 GB | Neither configuration fits; roughly 25 GPUs are needed with a 20% reserve |
| 141 GB | 1,128 GB | 2,256 GB | Eight do not fit; 16 fit the weights |
| 192 GB | 1,536 GB | 3,072 GB | Eight fall short by about 24 GB, with no headroom; 16 fit the weights |
| 288 GB | 2,304 GB | 4,608 GB | Eight theoretically fit, leaving about 744 GB before runtime allocations |
Among these configurations, eight-GPU deployment requires the 288 GB memory tier. Even eight 192 GB GPUs cannot hold all the reported weights.
Actual deployment also needs room for KV cache, activations, and framework overhead. A preliminary GPU-count estimate is:
N ≥ W / [M × (1 − r)]
Where:
Nis the required GPU count, rounded up.Wis total weight memory.Mis memory per GPU.ris the reserved fraction of GPU memory.
A starting reserve of 15%–25% is reasonable for planning, but must be validated through testing. With a 20% reserve and 1,560 GB of weights, the estimates are:
| Memory per GPU | Estimated minimum GPU count |
|---|---|
| 80 GB | 25 |
| 141 GB | 14 |
| 192 GB | 11 |
| 288 GB | 7 |
Round these counts up to a parallel configuration supported by the framework, such as 8, 16, or 32 GPUs.
The reported 896 experts divide evenly across 8, 16, and 32 partitions. Whether the inference framework supports the corresponding expert-parallel layout remains to be verified.
5.2 Estimating KV Cache Memory
For a conventional KV cache, estimate memory as:
KV cache bytes =2 × layers × KV heads × head_dim × sequence length × batch size × bytes per element
The factor of two accounts for separate key and value caches. Read the actual layer count, KV head count, and head_dim from the model configuration before calculating production requirements.
Consider this illustrative configuration, which is not a verified description of the model:
- 93 layers
- 8 KV heads
head_dimof 128- BF16 cache storage at 2 bytes per element
- Sequence length of 1,000,000 tokens
- Batch size of 1
The cache footprint per token would be:
2 × 93 × 8 × 128 × 2 = 380,928 bytes
For one million tokens:
380,928 × 1,000,000 ≈ 381 GB
A single million-token request can therefore require hundreds of gigabytes of KV cache in this example.
This estimate assumes cache storage across all 93 layers. It would overestimate memory if the implementation retained KV only for the full-attention layers. FP8 cache storage could halve the footprint relative to BF16, but its accuracy impact requires separate evaluation.
5.3 Activation and Framework Headroom
Activation memory and framework overhead grow with batch size and concurrent requests. Begin with a 15%–25% reserve, then adjust it through load testing.
Record peak GPU memory, not just average usage. A configuration that fits under light load may fail at peak concurrency.
5.4 Download Verification and Storage Planning
Use the following download procedure:
- Obtain the hash manifest from the release materials and record the expected SHA-256 value for each shard.
- Download shards with a tool that supports resumable transfers. Retry only failed shards.
- Verify every completed shard against its expected SHA-256 value. Download failed shards again individually rather than overwriting the entire model.
- Do not unpack or load files that fail verification.
Estimate storage requirements as:
Storage required ≈1.56 TB × number of copies+ temporary space for unpacking or format conversion+ approximately 10% additional headroom
For example, two copies require 3.12 TB. Adding 0.5 TB of temporary space brings the subtotal to about 3.6 TB, or roughly 4 TB with headroom.
Transfer speed also affects preparation time:
| Transfer rate | Approximate time for 1,560 GB |
|---|---|
| 1 GB/s | 26 minutes |
| 100 MB/s | 4.3 hours |
For multi-node deployments, first measure storage read speed on one machine before estimating overall model-loading time.
Common risks include disks filling during download, partially written files being mistaken for complete shards, and files being checked only at download time. Verify them again before loading and retain the verification logs.
5.5 Choosing Cloud Infrastructure
Compare cloud configurations on:
- Memory per GPU.
- Inter-GPU interconnect bandwidth.
- Native MXFP4 compute support.
- Framework support for MoE expert parallelism.
- Price.
Expert parallelism introduces substantial communication between GPUs. Insufficient interconnect bandwidth can significantly reduce throughput.
A simple monthly GPU cost estimate is:
Monthly cost ≈hourly price per GPU × GPU count × operating hours per day × 30
Before selecting an instance or cluster, check that:
- Aggregate GPU memory can hold the weights, KV cache, and runtime reserve simultaneously.
- The interconnect can handle expert-parallel communication.
- The inference framework supports MXFP4 weight loading and the required MoE parallelism.
- Suitable sparse-attention kernels are available. If the framework falls back to full attention, memory use and latency will differ.
- The framework provides paging or offloading strategies for million-token contexts.
Use each cloud provider’s official documentation for current GPU models, specifications, and prices.
6. Who Should Evaluate It—and What to Verify Next
Suitable Candidates
Teams can reasonably move into evaluation if they:
- Already have a high-memory cluster with 16 or more GPUs and workloads centered on code generation, tool use, or agent orchestration.
- Have confirmed that the additional license conditions do not apply, or have already signed the required agreement with the publisher.
Poor Fits or Premature Deployments
Deployment is less suitable when:
- The available hardware is eight 80 GB GPUs or less. This is insufficient for local deployment of the full model; consider a smaller model.
- The main workload is scientific computation or multi-step mathematical reasoning. Do not rely on this model alone; retain human review.
- A large internet company or cloud provider has not completed legal review. Resolve the licensing requirements before committing to the project.
Recommended Validation Order
- Verify the model card and configuration. Confirm the reported 896 experts, 93 layers, one-million-token context window, and other architectural details.
- Review
LICENSE. Complete the five licensing checks above. - Plan storage and verify downloads. Ensure sufficient capacity and complete SHA-256 validation.
- Test MXFP4 loading on the target hardware. Record peak GPU memory.
- Evaluate against relevant baselines. Use your own task set and record accuracy, throughput, and latency.
The practical deployment question is not simply whether 104 billion parameters are active per token. The full weight footprint, long-context cache requirements, quantization support, parallel execution, and licensing terms all need to fit before the model is ready for production.
