Deploying a 2.8T Open-Weights MoE Model Locally: Specs, License, and Memory Requirements
This guide walks through local deployment considerations for a newly released 2.8T open-weight multimodal MoE model. It explains the parameter structure, expert routing, MXFP4/MXFP8 quantization, vendor-reported benchmark results, and custom license constraints, then estimates memory requirements for an eight-accelerator setup. It also compares self-hosted, workstation-offloaded, and cloud-hosted inference options, and covers download verification, parallel deployment strategies, and operational monitoring for teams evaluating whether to run the model in-house.
1. What Happened: A Conditional Open Release
On November 4, a leading model vendor released the complete weights of its flagship multimodal MoE model as promised. The model has 2.8T total parameters, and all weight files together are about 1.56 TB. The vendor described it as the largest open-weight model currently available for download.
The timeline was as follows: the model first became available through an API and a web interface on October 24, and the vendor promised to release the complete weights before November 4. The weight repository was created on a model hosting platform on November 2. It remained closed on the morning of November 4 and was officially opened that evening, allowing downloads without an application. The repository contained 112 files, including 96 binary weight shards totaling about 1.56 TB.
However, this release came with two prerequisites:
- The license is not a standard open-source license: the repository labels the weights as a custom model license.
- The hardware requirement is high: the official guidance says local execution requires at least eight accelerator cards.
To judge whether an open-weight model is worth the investment, three questions are useful:
- What is in the weights: how parameters are organized determines compute, memory, and inference-engine support.
- Whether commercial use is legally allowed: license terms determine the permitted scope.
- Whether deployment cost is manageable: a memory bill is enough to determine whether a plan is feasible.
The rest of this article examines these questions in order.
2. Breaking Down 2.8T: Parameter Structure and Inference Implications
2.1 Overall Specifications
| Dimension | Specification | Practical impact |
|---|---|---|
| Model type | Native multimodal agent model, continued training from the previous generation in the same family | Text, images, and tool calls share one set of weights; native visual understanding |
| Total parameters | 2.8T | Main driver of VRAM capacity and bandwidth requirements |
| Activated parameters per token | About 104B | Main driver of compute and compute cost |
| Expert structure | 896 routed experts, 16 selected per token, plus 2 shared experts | Expert-sharded deployment; communication overhead grows with GPU count |
| Layers | 93 | Basic factor in VRAM estimation |
| Context length | 1 million tokens | Attention compute and KV cache grow with sequence length |
| Vision encoder | Proprietary vision encoder | Released together with the text weights |
| Weight format | MXFP4 weights, MXFP8 activations | Only one quantization version is provided; no higher-precision version |
Compared with the previous generation in the same family, the vendor says only one structural change was made: full attention in 24 layers was replaced with sparse attention. The remaining layers keep full attention.
This change was designed to save compute for long context, but it imposes three requirements on the inference engine:
- It must support attention kernels for sparse attention.
- It must maintain different KV layouts for sparse-attention layers and full-attention layers.
- It must be able to fall back to dense attention in extreme cases to preserve correctness.
Before deployment, confirm that the target inference engine explicitly supports this structure. Otherwise, the engine may report errors, or incorrectly fall back to full computation in sparse layers, significantly reducing long-context performance.
2.2 Understanding the MoE Cost Model Through the Activation Ratio
The Mixtral paper proposed two empirical rules:
- Compute depends on activated parameters: MoE FLOPs are proportional to the number of parameters activated per token.
- Memory depends on total parameters: weights must remain resident in VRAM or memory throughout inference.
Because memory is determined by total parameters and compute is determined by activated parameters, the industry has further proposed the metric of parameter efficiency:
Parameter efficiency = activated parameters / total parameters= 104B / 2.8T ≈ 3.7%
Here, 104B is taken directly from the official model card under the “per activation” definition, assuming that all activated parameters participate in the computation for every token. This differs slightly from measured forward FLOPs, but it is sufficient for order-of-magnitude analysis.
A 3.7% activation ratio means:
- Each token uses only about 1/27 of the weights.
- The compute for a single forward pass is comparable to a dense model in the tens-of-billions-parameter range.
- The VRAM requirement, however, must be paid for all 2.8T parameters.
This is a typical characteristic of very large sparse MoE models: trading VRAM and communication for compute. It leads to three implications:
- The hardware bottleneck shifts from “not enough compute” to “not enough VRAM.” Selection should first consider per-GPU capacity and inter-GPU interconnect.
- Inference is memory-access-intensive: for every generated token, activated expert weights must be read from VRAM. Decode speed depends more on memory bandwidth and on benefits from cache hits for activated experts.
- VRAM usage is independent of which experts are activated by the current token, because experts remain resident in VRAM and cannot be swapped in and out on demand.
With only 16 out of 896 experts participating in routing, about 1.8%, new engineering problems emerge:
- Load imbalance: if token routing is uneven, a small number of “hot experts” can become latency outliers.
- Pipeline bubbles: under expert parallelism, uneven load across GPUs can lengthen synchronization waits.
- Fine-tuning constraints: models with high parameter efficiency are more sensitive to perturbations in routing and activation patterns. Fine-tuning should preferably use parameter-efficient fine-tuning to avoid damaging pretrained routing.
2.3 Inferring Quantization From Model Size: What 1.56 TB Indicates
MX, or Microscaling, is a class of microscaled quantization formats. A group of values, usually 32 elements, shares a low-precision scale factor. The dequantization formula is:
real_value = quant_value × scale
1.56 TB ÷ 2.8T ≈ 4.46 bits/parameter
Four-bit raw data plus one 8-bit scale factor per 32 elements theoretically gives about 4.25 bits/parameter, close to the observed value. This indicates:
- Weights are primarily 4-bit, and 8-bit scale factors also add overhead.
- Activations use 8-bit, offering higher precision than the weights.
This is an asymmetric quantization approach: weights are compressed more aggressively to save VRAM, while activations retain higher precision to protect accuracy. This is common in quantization-aware training. It leads to two constraints:
- No higher-precision official version: the vendor released only this quantized version, with no BF16 or FP8 weight variant. Downstream users who need higher precision can only dequantize the weights themselves or continue training, and they cannot avoid the quantization error introduced in the original weights.
- Format compatibility risk: MXFP4 and MXFP8 are relatively new formats. Some inference engines or hardware lack native support. Before deployment, confirm that the target engine has corresponding dequantization kernels or can load the model through a generic block-dequantization path.
3. How to Read Benchmark Results: The Right Way to Interpret Vendor-Reported Data
The vendor published the following comparison data. All numbers are vendor-reported, and competitors are anonymized as A and B:
| Evaluation | Focus | This model | Competitor A (flagship) | Competitor B |
|---|---|---|---|---|
| Terminal Coding Benchmark 4.0 | Completing engineering tasks in terminal environments | 65.7% | 66.4% | 58.2% |
| Long-Horizon Software Engineering Task Benchmark | Cross-file, long-chain code modifications | 51.7% | 46.3% | — |
| Terminal Coding Benchmark - Science Subset | Scientific computing tasks | 38.1% | 63.3% | — |
| Cross-disciplinary Hard Reasoning Evaluation (Diamond subset) | High-difficulty disciplinary reasoning | 39.4% | 60.9% | — |
A third-party evaluation organization gave the model a composite intelligence index of 55, ranking it fourth globally. The vendor also said that 16 of 37 benchmarks outperformed Competitor A.
Looking at the table, the results are mixed: the model performs relatively strongly on coding and agent tasks, but weaker on science and some high-difficulty reasoning tasks.
The competitor names and data in the table are self-reported by the vendor and have not been independently verified. They are shown only to illustrate capability distribution and should not be treated as a selection conclusion.
When reading these results, watch out for several issues:
- Self-reported methodology: prompts, sampling parameters, and scoring scripts can significantly affect scores.
- Benchmark saturation: training-data contamination is hard to rule out for high-visibility benchmarks.
- Difficulty stratification: difficulty subsets, such as the science subset or the Diamond subset, often reveal weaknesses more clearly than overall rankings.
- Composite scores: third-party indices depend on the composition of the evaluation suite, so looking at ranking alone has limited meaning.
For practical deployment, use 50–200 of your own tasks for small-sample validation. Fix decoding parameters for each model, such as temperature, top-p, and maximum output length, run each configuration 3–5 times, record the mean and variance, and compare the results with the current solution.
# Example: run each task in your own evaluation set multiple timesrun_one () { # $1: task ID, $2: repeat indexcurl -s "$API_BASE/chat/completions" \-H "Content-Type: application/json" \-d @<(python3 - "$1" <<'PY'import json, systask = open(f"tasks/{sys.argv[1]}.json", encoding="utf-8").read()print(json.dumps({"model": "default", "temperature": 0,"messages": json.loads(task)}))PY)}for i in $(seq 1 100); dofor r in $(seq 1 3); dorun_one "$i" "$r" >> results.jsonldonedone
4. License: Open Weights Do Not Mean Open Source
This is the part most easily overlooked, yet it determines commercial boundaries.
The repository uses a vendor-defined custom model license. On the hosting platform, it is marked as “Other/Custom” rather than as a standard open-source license such as MIT or Apache 2.0. The vendor said the terms are verbatim identical to the previous generation’s license, only with a new name.
The core grant is MIT-style permissive: free use, copying, modification, distribution, fine-tuning, and commercial use. Two additional restrictions are layered on top, both aimed only at large-scale external business:
- Providing the model as a service (MaaS) externally, with cumulative revenue exceeding $20 million over a 12-month period: a separate agreement with the vendor is required before commercial use.
- A commercial product exceeding a specified monthly-active-user scale: a separate authorization is also required.
Three conclusions follow:
- For most enterprises, this license does not prohibit internal use or commercial integration. The restrictions mainly target businesses that have already reached a significant scale.
- The difference between “permissive grant plus separate agreements for large customers” and a standard open-source license is that the final interpretation of custom terms belongs to the licensor. If a dispute arises, teams cannot rely on open-source community conventions.
- Large teams rushing toward commercialization should contact the licensor early to avoid triggering the thresholds after a product has already launched.
Enterprises can perform the following five-step self-check:
- Have legal counsel read the full license and confirm definitions, such as the statistical scope and consolidation scope of “revenue” and “monthly active users,” as well as obligations for derivative models.
- Record usage patterns, including internal versus external use, business revenue, and monthly active users, and periodically assess whether the thresholds are approaching.
- Confirm whether derivative models and fine-tuned artifacts remain subject to the same license.
- For external services, communicate with the licensor early about authorization.
- When evaluating alternatives, prefer models under standard open-source licenses.
5. Running It Locally: The VRAM Math Behind Eight Accelerators
5.1 Calculating VRAM Requirements
# Units: GiB (1 GiB = 2^30 bytes)# Estimate assumes 1 TB = 10^12 bytes; use only as an order-of-magnitude referenceTotal weights = 1.56 × 10^12 / 2^30 ≈ 1453 GiBPer GPU (8) = 1453 / 8 ≈ 182 GiB/GPUTotal per GPU ≈ weights + KV cache + activation/runtime overhead
Assuming 8 KV heads, head dimension 128, and FP8 KV cache:
Per token per layer KV = 2 × 8 × 128 × 1 byte = 2 KiBFull model = 2 KiB × 93 ≈ 186 KiB/token10k tokens ≈ 1.9 GiB100k tokens ≈ 18 GiB1M tokens ≈ 182 GiB
The KV values above are estimates under the assumed configuration. Actual requirements should be based on the model card and inference engine.
The conclusion is that eight GPUs mean each GPU needs more than 180 GiB of usable VRAM, which corresponds to high-end data-center accelerators in the 192 GB class. Eight such GPUs provide 8 × 192 GB = 1536 GB, just enough to hold about 1453 GiB of weights, leaving almost no room for KV cache, activations, or runtime overhead. Accelerators with 80 GB or 141 GB of VRAM, even eight of them, would provide only 640 GB or 1128 GB, which is not enough to hold the weights.
5.2 Three Deployment Paths
| Dimension | A. Self-built 8-GPU cluster | B. Single workstation + memory offloading | C. Cloud-hosted inference service |
|---|---|---|---|
| Hardware requirements | 8 GPUs in the 192 GB class with high-speed interconnect | 1–2 accelerators plus 512 GB or more of memory; larger NVMe storage is better | None |
| First-token latency | Low | Very high due to weight swapping in and out | Low to medium, depending on service load |
| Throughput | High; supports continuous batching | Low; suitable for single-user interaction | Limited by quotas and rate limits |
| Data control | Full autonomy | Full autonomy | Requires evaluation of data compliance and de-identification |
| Main cost | Hardware CapEx or long-term rental | One-time hardware investment | OpEx billed per token or time |
| Suitable users | Scaled businesses with MLOps teams | Developer evaluation and privacy-sensitive individual use | Rapid validation and spiky traffic |
A three-step decision approach is recommended:
- Start with C to validate value: test effectiveness on your own task set and calculate costs.
- Move to A when scale grows and data is sensitive: build a cluster and plan capacity based on concurrency.
- Use B only for evaluation and small traffic: because cold experts may be swapped in and out at every layer, the weights that need to be read for a single token far exceed the activated-parameter count, so generation speed drops significantly.
5.3 Parallel Deployment Strategy for a Self-Built Cluster
A 2.8T-parameter model cannot fit into a single GPU and must be sharded. Two types of parallelism need to be distinguished:
- Tensor parallelism (TP): splits weight matrices of a single layer across multiple GPUs. Each layer requires all-reduce synchronization and is suited to high-bandwidth interconnect within a node.
- Expert parallelism (EP): distributes the 896 routed experts across GPUs, about 112 per GPU. Tokens are routed to the corresponding GPUs through all-to-all, and results are gathered through all-to-all.
In production, TP and EP are often combined. The following is an illustrative configuration; key names are only examples, while parameter meanings are standard across the industry:
model_path: /models/k3-pro-mxfp4tensor_parallel_size: 4 # TP degree: generally no more than the number of GPUs per machinedata_parallel_size: 2 # number of data-parallel replicas: × TP = total GPU countenable_expert_parallel: true # enable expert parallelismexpert_parallel_size: 4 # 896 experts distributed across GPUs, about 224 per GPUmax_model_len: 262144 # set according to KV budget; no need to start with 1Mgpu_memory_utilization: 0.90enable_chunked_prefill: true # chunk long-context prefill to reduce instantaneous activation peakskv_cache_dtype: fp8 # KV cache quantization, significantly reducing long-context memory
Additional points:
- Separate prefill and decode. Prefill is compute-bound and compute-intensive; decode is memory-access-bound and generates tokens one by one. Under high concurrency, separate deployment may be considered because the two have different scaling rhythms.
- MoE-specific bottlenecks. Uneven expert routing can cause tail latency. Chunked prefill, load balancing, and capacity planning require specific tuning.
- Long context is not zero-cost. Sparse attention reduces attention compute, but KV cache memory still grows linearly with sequence length.
- Reserve VRAM margin. Leave 10%–15% for runtime fragmentation and activation peaks; do not fill memory to the theoretical limit.
5.4 Download and Verification
Downloading weights on the order of 1.5 TB makes integrity checks critical to later troubleshooting:
- Use a resumable download tool and verify each file against the repository’s checksum file. For sharded models, missing any shard may make loading impossible.
- Use metadata files to confirm shard numbers, order, and tensor-to-shard mapping. Shards are not simply concatenated; the loader reads them by index.
- After downloading, first estimate VRAM requirements, then start the service.
When deploying to the cloud, object storage cross-region replication or synchronization can be used to prewarm the weights in the target region. Then container orchestration can bring up inference instances. Logging and monitoring services can track latency, throughput, and VRAM usage.
5.5 Operational Points After Serving
- Metrics: first-token latency (TTFT), time per output token (TPOT), throughput (tokens/s), VRAM usage, and expert load distribution.
- Logs: redact sensitive fields when recording prompts and outputs, and set retention periods.
- Version management: pin the weight version and inference engine version. The structure in the model card is the basis for judging inference engine support.
- Upgrade strategy: quantized weights are more sensitive to engine versions. Before upgrading, run regression evaluations on a canary instance.
6. Conclusion
Returning to the three questions at the beginning:
- What is in the weights: a 2.8T sparse MoE with about 3.7% parameter efficiency and a single MXFP4/MXFP8 quantization version. This determines the high requirements for VRAM capacity and engine compatibility.
- Whether commercial use is legally allowed: a custom license with a permissive core and two large-customer clauses. Legal review should be completed before commercial scale is reached.
- Whether the cost is manageable: eight accelerators in the 192 GB class are a hard threshold. It is advisable to validate with cloud-hosted services first and then decide whether to build a cluster.
What open weights truly change is “controllability”: data can stay within your environment, the model can be fine-tuned for your business, and you are not locked to a single supplier. Whether it can be used effectively depends on three preparations: the VRAM math, the license boundary, and the evaluation system.
