Open-Weight 2.8T MoE Model Released: Architecture, Benchmarks, License Terms, and Local Deployment
A newly released open-weight sparse MoE model has 2.8T total parameters, activates about 104B parameters per token, ships approximately 1.56TB of MXFP4 weights, and requires at least eight high-performance accelerators for local inference. This article examines its architecture, published benchmark strengths and weaknesses, custom Pro License restrictions, deployment costs, and practical adoption considerations for technical decision-makers.
As large language models move into the trillion-parameter era, the open community has recently received a major weight release: a sparse mixture-of-experts (MoE) model with 2.8T total parameters. Its quantized weight files total about 1.56TB, and the official documentation explicitly states that local inference requires at least eight high-performance accelerators. This is not only a breakthrough in scale, but also a trigger for broader discussion about the boundaries of “open weights” and practical deployment. This article analyzes the model across four dimensions: technical architecture, performance, license terms, and deployment cost, providing an objective and detailed reference for technical decision-makers.
1. Model Architecture: The Design Logic Behind 2.8T Total Parameters
The model uses a typical MoE architecture with 2.8T total parameters, but activates only about 104B parameters per token during inference. This sparse activation mechanism is the key to balancing capability and compute. Specifically, the model contains 896 expert networks and dynamically selects 16 experts per inference, along with two always-active shared experts that capture general knowledge. The model has 93 layers, supports a 1 million token context window, and includes a vision encoder for native multimodal understanding, allowing it to process both image and text inputs.
Notably, the model was not trained from scratch. It was iterated from the previous-generation K3 base. Compared with the previous generation, there is only one significant architectural adjustment: the full attention mechanism in 24 layers was replaced with sparse attention. This change is intended to reduce compute and memory overhead when processing long sequences, making a 1 million token context feasible under reasonable resources. The weights are released in MXFP4 format, with MXFP8 activations. This is a direct product of quantization-aware training: the official release provides only this quantized version, without BF16 or FP8 unquantized versions. As a result, users cannot directly obtain original-precision weights and must accept the accuracy loss caused by quantization, although the vendor claims the impact is minimal.
2. Performance Benchmarks: Clear Strengths, Obvious Weaknesses
According to official data, all from vendor self-testing, the model performs strongly on coding and agent tasks. On a terminal-operation benchmark, it scored 65.7%, only 0.7 percentage points behind the strongest closed model at 66.4%, and significantly above another mainstream commercial model at 58.2%. In a software-engineering marathon-style test, it scored 51.7%, surpassing the closed competitor at 46.3%. However, it shows disadvantages in scientific reasoning and some high-difficulty reasoning tasks: it scored 38.1% on a science benchmark, far below the competitor’s 63.3%, and 39.4% on diamond-tier hard problems, while the competitor reached 60.9%. The vendor says it exceeds the strongest closed model on 16 of 37 benchmarks, producing an overall distribution characterized by “strong coding/agent performance, weak science/complex reasoning.” Another independent evaluation organization gives it an intelligence index score of 55, ranking fourth globally, but that index combines multiple task categories and does not disclose specific weights.
For developers, the implication is that the model is better suited to code generation, tool calling, automation tasks, and similar scenarios. For rigorous scientific computing or high-order logical reasoning, it should be evaluated carefully and may need to be combined with other specialized models or post-processing mechanisms.
3. License: Open Weights Are Not Open Source, and Commercial Restrictions Require Attention
The most easily overlooked yet most critical point about this release is the nature of the license. The model does not use a standard open-source license such as MIT or Apache 2.0. Instead, it uses a custom “Pro License.” From the license text, the main body does grant rights for free use, copying, modification, distribution, fine-tuning, and commercial use, but it adds two restrictions aimed at large-scale external business:
- Service revenue threshold: If any company provides model-as-a-service (MaaS) based on this model and its 12-month cumulative revenue exceeds US$20 million, it must sign a separate commercial agreement with the model copyright holder before commercial use.
- Monthly active user threshold: If a commercial product exceeds 100 million monthly active users, an additional negotiated license is also required. The source text is incomplete at this point, so this threshold is an inference.
These two restrictions effectively raise the ceiling for “free commercial use.” Small and medium-sized teams and pilot projects can use the model freely, but once a business scales, the cost is no longer zero. This also explains why the official release calls it “open weights” rather than “open source”: the latter usually requires a license certified by the Open Source Initiative (OSI) and prohibits additional restrictions. Developers should assess their business growth at the project-planning stage to avoid being pushed into passive commercial negotiations later.
4. Local Deployment: Eight GPUs Is Not a Gimmick but a Hard Quantized Threshold
The weight release contains 112 files, including 96 safetensors shards, totaling about 1.56TB. The official documentation clearly states that running the model requires at least eight accelerators with high VRAM, such as 80GB or more. This is an estimate based on the memory requirements of the MXFP4 quantization format: 2.8T parameters at 4-bit imply a theoretical memory requirement of about 1.4TB, excluding KV cache and activations. Eight 80GB cards provide 640GB in total, which is clearly insufficient. In practice, the model relies on multi-card tensor parallelism and expert parallelism, and the KV cache for a 1 million token context will consume hundreds of additional gigabytes. Therefore, eight cards are only the minimum startup configuration. To achieve the full 1 million token context, the actual number of cards required may double.
For deployment, mainstream inference frameworks already support automatic parallel sharding for sparse MoE, but users must handle the download and merging of model shards themselves. Community mirrors can help with downloading model shards and may reduce network bottlenecks. In addition to VRAM, CPU memory, motherboard PCIe lanes, high-speed storage (NVMe is recommended), and communication bandwidth (such as NVLink or high-speed Ethernet) can all become bottlenecks. For most small and medium-sized businesses, renting cloud GPU instances is a more realistic path than purchasing hardware.
5. Practical Recommendations: How to Evaluate Whether Adoption Is Worthwhile
- Clarify the task type: If the project focuses on code assistance, agent workflows, or long-text analysis, the model’s cost-performance ratio may be better than closed APIs. If it involves high-precision reasoning, run internal evaluations first.
- Quantify cost and benefit: The US$20 million revenue threshold may seem distant, but if the business grows rapidly, later negotiation costs are unpredictable. Prioritize non-core paths to isolate risk.
- Consider the quantization compromise: MXFP4 weights have compatibility requirements for inference frameworks. Confirm that the framework in use supports this format; otherwise, forced dequantization may sharply increase VRAM requirements.
- Monitor model updates: Open-weight models iterate quickly, so migration costs for future upgrades should be planned.
Overall, this release raises the ceiling of open-weight large models to 2.8T parameters and demonstrates the feasibility of MoE architectures at extreme scale. However, its nonstandard license, single quantization format, and high deployment barrier all remind us that true “open-source AI” is still in an exploratory phase. Between technical breakthroughs and ecosystem maturity, there is still a long way to go. Developers should remain cautiously optimistic: recognize the capability leap while also accounting for every hidden cost.
