2.8T-Parameter Open-Weight Model Released: 1.56TB Weights and an 8-GPU Minimum for Local Deployment
A leading large-model team has released the full weights for a 2.8T-parameter MoE model, distributed as roughly 1.56TB across 112 files and 96 shards. The model uses 104B active parameters per token, 896 experts, a 1M-token context window, native multimodal input, and MXFP4/MXFP8 quantization-aware training. This article breaks down the architecture, quantization strategy, benchmark profile, custom license terms, and the practical deployment threshold, where eight large-memory accelerators are only the starting point.
1. Event Recap: A Massive Weight Release Arrives as Promised
On November 4, a leading large-model team fulfilled an earlier commitment and released the complete weight package for its flagship model. The package is enormous: 2.8 trillion total parameters, a download size of about 1.56TB, and a distribution made up of 112 files and 96 shards. It is currently the largest freely downloadable open-weight model by parameter count.
The timeline is straightforward. The model first launched through API and web chat on October 24, when the team promised that full weights would be available by November 4. The weight repository was created two days early and opened on the promised date, with no application required for download.
Readers should set expectations early: the “open” label comes with two restrictions, and the local runtime threshold is far higher than for ordinary open models. Eight large-memory accelerators are only the starting configuration.
2. Architecture Breakdown: How the 2.8T Parameters Are Structured
To understand the weight package, start with its MoE structure. According to the model card, it can be broken down along several dimensions:
- Overall scale: 2.8T total parameters, but only 104B parameters are activated per token. This is the typical MoE pattern: a large parameter count with a smaller compute footprint. The model “knows” more, but each inference call uses only a subset of experts.
- Expert configuration: 896 experts in total, with each token routed to 16 experts. Two additional shared experts always participate in computation. The shared experts help capture general knowledge across requests, while routed experts provide specialization.
- Network depth and context: 93 layers, with support for a 1 million token context window, placing it at the leading edge of current long-context capabilities.
- Multimodal capability: The model is positioned as a native multimodal agent model with a built-in vision encoder. It can process image input directly rather than relying on external OCR or a separate vision model chained into the pipeline.
Compared with the previous generation, the architecture changes in only one place: 24 layers of full attention are replaced with a sparse attention mechanism (DSA). This change directly serves long-context scenarios. When context extends to the million-token level, the quadratic complexity of full attention becomes a compute bottleneck, and sparsification is a widely recognized response.
3. Quantization Strategy: Why 1.56TB Can Hold 2.8T Parameters
One detail is worth calculating. If 2.8T parameters were stored in common BF16 format at 2 bytes per parameter, the theoretical size would be about 5.6TB. The actual download is only 1.56TB. The difference comes from quantization-aware training (QAT):
- Weight format: MXFP4, a 4-bit-level scaled floating-point format at roughly 0.5 bytes per parameter.
- Activation format: MXFP8, keeping 8-bit precision during the compute stage.
- Key distinction: This is not a post-training quantization step applied after training. The model is optimized for the low-precision format during training, so it can maintain stable performance under low-bit representation.
This means developers receive only this quantized version. The official release does not provide BF16 or FP8 high-precision weights. For teams that need to continue pretraining or perform high-precision fine-tuning on top of it, this is a limitation that must be evaluated. Full training on low-bit weights introduces additional numerical risk; in practice, it is usually suitable only for LoRA-style lightweight fine-tuning or direct inference.
4. Benchmark Profile: Strong in Coding and Agents, Weaker in Science and Some Reasoning
The vendor-reported comparison table shows a clearly uneven profile:
- Strengths: Coding and agent benchmarks perform well. On a terminal task benchmark, it scores 65.7%, slightly below a leading international competitor at 66.4%, but significantly above another competitor at 58.2%. On a long-horizon software engineering benchmark, it scores 51.7%, ahead of a reference competitor at 46.3%.
- Weaknesses: On a science benchmark, it scores only 38.1%, compared with 63.3% for a reference competitor. On a high-difficulty reasoning benchmark, it scores 39.4%, compared with 60.9% for a reference competitor. The gap is significant.
- Overall assessment: The vendor claims that 16 of 37 benchmarks surpass the leading competitor. A benchmark-by-benchmark comparison shows a mixed picture. A third-party evaluation organization’s intelligence index gives it 55 points and ranks it fourth globally.
The implication for developers is clear. If your business centers on code generation, tool calling, or multi-step agent orchestration, the model’s price-performance is worth serious evaluation. If your workload depends heavily on scientific reasoning or complex logic chains, be cautious about the “strongest” label in vendor marketing and test the model on your own task set.
5. License Analysis: Open Weights Are Not the Same as Open Source
This is the part of the release that deserves the most separate attention. The weight repository carries a custom license agreement, labeled by the platform as license: other, rather than a standard open-source license such as MIT or Apache 2.0. Its terms can be summarized as an MIT-style main grant plus two large-scale commercial restrictions:
- Main grant: Free use, copying, modification, distribution, fine-tuning, and commercial use are allowed. This part is consistent with the spirit of MIT.
- Restriction one: Companies that offer the model as a “model-as-a-service” product and have cumulative revenue exceeding $20 million over 12 months must sign a separate agreement with the release party before commercial use.
- Restriction two: Commercial products with monthly active users above a certain threshold, around the 100 million scale in the original text, also trigger additional agreement requirements.
Under the Open Source Initiative’s definition of “open source,” a license with use-scenario restrictions does not satisfy the requirement to be “licensed to anyone without discrimination.” Strictly speaking, therefore, this is “open weights,” not “open source.” For most small and medium teams and individual developers, the practical freedom is close to MIT. For top cloud vendors and large commercial products, however, this is a threshold that requires legal review. The pattern of using an open ecosystem to gain influence while negotiating terms for large-scale commercial use is becoming an industry convention for very large model releases.
6. The Real Threshold for Local Deployment
How many resources does it take to run 1.56TB of weights? A rough calculation helps:
Weight size: ~1.56 TB (MXFP4)Activations / KV cache: Scales with context length and concurrencyMinimum inference config: 8 large-memory accelerators to startStorage recommendation: 2TB+ high-speed NVMe, with room for decompression and shardsDownload cost: ~3.5 hours on 1 Gbps; more than a day on 100 Mbps
Even with weights compressed to 4-bit, 1.56TB still means that eight GPUs are the entry line. It also usually requires large host memory for expert-weight offloading and scheduling. MoE models activate few parameters per token, but the total parameter set must remain resident or addressable. This tension determines the cost structure of local deployment. For individual developers, getting it running locally is more symbolic than practical. For enterprises, there are three more realistic paths:
- API calls: Zero operations, pay-as-you-go, suitable for validation stages and workloads with volatile traffic.
- Cloud hosting: Deploy private instances through mainstream cloud platforms’ LLM services, such as Baidu AI Cloud’s LLM services and GPU compute offerings, balancing data isolation and elastic scaling.
- Self-built clusters: Cost-effective only in scenarios with extremely high data compliance requirements, stable call volume, and sufficient scale.
7. Conclusion: What a Very Large Open-Weight Model Signals
This release sends two signals. First, the scale ceiling of the open-weight camp has been substantially raised. The combination of MoE and low-bit QAT makes “downloadable trillion-parameter models” a reality. Second, the refined license terms show that the open ecosystem is moving from unconditional sharing toward tiered commercialization. For developers, the pragmatic approach is to think clearly about three things: the model’s uneven performance profile, the license boundary, and deployment cost. Only then should you decide whether to call an API, use cloud hosting, or actually request racks and fill eight GPUs.
