2.8T MoE Model Weights Released: License, Benchmarks, and 8-GPU Deployment
A Chinese model provider has publicly released the full weights of a 2.8T-parameter mixture-of-experts (MoE) model under a custom license. This article examines the model’s architecture, including 104B activated parameters, sparse attention, MXFP4/MXFP8 quantization, self-reported benchmark results, licensing constraints, and practical deployment considerations. It also covers memory budgeting, expert placement, inference configuration, and how to choose between local deployment, public cloud, and API access.
2.8T MoE Model Weights Released: License, Benchmarks, and 8-GPU Deployment
1. What Happened: A Promised Weight Release
On November 4, a Chinese model provider publicly released the complete weights of a 2.8T-parameter MoE model, following an earlier commitment. The weights were made available through a public model community and can be downloaded without application. The provider described the release as the largest openly downloadable weight set of its kind at the time.
The timeline was roughly as follows:
- October 24: The model became available through an API and a web interface. The provider also said the weights would be released by November 4.
- November 2: A corresponding repository appeared on the public model community, but it was closed.
- November 4, morning: The repository remained closed.
- November 4, evening: The repository was officially opened, and access no longer required an application.
A total of 112 files were released, including 96 weight shards, totaling about 1.56TB. This is much larger than any previous open-weight large model release.
The previous generation of the model had already generated substantial discussion around its performance. This article focuses on more practical questions after the weights are available: what is inside the release, how the license works, and whether ordinary developers can run it.
2. Model Architecture: 2.8T Total Parameters and 104B Activated Parameters
According to the official model card, this is a natively multimodal agent model continued from the previous generation. Its core specifications are:
| Dimension | Configuration |
|---|---|
| Total parameters | 2.8T |
| Activated parameters per token | 104B, about 3.7% of the total |
| Architecture | MoE with 896 routed experts and 2 shared experts; 16 routed experts selected per token |
| Layers | 93 |
| Context length | 1 million tokens |
| Vision | Native multimodal, with an in-house vision encoder |
| Quantization | Weights in MXFP4, activations in MXFP8; result of quantization-aware training |
1. Sparse Activation Changes the Compute Accounting
The key point of MoE is not the total parameter count, but the activation ratio. Only about 3.7% of parameters are computed per token. The compute per token is close to that of a dense model with tens of billions of parameters, while the model carries the knowledge capacity of a trillion-scale parameter system.
The routing mechanism deserves a closer look. For each token, 16 routed experts are dynamically selected by a gating network, and 2 shared experts are always active. Shared experts are suitable for carrying general-purpose knowledge because they participate in computation for every token.
Two details are important:
- 3.7% is only an average activation figure. The experts activated by a token generally need to remain resident in memory during decoding. Otherwise, each access would require loading them from storage, which would create unacceptable latency.
- Routing is dynamic. During decoding, the combination of experts activated by each new token can differ. In high-concurrency batches, many tokens are processed together, and the union of activated experts can expand rapidly. As a result, the amount of weights that must actually remain resident is often much larger than 104B. This is a key consideration for later capacity estimation.
2. Long-Context Trade-off: Sparse Attention Replaces Some Full-Attention Layers
Compared with the previous generation, the structure changes in one main way: full attention in 24 layers is replaced with sparse attention, referred to as DSA, in order to save compute for long-context workloads.
Attention computation and memory usage both grow quadratically with context length. Full-attention layers also need to retain the complete KV cache. With sparse attention, these layers only attend to a selected set of previous tokens. This can significantly reduce both long-prompt prefill computation and KV storage.
There are two trade-offs:
- Some long-range details may be lost, so retrieval-oriented evaluations are usually needed to verify behavior.
- Deployment requires the inference engine to support this sparse mode. Otherwise, it may fall back to a generic attention implementation, reducing the efficiency advantage of long-context inference.
3. Quantization Format: Saves Memory, Raises the Hardware Bar
The weights use MXFP4, a 4-bit floating-point format based on microscaling. Every 32 weights share an 8-bit scaling factor. The 4-bit value itself has a narrow range, while the scaling factor restores dynamic range. Activations use MXFP8.
This combination comes from quantization-aware training. The model was trained to adapt to low-precision inference rather than being compressed after training.
This creates two deployment prerequisites:
- The hardware must natively support FP4/FP8 data types, provided through both the accelerator and the lower-level software stack.
- The provider has released only this one version. There is no BF16 or FP8 version. This means there is no option to wait for a higher-precision release. If the hardware does not support FP4, the only options are dequantization or additional conversion, which significantly increase memory and time costs.
3. Capability Profile: How to Read Vendor-Reported Benchmarks
The provider’s comparison table is shown below. All results are self-reported.
| Benchmark | This model | Comparison model | Result |
|---|---|---|---|
| Terminal Agent tasks | 65.7% | 66.4% | Slightly lower |
| Long-horizon coding tasks | 51.7% | 46.3% | 5.4 percentage points higher |
| Terminal science tasks | 38.1% | 63.3% | 25.2 percentage points lower |
| Hard knowledge reasoning | 39.4% | 60.9% | 21.5 percentage points lower |
The provider said that the model outperformed the comparison model on 16 of 37 benchmarks. A third-party evaluator’s intelligence index gave it a score of 55, ranking it fourth globally.
Overall, its capability distribution is uneven:
- Coding and agent-related tasks are close to the first tier.
- Some long-horizon coding tasks show a clear lead.
- Science-related tasks and hard reasoning tasks lag well behind leading models.
More specifically:
- Terminal Agent tasks are slightly lower, by less than one percentage point, placing them in a similar range.
- SWE-style coding tasks are more than five percentage points higher, showing a clear advantage.
- Science-related and HLE-style reasoning tasks are more than 20 percentage points lower, revealing a clear weakness.
There are three points to keep in mind when reading this kind of comparison table:
- The data is self-reported. The publisher has an incentive to select benchmarks and prompt templates that favor the model. Horizontal comparison therefore has limited value, while vertical comparison of strengths and weaknesses is more useful.
- Counts such as “16 of 37” hide the size of the gaps. A one-percentage-point lead and a twenty-percentage-point lead have very different significance.
- It is best to re-test on your own evaluation set. Extract a small set of tasks with reference answers from real business scenarios, such as online ticket repair, SQL generation and tuning, or long-document question answering. Then combine accuracy, latency, and cost from the business side to make a decision.
4. License: Open Weights Are Not the Same as Open Source
This is one of the most important aspects of the release. The repository uses a custom license. The model card labels the license as “Other / custom license” rather than a standard open-source license such as MIT or Apache.
The license text is identical word-for-word to the previous generation, except for the name. Product names and contact information appearing in the license should be replaced with your own identifiers and updated contact channels according to the license requirements. For official commercial use, the original license text should take precedence.
The main grant is MIT-style, allowing free use, copying, modification, distribution, fine-tuning, and commercial use. However, two additional thresholds apply to large-scale commercial use:
- Enterprises that provide the model as a service, meaning MaaS, and whose cumulative revenue over 12 months exceeds $20 million, must sign a separate agreement with the model provider before commercial use.
- Commercial products with more than 100 million monthly active users also require separate authorization.
1. Why It Should Not Be Called “Open Source”
Open weights only mean that the weights can be downloaded. Whether the license satisfies the definition of open source is a separate question.
Open-source licenses generally require free use, free redistribution, permission to create derivative works, and no discrimination against particular use cases. Revenue and user-scale restrictions create additional obligations for large commercial entities, so “open weights” is a more accurate description than “open source.”
For most developers, the MIT-style base license already covers research, internal evaluation, fine-tuning, and small- or medium-scale commercial use. However, license review is required in the following situations:
- Providing an API service where revenue is near or above the $20 million threshold.
- A product with monthly active users near or above 100 million.
- Packaging model capabilities into a commercial product for sale.
- Redistribution, or redistribution after fine-tuning.
A practical recommendation is to add a step to the CI pipeline that automatically scans model license metadata. Weights marked as “Other / custom license” should not automatically inherit open-source component compliance conclusions. They should be routed to legal review.
2. Beyond the License: Missing Open-Source Companion Artifacts
“Open source” is also often understood to mean open training code, data, and evaluation scripts. In this release, only the weights are available. Training code and data have not been released.
If you want to continue pretraining on your own data, you need to build a complete training and evaluation system yourself. The cost is fundamentally different from what it would be with full companion support.
5. Local Deployment: Capacity Planning for an 8-GPU Starting Point
Running the model locally requires at least eight accelerators on a single node as a starting point. The lower bound is determined by the inference framework based on memory budget, but it can also be checked using the formula below.
1. Memory Budget Formula
The three main quantities that determine deployment scale are:
- Weights that must remain resident. During decoding, every generated token requires reading all currently activated experts. Any portion that is not resident must be fetched from main memory or disk.
- KV cache. This grows linearly with concurrency and context length.
- Runtime overhead. This includes activations, communication buffers, framework overhead, and reserved headroom.
The following code is illustrative. It explains the estimation method rather than the configuration of a specific product.
# Memory budget illustration: fill in values based on the actual model architectureWEIGHT_BYTES = 1.56e12 # Total weight bytesN_LAYERS = 93N_HEADS = 64 # Example valueHEAD_DIM = 128 # Example valueKV_BYTES_PER_ELEM = 1 # Bytes per element after KV quantizationdef kv_cache_bytes(ctx_len, batch=1):# Full-attention layers are estimated using full KV cache;# sparse-attention layers may be significantly lower.return 2 * N_LAYERS * N_HEADS * HEAD_DIM * KV_BYTES_PER_ELEM * ctx_len * batchdef min_gpu_memory(mem_per_gpu, ctx_len=32768, batch=8, reserve=0.15):usage = WEIGHT_BYTES + kv_cache_bytes(ctx_len, batch)usage *= 1 + reserve # Communication buffers, activations, and framework reservereturn (usage + mem_gpu - 1) // mem_gpu # Round up
With 141GB of memory per GPU, the 1.56TB weight files alone would require at least 12 GPUs. Increasing ctx_len, batch, or reserve would raise the requirement further.
This also helps explain the meaning of the “8-GPU starting point.” It is closer to an empirical recommendation than a strict mathematical minimum. When memory conditions allow, eight GPUs can be a reasonable starting point for single-node deployment. The final card count should be determined by measurements from the inference framework.
2. Why 8 GPUs: Clean Partitioning and Bandwidth
Beyond capacity, the number 8 also matters for parallel partitioning:
- 896 routed experts divided by 8 gives exactly 112 experts per GPU, with no remainder.
- Attention layers can be partitioned according to the number of attention heads.
- The 93 layers can also be assigned across an 8-way pipeline.
For topology, single-node deployment with eight GPUs is preferred so that inter-GPU communication uses high-bandwidth links. This minimizes the communication overhead of tensor parallelism and expert parallelism. Scaling across machines requires a high-speed network for tensor parallelism or expert parallelism, with a clear latency cost.
3. Placement Strategies: Trade-offs Among Three Options
| Option | Approach | Cost |
|---|---|---|
| Fully resident | Keep all weights in GPU memory | Requires the most GPUs and the highest cost |
| Expert offloading | Keep cold experts in host memory or SSD, keep hot experts resident | Cold experts must be fetched when activated; hit rate drops under high concurrency |
| On-demand loading | Load weights from disk using memory mapping | Longer memory access path; suitable for offline batch processing |
A common practical approach is hot/cold tiering: keep only the set of experts with high activation probability and large compute share resident in GPU memory, while placing the rest in host memory. The number of resident experts and cache capacity can be treated as adjustable parameters, creating a trade-off between memory usage and throughput.
Traffic characteristics also matter. Agent-style products often involve multi-turn dialogue and long contexts, making KV cache a much larger share than in ordinary chat. Multimodal inputs also concatenate visual tokens generated from image encoding into the context, further lengthening sequences. Capacity planning should be based on the worst case rather than only the average conversation length.
4. Quantization Side Effects
- The weights are 4-bit quantized, but the KV cache is not quantized by default. Based on the 1.56TB total weight size and 2.8T parameters, the average weight footprint is about 4.46 bits per parameter, including scaling factors and a small number of non-quantized normalization parameters. If the KV cache is stored in FP16, its memory footprint can become much heavier than the weights.
- If the hardware does not support FP4, a common approach is to dequantize before inference. Storage per parameter doubles to 8 bits, and GPU memory usage roughly doubles. The eight-GPU plan is basically no longer viable.
- The vision encoder is usually stored at a higher precision and does not follow the backbone quantization. It also requires separate memory reservation.
5. An Adjustable Deployment Configuration Example
Parameter names differ across inference frameworks. The following is a generic approach.
model:quantization: mxfp4 # 4-bit weights and 8-bit activations; requires hardware and inference stack supportparallel:tensor_parallel_size: 8 # Partition by attention headsexpert_parallel_size: 8 # 896 experts divided evenly into 112 per GPUpipeline_parallel_size: 1runtime:max_model_len: 262144 # Start with a measured value, then gradually approach 1Mkv_cache_dtype: fp8 # Reduce KV cache memory usageenable_chunked_prefill: true # Chunked prefill for long prompts to avoid latency spikesplacement:resident_experts: [...] # Frequently activated experts kept in GPU memory; determine from routing statisticsoffload_target: host_memory # Offload other experts to host memory
Before going live, the following commands can help assess feasibility. Exact parameters should follow the documentation of the framework in use.
# 1. Check hardware and GPU topology: confirm number of GPUs, per-GPU memory, and interconnectnvidia-smi topo -m# 2. Load only without inference to verify whether 8 GPUs can hold the weights and leave room for KV cachevllm serve <model_path> \--tensor-parallel-size 8 \--max-model-len 8192 \--gpu-memory-utilization 0.90# 3. Run a small-batch, long-context stress test and record TTFT, per-token latency, and throughputvllm bench serve --model <model_path> \--dataset-name random --random-input-len 32768 \--random-output-len 512 --max-concurrency 8
When interpreting stress-test results, focus on three metrics:
- Time to first token, TTFT: Mainly determined by prefill and especially important for long contexts.
- Time per output token, TPOT: Mainly determined by decoding and directly related to expert residency strategy.
- Total throughput: Reflects cost and capacity planning.
6. Deployment Path Choice: Local, Cloud, or API
Once the GPU count has been estimated, you still need to decide where to run the model. Each path has suitable conditions:
- Self-hosted: Suitable when data cannot leave the internal network, when deep customization is required, such as continued pretraining or modifying inference logic, when idle compute is already available, or when usage is long-term and high-frequency. The main constraints are high upfront investment and heavy operations. If the hardware does not support FP4, the local route is largely blocked.
- Public cloud on demand: General-purpose GPU cloud instances, elastic container services, and similar options can provide multi-GPU instances. This is suitable for stress testing, short-term peaks, and validating before committing to larger investment. Make sure the selected instance uses accelerators that support FP4/FP8 and has compatible images and inference frameworks.
- Continue using the API: Suitable for teams that only want to use the capability and do not want to manage deployment. Pricing is based on usage, and it avoids capacity planning and operations. The trade-offs are that data must leave the internal network, long-term high-frequency use may cost more than self-hosting, and inference logic cannot be customized.
Regardless of the path, data is best placed on object storage or a shared file system, multi-GPU inference services should be orchestrated through a container platform, and monitoring and alerting should track GPU memory, throughput, latency, and error rates. If providing a public generative service, content safety, generated-content disclosure, and other applicable compliance requirements should also be considered.
7. Conclusion: Opportunity and Barriers Coexist
The significance of this weight release is clear. A 2.8T-parameter open-weight model further raises the ceiling for downloadable models. Its coding and agent capabilities are strengths, and the custom license leaves room for small- and medium-scale commercial use.
The barriers are equally clear:
- The eight-GPU hardware requirement blocks many individual developers and small teams.
- The quantization format ties deployment to hardware capability.
- The custom license requires legal review.
- Weaknesses in science and hard reasoning need to be evaluated during model selection.
A pragmatic approach has three steps: first, use the API to validate whether the model matches your business on your own evaluation set; second, estimate resources using the capacity formula; finally, decide between local deployment, cloud deployment, or continuing with the API based on the license terms.
