2.8T Sparse MoE and 65B Dense Open-Weight Models: Architecture, Deployment, and Selection Guide
Two new open-weight releases take different scaling paths: a 2.8T sparse mixture-of-experts model and a 65B dense model. This article summarizes their vendor-disclosed architectures, benchmark claims, memory and communication requirements, quantization trade-offs, and suitable deployment scenarios, and provides a comparison table and practical guidance for developers evaluating open models. Vendor-disclosed figures should be verified against official releases; the article does not independently reproduce them.
1. What Was Released This Week
The open-weight model community recently saw two notable new models. The first is a sparse Mixture-of-Experts (MoE) model with 2.8 trillion (2.8T) total parameters; the publisher says its parameter scale exceeds that of previously public open-weight models. The second is a 65 billion (65B) dense model trained from scratch by a chip design company and released under a permissive open-source license.
The two models take different routes. The former uses sparse activation to control compute cost at a very large parameter count; the latter uses a mid-sized dense architecture to lower the deployment barrier. Both point to the same question: can model capability continue to improve while deployment costs come down?
All data cited in this article comes from materials published by the vendors. We have not independently reproduced the results. For specific scores and parameters, refer to the official release pages and technical reports.
2. Sparse Expert Model: Many Parameters, Only a Subset Used per Token
2.1 Architecture Highlights
A dense model invokes all parameters at every inference step. An MoE model works differently. Through a router network, it selects the most relevant expert sub-networks for each input token and computes only the outputs of those experts.
According to the configuration provided by the publisher, the model’s main parameters are as follows:
- Total parameters: 2.8 trillion (2.8T)
- Activated parameters per token: about 104 billion (104B)
- Context window: the publisher claims native support for 1 million (1M) tokens
- Modality: the publisher claims native multimodality
This means that about 3.7% of total parameters are activated per token (104B divided by 2.8T). The compute per inference pass is therefore close to that of a hundred-billion-parameter dense model, while total model capacity is much larger. The benefit is that the model can store more knowledge for the same compute. The trade-off is that all expert parameters still need to reside in VRAM or memory, so storage and communication burdens do not shrink accordingly.
2.2 How to Interpret Performance Claims
The publisher says that across 37 benchmarks covering mathematical reasoning, code generation, multilingual understanding, and multimodal perception, the model outperforms its selected leading closed-source commercial model on 16 of them. The release materials do not list the benchmark names, test conditions, or the versions of the comparison models, so this figure should currently be treated as a vendor claim.
Benchmarks usually measure objective tasks with standard answers. On open-ended tasks such as complex instruction following, long-form writing, and fine-grained image understanding, the gap between benchmark scores and real-world experience is often larger. A safer judgment is that the model is competitive on objective task types, while subjective experience still needs to be tested with your own workloads.
2.3 Practical Deployment Barriers
Loading and running a 2.8T-parameter model places high demands on hardware. Several points need to be considered in advance:
- VRAM usage: at BF16 precision (2 bytes per parameter), weights alone require about 5.6 TB of VRAM. This must rely on multi-node, multi-GPU deployment, usually combined with expert parallelism.
- Communication overhead: an MoE model needs to distribute and aggregate tokens across devices at every layer, which is All-to-All communication. When interconnect bandwidth is insufficient, throughput drops noticeably. The share of communication in total latency depends on network topology and parallelism strategy. The release materials do not provide verifiable measurements, so real projects should test on the target cluster.
- KV Cache for long contexts: when context is extended to the million-token level, KV Cache can quickly fill VRAM. Common approaches include PagedAttention and sliding-window attention, but all require tuning for the specific model.
For most small and mid-sized teams, building such a cluster is not realistic. More common approaches are to use elastic compute or model APIs from cloud providers, or to use the model as a teacher and distill its capabilities into smaller models.
3. 65B Dense Model: A Mid-Sized Option for Local Deployment
3.1 Basic Information
This model was trained from scratch by a chip design company. The main information is as follows (all disclosed by the publisher):
- License: Apache 2.0, allowing commercial use and modification
- Pretraining data: 12 trillion (12T) tokens
- Context length: the publisher claims support for 1 million (1M) tokens
- Hardware: the publisher says it can run with 4-bit quantization on a single GPU with 128GB of VRAM
Note that the release materials do not provide the model name, weight download address, or the full license text. Before use, check the publisher’s official channels to confirm the license terms and weight version.
3.2 VRAM Estimate After Quantization
Multiplying parameter count by bytes per parameter gives an estimate of the VRAM required for weights:
- BF16 (16-bit): about 130GB
- 8-bit quantization: about 65GB, plus runtime overhead, around 68GB
- 4-bit quantization: about 32.5GB, plus runtime overhead, around 36GB
These are estimates for weights only. In actual deployment, KV Cache, activations, and framework overhead all consume additional VRAM, and longer contexts consume more. Therefore, a claimed “single-GPU runnable” setup usually refers to scenarios with shorter context or lower concurrency.
Quantization also introduces precision loss. 8-bit usually causes smaller loss; 4-bit may show more noticeable degradation on precision-sensitive tasks such as mathematical reasoning and code. Throughput depends on the inference framework and hardware, and quantization does not necessarily bring a proportional speedup. It is advisable to compare different quantization levels using your own evaluation set.
3.3 Suitable Scenarios
This type of mid-sized dense model is relatively suitable for the following scenarios:
- Enterprises whose data should not leave their premises, using private deployment on internal servers;
- Teams that need to fine-tune locally, for example training risk-control or compliance-related models on internal documents;
- Offline applications on edge devices or workstations, such as the inference module of a local voice assistant.
The Apache 2.0 license allows modification and commercial use, but you should still check whether training data and downstream dependencies carry other restrictions.
4. Comparing the Two Routes
| Dimension | 2.8T sparse MoE model | 65B dense model |
|---|---|---|
| Total parameters | 2.8 trillion | 65 billion |
| Activated per inference | About 104 billion | All 65 billion |
| Weight VRAM (BF16) | About 5.6TB | About 130GB |
| Main bottleneck | Multi-node communication, cluster scale | VRAM capacity, quantization precision |
| Suitable for | Large cloud providers, infrastructure teams | Small and mid-sized businesses, edge and private deployment scenarios |
Figures in the table come from vendor disclosures or direct conversion from parameter counts; the conversion method is described above.
5. Developer Selection Advice
- Define constraints first. If your business is not latency-sensitive and you are pursuing the highest capability ceiling, you can watch very large MoE models, but you need to accept their cluster costs. If data is sensitive, local execution is required, or budget is limited, a mid-sized dense model is easier to deploy.
- Evaluate with your own tasks. Public benchmarks can only provide a reference. It is advisable to collect dozens to hundreds of real samples covering your main business and compare candidate models and quantization levels.
- Use inference and quantization tools effectively. Quantization methods such as GPTQ and AWQ, and inference engines such as vLLM and TensorRT-LLM, can improve throughput on the same hardware. Before selection, verify compatibility in a small-scale environment.
- Evaluate the cost of cloud-hosted services. For individual developers or early-stage teams, calling model services on demand is often more cost-effective than purchasing high-end GPUs. When choosing a provider, compare unit price, concurrency limits, data processing terms, and model version update policies.
6. Conclusion
This week’s two models show two different sides of open-weight models: one demonstrates the upper limit of scale, while the other is closer to everyday deployment needs. The former proves that the ultra-large sparse model route is technically feasible; the latter determines how many teams can actually use open models.
For developers, the next things worth watching are inference cost, evaluation reliability, and access to high-quality training data. The parameter race may slow down, and how to use models stably and at low cost will be a more practical topic.
