A 2.8T-Parameter MoE Open-Weights Model: Why the 1.56TB Download Still Needs 8 GPUs
This article takes a first practical look at a newly released open-weights Mixture-of-Experts model with 2.8T total parameters and 104B activated per token. It breaks down the architecture — 896 routed experts, 2 shared experts, 93 layers, sparse attention in 24 of them, and a native vision encoder — then works through the numbers that decide whether you can run it: why MXFP4 weights land at 1.56TB instead of the theoretical 1.4TB, how much VRAM 8/16/32 GPUs each need, why KV cache at 1M context is a separate problem from compute sparsity, and how expert parallelism, tensor parallelism, and pipeline parallelism fit together. It also covers how to read the vendor-reported benchmarks, and why "open weights" is not the same as open source here: the license is MIT-style in body but carries two revenue- and user-scale triggers that decide whether a separate commercial agreement is required.
The Weights Are Live
On October 24, a Chinese model team launched a Mixture-of-Experts model with 2.8T total parameters, first as an API and a web app, and promised to release the full weights by November 4. On November 4 the weights appeared in a major open model community, downloadable publicly without an application or approval step.
The repository was created on November 2, was still closed on the morning of November 4, and opened that evening. It contains 112 files, of which 96 are weight shards totaling roughly 1.56TB.
This is the largest open-weights model released to date, well ahead of the previous generation of open models in size. The API performance and launch results have already been reported elsewhere. This article looks at three practical questions that only become relevant once the weights are actually on disk:
- What is inside: how the parameters are distributed, and what changed relative to the previous generation.
- How the license works: the difference between open weights and open source, and where the commercial boundary sits.
- Can ordinary people run it: the VRAM math for a 1.56TB checkpoint, and the engineering difficulty of local deployment.
Inside the Model: 2.8T Total Parameters, 104B Activated
Basic Specifications
According to the model card:
- It is a native multimodal agent model, continued from training on the previous generation.
- 2.8T total parameters, with only 104B activated per token.
- MoE structure with 896 routed experts, 16 selected per token, plus 2 shared experts.
- 93 layers, with a 1M-token context length.
- A self-developed vision encoder providing image understanding.
MoE: Separating Parameters from Compute
Understanding this model means understanding Mixture-of-Experts. In a dense model, total parameter count and the compute used per token scale roughly together. MoE splits each feed-forward layer into many parallel “experts”, and a gating network picks a few experts to participate in computing each token. For this model:
- Each token routes to 16 experts, plus 2 permanently resident shared experts — 18 out of 898 candidate experts (896 + 2).
- The activation ratio is about 16/898, under 2%.
- Even counting attention parameters and shared experts, the activated portion is only about 3.7% of the total (104B / 2.8T).
Shared experts typically carry cross-domain general knowledge, while routed experts each cover a narrower domain. This design lets the model grow in parameter count without a proportional increase in per-token compute, which is the basis for scaling it to 2.8T.
Versus the Previous Generation: One Structural Change
Relative to the previous generation, only one structural change was made: 24 layers of full attention were replaced with sparse attention (DSA), to save compute on long contexts.
Standard attention compute grows quadratically with context length. At a 1M-token context, full attention becomes the throughput bottleneck. Sparse attention lets each token attend to a carefully selected small set of past tokens rather than scanning the entire context.
One point is easy to confuse: MoE sparsity is conditional computation, used to save compute. Sparse attention is algorithmic sparsity, used to save both compute and memory bandwidth. The two solve different problems and should be understood separately.
Native Multimodality: Vision and Language in One Model
The model uses a self-developed vision encoder, which makes it a “native multimodal” design: visual understanding and language generation happen in the same backbone, rather than bolting a separate recognition model onto a language model. Models like this have a natural advantage in mixed image-and-text understanding, UI screenshot parsing, and document processing. That matters especially for agent applications, which often need to “read” an interface or document before deciding the next action.
Running It Locally Takes 8 GPUs: The VRAM Math and Parallelism Strategy
Why the Weights Are 1.56TB
Checkpoint size is determined by total parameters and precision:
2.8 × 10^12 parameters × 0.5 bytes/parameter (4 bit) = 1.4 TB
The actual download is about 1.56TB, roughly 11% more than the pure 4-bit theoretical value. There are three common reasons:
- Quantization blocks require extra metadata such as scaling factors.
- Some modules (embedding layers, shared experts) may be kept at higher precision.
- Storage is aligned to fixed block sizes.
Working backwards, that puts the average at about 4.46 bits per parameter (1.56TB ÷ 2.8T).
The Precision Scheme: MXFP4 Weights, MXFP8 Activations
The team published only one version:
- Weights use MXFP4 (Microscaling FP4), averaging 4 bits per parameter with small-block-granularity scaling factors.
- Activations use MXFP8.
- This version is the product of quantization-aware training (QAT): quantization error was incorporated into the optimization during training, rather than applied as post-training quantization (PTQ) afterwards.
No BF16 or FP8 version was released, and no original high-precision weights were published. One thing deserves particular attention: 4-bit weights do not mean “all computation runs in 4-bit”.
- Matrix multiplications are typically dequantized or run in mixed precision, then accumulated at higher precision.
- 4-bit mainly reduces VRAM footprint and memory bandwidth pressure.
- In memory-bound scenarios such as autoregressive decoding, weight-read speed is often the main bottleneck, so low-bit quantization translates directly into higher decode throughput.
- It also means you need hardware with high-bandwidth memory that supports the format, plus a mature inference framework. Otherwise the VRAM savings never turn into speed.
Estimating the VRAM Requirement
Total VRAM demand is made up of at least four parts:
total VRAM ≈ weights + KV cache + activations and temporary buffers + runtime overhead
The script below estimates the per-GPU weight shard at different parallelism levels:
# Rough VRAM estimate, in GB (1 GB = 10^9 bytes)params = 2.8e12 # total parametersbits = 4.46 # average bits including metadataweight_gb = params * bits / 8 / 1e9 # ≈ 1561 GB, matching the 1.56 TB downloaddef kv_gb(layers, kv_heads, head_dim, dtype_bytes, tokens):# factor of 2 for K and V; layers/kv_heads/head_dim are assumed valuesreturn 2 * layers * kv_heads * head_dim * dtype_bytes * tokens / 1e9# Assumed parameters: 93 layers, 4 KV heads, head_dim 128, 1 byte for the KV cachekv = kv_gb(layers=93, kv_heads=4, head_dim=128, dtype_bytes=1,tokens=32_768) # ≈ 3.0 GBfor gpus in (8, 16, 32):per_gpu = weight_gb / gpus + kv / gpusprint(f"{gpus:>2} GPUs: weights ≈ {weight_gb/gpus:6.1f} GB per GPU, "f"total ≈ {per_gpu:6.1f} GB per GPU (incl. 32K context KV)")
Output:
8 GPUs: weights ≈ 195.1 GB per GPU, total ≈ 195.5 GB per GPU (incl. 32K context KV)16 GPUs: weights ≈ 97.6 GB per GPU, total ≈ 97.8 GB per GPU (incl. 32K context KV)32 GPUs: weights ≈ 48.8 GB per GPU, total ≈ 49.0 GB per GPU (incl. 32K context KV)
Three conclusions follow:
- The “8 GPUs” premise is high per-card memory. Eight cards work only if each card has at least around 192GB (8 × 192GB = 1536GB), or if you spread the weights across two 8-card servers (16 cards total). With common 80GB-class accelerators, total VRAM is only 640GB, so the remaining weights must be offloaded to host memory, NVMe, and other media. That adds substantial latency and generally suits offline batch processing only, not online serving.
- Compressing the weights further is not advisable. The QAT release is already 4-bit; going to 3-bit or even 2-bit post-training quantization stacks and amplifies error. A safer route is a smaller model from the same family, or a distilled model.
- The 104B activation figure is a cluster-level number. It describes the portion shared across all cards in the cluster. At 4-bit that is roughly 52GB, spread across multiple GPUs and shared across the batch — not one copy per card.
Parallelism: Expert Parallelism Plus Tensor Parallelism
The core difficulty of deployment is not fitting the model in memory, but computing efficiently once it is loaded. The typical MoE approach combines expert parallelism with tensor parallelism:
# Illustrative parallel split (pseudocode)world_size = 8 # assume 8 GPUsnum_experts = 896 # routed expertsexperts_per_rank = num_experts // world_size # 112 experts per GPU# Each rank loads only 1/world_size of the expert weightslocal_experts = load_shard(shard_id=rank,range=(rank * experts_per_rank,(rank + 1) * experts_per_rank))# Dense parts (attention, shared experts, embeddings) are split along the tensor dimensionattention = shard_tensor_model_parallel(rank, world_size)def forward(x):h = attention(x)# All-to-All: send tokens to the GPU holding their target expertsdispatched = all_to_all(h, routing=gate(h))# Each GPU computes only its local expertsy = moe(dispatched, local_experts)# All-to-All: send results back to the originating GPUreturn all_to_all(y, reverse=True)
- Expert parallelism (EP): distributes the 896 experts across GPUs, each holding a subset. Its strength is flexibility — adding GPUs lets you hold more parameters. The cost is two All-to-All rounds per layer, which puts heavy demands on inter-GPU interconnect bandwidth.
- Tensor parallelism (TP): splits a single matrix multiplication into pieces spread across GPUs. Its communication pattern is regular, but it requires frequent synchronization, so it is generally used within a single machine.
- Pipeline parallelism (PP): when splitting across machines, if cross-machine All-to-All latency is unacceptable, you can split along layers so that machines only pass inter-layer activations. The cost is pipeline bubbles.
The generally accepted engineering rules:
- Use tensor parallelism over fast intra-machine interconnects.
- Use expert parallelism or pipeline parallelism across machines.
- On a high-speed RDMA network, All-to-All latency mostly comes from the gateway and from uneven routing load. If a particular expert becomes hot, its GPU becomes a bottleneck for the whole cluster, so gating load-balancing metrics are worth watching.
The 1M Context: More Than a Memory Problem
KV cache. The longer the context, the larger the cache. Using the assumptions above, a 32K context takes about 3GB. At 2-byte KV precision, a 1M context takes roughly 190GB — about 24GB per GPU across 8 cards. Sparse attention significantly reduces long-context compute, but the KV of the selected historical tokens still has to stay resident in VRAM, so “compute sparsity” does not imply “memory sparsity”.
Attention overhead. As soon as request length exceeds the prefill chunk size, long-context requests must be prefilled in chunks. A single very long request occupies compute resources for a long time and degrades time-to-first-token for other requests.
Practical advice. Very few business workloads actually need a 1M context. Rather than chasing maximum length, do three things well:
- Retrieval augmentation: put only the relevant passages into the context.
- Context compression: summarize, trim conversation history.
- Prefix caching: reuse the KV of system prompts and fixed documents to cut repeated compute.
Reading the Vendor-Reported Benchmarks
All the comparative numbers published officially are vendor-reported:
| Benchmark | This model | Closed-source model A | Closed-source model B |
|---|---|---|---|
| Terminal task benchmark | 65.7% | 66.4% | 58.2% |
| Long-horizon software engineering | 51.7% | 46.3% | — |
| Terminal research benchmark | 38.1% | 63.3% | — |
| Hard knowledge reasoning benchmark | 39.4% | 60.9% | — |
A third-party independent evaluation organization also gave it a composite intelligence score of 55, ranking it fourth globally.
Reading these numbers together suggests the following:
- Coding and agent capability are first-tier, but the model trades wins with the strongest closed-source models rather than leading across the board.
- Scientific reasoning and some hard reasoning tasks clearly lag, consistent with the model card’s claim of leading on 16 of 37 benchmarks.
- Self-reported scores need discounting, for several reasons: the vendor controls the test prompts and scoring scripts; training data may overlap with public test sets (data contamination); and different vendors use different agent scaffolds, tool permissions, and retry policies, all of which directly affect scores.
The more reliable approach is to build your own evaluation set:
- Sample 100 to 300 tasks from real business traffic.
- Standardize tool permissions, system prompts, and scoring scripts.
- Record both task pass rate and token consumption, since the cost of agent tasks is driven mainly by multi-turn tool calls.
- Pin versions and random seeds for a clean A/B comparison.
The License: Open Weights Is Not Open Source
This is the part of the release that most deserves separate scrutiny.
Not a Standard Open Source License
The repository ships a team-authored custom license, classified as “other” in the model community rather than a widely recognized permissive license like MIT or Apache. The license text is identical word for word to the previous generation’s license, with only the name changed. That indicates it is the team’s consistent strategy for very large models, not an ad-hoc arrangement for this release.
The Body Is MIT-Style
The main text is an MIT-style grant:
- It grants the rights to use, copy, modify, distribute, sublicense, and sell copies free of charge.
- It requires retaining the copyright and license notices.
- The software is provided “as is”, with no warranty.
Under this grant, the following typically need no additional permission:
- Internal enterprise use.
- Internal deployment of a fine-tuned derivative.
- Small- and mid-scale commercialization.
Two Additional Restrictions: Scale-Triggered Commercial Obligations
On top of the MIT-style grant, two restrictions apply. Both target only large-scale external business:
- Clause 1 (model-as-a-service): companies offering the model as a service externally, with cumulative revenue above $20M over the trailing 12 months, must sign a separate agreement before commercial use.
- Clause 2 (user scale): commercial products or services with more than 100 million monthly active users likewise require a separate agreement (dimensions per the official text).
There is a difference between “free for commercial use” and “unrestricted commercial use”. To decide whether a separate agreement is needed:
Are you offering the model externally (SaaS, API, plugins, etc.)?├─ No → internal use, ready to go└─ Yes → trailing 12-month cumulative revenue > $20M?├─ No → monthly active users > 100M?│ ├─ No → ready to go│ └─ Yes → separate agreement required└─ Yes → separate agreement required
Two more details are easy to misread:
- Do modifications inherit the restrictions? The additional clauses govern behavior — offering the model externally, and the scale of a commercial product. Even after fine-tuning, if you still offer the model as a service externally and cross the revenue threshold, you fall under the same constraints. “It has been modified” is not a way around them.
- Threshold definitions. The documentation does not clearly define how “cumulative revenue” is calculated (whether costs are deducted, which legal entity it is measured for, whether affiliates are consolidated), and it is unclear whether companion documents such as a model card or usage policy are issued alongside. Companies should confirm this in writing before signing and route it through legal review.
A Deployment Checklist for Enterprises
| Check | What to do |
|---|---|
| License version | Confirm you are using the current repository version, and keep a snapshot |
| Commercial thresholds | Estimate trailing 12-month revenue and monthly active users |
| Redistribution | Include the license and copyright notice when distributing weights externally |
| Compliance review | Generative services carry content-safety and filing obligations |
| Trademark use | Do not promote under the model’s name without authorization |
| Supporting documents | Confirm the model card, usage policy, and safety assessment notes are complete |
Conclusions and Recommendations
Who should consider local deployment:
- Enterprises with large pools of high-memory accelerators that need data to stay on-premises.
- Organizations with strong engineering teams that can absorb the cost of parallelism tuning and operations.
- Workloads that are mostly offline batch inference and insensitive to latency.
Who is better off calling a hosted API:
- Teams whose business volume has not crossed the license thresholds.
- Teams that need to ship quickly without building their own cluster.
- Teams whose main need is coding and agent capability rather than scientific reasoning.
General advice:
- Read “8 GPUs” as a high-memory single node or a two-node start, not 8 commodity 80GB cards.
- Do not requantize the QAT weights. When VRAM is short, adjust the parallelism strategy or switch to a smaller model first.
- Use long context only where needed, with retrieval and caching in place.
- Validate vendor-reported numbers against your own evaluation set.
- Complete license and compliance review before commercial use.
Object storage and container orchestration from cloud providers can be used to distribute the weights and run the inference cluster, with monitoring and alerting tracking throughput and memory headroom. For most teams, the more pragmatic path is to validate business value through a hosted model service first, then decide whether to self-host based on scale and compliance requirements.
The landing of very large open-weights models marks another step up in parameter scale for the open ecosystem. But the gap between “open weights” and “open source”, and between “can be downloaded” and “can be run efficiently”, is still real. Reading the architecture, doing the VRAM math, and reading the license carefully is where working with models like this actually begins.
