How to Choose and Locally Deploy a 2.8T Open-Weight Model
A flagship model with 2.8T total parameters and a roughly 1.56TB weight package has been released as open weights, with a deployment floor of "8 GPUs minimum." This article breaks down the model's MoE architecture, sparse attention, and native multimodal design, then walks through the three hurdles that sit between downloadable weights and a working service: fitting the model into VRAM, using its low-precision MXFP4 weights well, and clearing its custom license. It explains what "8 GPUs minimum" means in practice—VRAM capacity, KV cache pressure at million-token contexts, and parallel partitioning—and closes with scenario-based selection advice covering self-hosting, cloud APIs, and smaller open models.
1. Introduction: Releasing Weights Is Only the Starting Point
On November 4, a Chinese AI vendor released the full weights of a flagship model as scheduled. The model has 2.8T total parameters and a weight package of about 1.56TB. It is billed as “the strongest open model currently available for download,” with a deployment floor of “8 GPUs minimum.”
Many teams treat “the weights are downloadable” as equivalent to “we can deploy it.” In reality, going from weights on disk to a service that stably serves external traffic still involves at least three hurdles:
- Can it even fit? Whether VRAM, bandwidth, and parallel partitioning line up;
- Can you use it well? The vendor ships only low-precision weights; does the inference engine support them?
- Can you use it commercially? What restrictions does the custom license carry?
This article addresses these three questions and ends with scenario-based selection advice.
2. What Makes Up the 2.8T: A Three-Layer Breakdown
The release timeline is clear:
- Late October: The model launched first via a cloud API and the web, with the vendor promising full weights within a month.
- Early November: Weight files were uploaded to an open-model repository but kept gated.
- Evening of November 4: The repository opened, and the files became downloadable without application. There are 112 files in total—96 weight shards, with the rest being configs, tokenizer, inference templates, and other supporting files.
2.1 Overall Specs
| Item | Value |
|---|---|
| Total parameters | 2.8T (trillion) |
| Activated parameters per forward pass | ~104B (~3.7% of the total) |
| Architecture | Mixture of Experts (MoE), 896 routed experts + 2 shared experts, 16 routed experts selected per token |
| Layers | 93, of which 24 use sparse attention and the rest use full attention |
| Context length | 1M tokens |
| Modality | Native multimodal, with a built-in vision encoder that can understand images directly |
| Weight precision | Weights MXFP4, activations MXFP8 (produced via quantization-aware training) |
| Weight size | ~1.56TB (≈1.42TiB) |
| License | Custom license (repository tagged license:other) |
2.2 MoE: Trading Sparse Activation for Capacity
The core idea of MoE is to split the feed-forward layers into a large number of “expert” subnetworks, with a router selecting a few experts to compute for each token. This brings two benefits:
- Decoupling capacity from compute: Parameter count determines how much knowledge the model can “store,” while activated parameters determine how much compute each token costs.
- Decoder-friendly: During autoregressive decoding, only one token is generated at a time, so compute is usually not the bottleneck. The bottleneck is that every generated token requires reading the activated parameters’ weights from VRAM. With an activation ratio of only 3.7%, the amount of data read is compressed to below the same order of magnitude as the total parameters.
The 2 shared experts are always active for all tokens, acting as a carrier of “general knowledge,” while the 896 routed experts handle domain and language specialization. This is a common design for large-parameter MoEs today, aimed at avoiding the loss of general capability that fully sparse routing would cause.
Estimate (for reference only)
Let E be the parameter count per expert (including the gate, up-projection, and down-projection matrices). The 896 × 93 routed experts contribute about 98% of the total parameters, giving E ≈ 34.4M.
Substituting into the activation side: 18 experts × 34.4M × 93 layers ≈ 57.6B, and the ~46.4B gap to the 104B activated parameters comes from dense parts such as attention and embeddings.
Actual values will vary with vocabulary size, shared-expert size, and other factors.
2.3 Sparse Attention: The Only Structural Change
Compared with the previous generation, only one thing changed structurally: 24 of the layers that previously used full attention now use sparse attention. The basic idea is:
- A lightweight “indexer” scores the historical tokens;
- Each query token attends only to the top-k highest-scoring tokens.
In million-token contexts, attention compute grows quadratically with sequence length. Sparsification reduces this cost to near-linear, making prefill and generation more tractable at long contexts.
Two points, however, need clarification:
- It saves compute, not capacity: the KV cache still has to cover the entire context, so VRAM pressure does not disappear (see Section 5);
- It depends on specific kernels: inference frameworks without support for the corresponding kernels either will not run or fall back to approximate full attention, which negates the long-context advantage entirely.
The model still follows a “native multimodal + vision encoder” approach during training, without an architecture that fuses visual information directly.
2.4 Version Cadence: Stable Structure, Growing Capacity
Models like this typically iterate on a rhythm of “basically stable structure, growing parameter count per generation”: once the routing mechanism and attention layout are validated, the next generation is more likely to improve capability through scaling and data upgrades rather than major architectural changes.
Notably, this model is the result of “continued training” on top of the previous generation rather than being pre-trained from scratch. This approach is increasingly common in the industry: first obtain a base, then strengthen target capabilities such as coding and tool use through larger-scale data and training. For users, this means the model’s capability profile often carries the “base color” of the previous generation, so when selecting a model it is better to look at the capability distribution than at a single benchmark score.
3. Reading the Benchmarks: Vendor Claims vs. Independent Evaluation
First, look at the vendor’s self-reported comparison data. This kind of data is only a reference and has not yet been independently reproduced:
| Capability dimension | This model | Comparison A | Comparison B |
|---|---|---|---|
| Terminal tasks (overall) | 65.7% | 66.4% | 58.2% |
| Long-horizon software engineering | 51.7% | 46.3% | — |
| Terminal research tasks | 38.1% | 63.3% | — |
| Hard general reasoning | 39.4% | 60.9% | — |
(Both comparison objects are current leading closed-source models.)
The vendor claims it surpasses Comparison A on 16 of 37 benchmarks. Reading the table row by row, the conclusion is clear: a mixed record.
- Strengths: terminal-operation tasks, long-horizon software engineering, and agentic abilities such as tool use and multi-step execution.
- Weaknesses: research-oriented tasks and hard reasoning, where there is a clear gap to leading closed-source models.
An independent third-party evaluation’s composite intelligence index ranks it fourth globally, which can be read as a side confirmation of the above.
From this we can extract a selection principle: open-weight models and closed-source flagships are not a wholesale replacement for one another, but more likely a division of labor by scenario. When building coding assistants or ops agents, this model can serve as the primary choice; for scientific computing and complex mathematical reasoning, leading closed-source models may still be the safer option. Treating this comparison table as a scenario map is more valuable than staring at the overall ranking.
4. License Breakdown: Open Weights ≠ Open Source
This is the most easily misread point in the release. The model repository carries the vendor’s custom license, with the license type marked as other in the metadata rather than a standard open-source license such as MIT or Apache. This license is word-for-word identical to the previous generation’s, with only the name swapped.
4.1 License Terms Breakdown
The main body adopts MIT-style permissive terms: free to use, copy, modify, distribute, and fine-tune, with commercial use allowed. On top of this, two restrictions are added for large-scale external business:
- Revenue-based limit: Companies whose primary business is “providing the model as a service externally” with cumulative revenue over $20M over 12 months must sign a separate agreement with the copyright holder before commercial use.
- User-scale limit: Once a commercial product’s monthly active users reach a certain scale, separate authorization is also required (see the license text for the exact threshold).
4.2 Compliance Self-Check
Most enterprise scenarios will not trigger these two restrictions. Legal and business teams are advised to self-check against the following:
| Check item | What to confirm |
|---|---|
| Business type | Whether providing model services externally is the main revenue source |
| Revenue scale | Whether cumulative revenue over the last 12 months exceeds $20M |
| User scale | Whether the product’s MAU reaches the license threshold |
| Distribution obligations | When redistributing the model or derivative models fine-tuned from it, whether the license text must be included and copyright notices retained |
| Output constraints | Whether the license attaches usage restrictions on model outputs |
| Change tracking | Whether existing versions’ terms continue to apply if later versions change the license |
If a restriction is triggered, complete the licensing discussion before launch to avoid scrambling for process after the business is already running.
4.3 The Boundary Between Open Weights and Open Source
Two concepts need to be distinguished:
- Open source: Usually means the code, weights, training methods, and even the data composition are all available, accompanied by a standard open-source license recognized by an authoritative organization.
- Open weights: Only the parameter files needed for inference are released; the training code, data recipes, and training pipelines are typically not disclosed, and the license is often custom text.
To judge whether a model can be “freely used,” do not rely on which repository it sits in or on the marketing language—read the license text line by line. The terms may look permissive but can still contain details such as patent grants, attribution requirements, and additional usage policies.
5. Local Deployment: What “8 GPUs Minimum” Means
5.1 VRAM: First Clear the “Can It Fit” Hurdle
The first hurdle for local deployment is VRAM. A conservative estimate based on the vendor’s data:
- VRAM capacity: 1.56TB ÷ 8 GPUs ≈ 195GB per GPU. If the weights are stored in BF16, the size is about 5.1TiB, which requires at least ~32 GPUs to fit. So low-precision weights are the key to bringing the floor down from “a small server room” to “a single server.”
- VRAM bandwidth: With 4-bit weights, generating each token requires reading about 52GB from VRAM (104B × 0.5 bytes), concentrated on the activated experts. If the machine’s total 8-GPU bandwidth is on the order of 25–30TB/s (~3–4TB/s per GPU), the decode speed ceiling for a single request is about 400–600 tokens/second, not counting communication and kernel overhead.
- Parallel partitioning: 896 experts divide evenly by 8 (112 per GPU), so expert parallelism is natural. 93 layers do not divide by 8 (93 = 11 × 8 + 5), which requires uneven partitioning or a hybrid “replicate attention, shard experts” approach:
# Illustrative code: uneven pipeline partitioning of 93 layers across 8 GPUsworld = 8layers = 93q, r = divmod(layers, world) # q=11, r=5parts = [q + 1] * r + [q] * (world - q) # first 5 GPUs get 12 layers each, last 3 get 11# Rough VRAM budget (after capacity unit conversion)weight_total_gib = 1.56e12 / 2**30 # ≈1453GiBper_gpu = weight_total_gib / world # ≈182GiB/GPUassert per_gpu < gpu_memory_gib # leave room for KV cache and activations
With enough GPUs, hybrid partitioning (replicating the attention part across GPUs and partitioning experts by GPU) is often friendlier than pure expert parallelism: attention compute no longer crosses GPUs, and only routed tokens need to be dispatched and gathered across GPUs.
5.2 KV Cache: The Real Long-Context Bottleneck
A common misconception is “sparse attention saves compute, so million-token contexts are easy.” Consider the estimate:
Single-sequence KV cache bytes ≈ 2 × layers × per-layer KV width × context length × bytes per elementWith 93 layers, KV stored in FP8 (1 byte), 1M context:= 2 × 93 × 1M × per-layer KV width≈ per-layer KV width × 174 MiB
Even with a per-layer KV width of only 1024, a single full-length sequence takes about 174GiB of VRAM—almost two GPUs’ worth. Larger actual configurations multiply this further.
Under a budget of ~182GiB per GPU, a single GPU with the weights loaded typically has only tens of GiB left. Long-context serving therefore basically relies on three measures:
- Storing KV in low precision;
- Paged KV management to reduce fragmentation and waste;
- Prefix caching to reuse repeated prefixes such as system prompts and code repositories.
To support multiple million-token context requests simultaneously, the KV cache must be offloaded to memory or fast storage, or the node count increased.
5.3 Performance and Cost: Don’t Just Look at Peaks
| Dimension | Reality |
|---|---|
| Interaction latency | Thanks to “large total parameters, small activated parameters,” single-request decode speed is top-tier among open models |
| Concurrent throughput | Limited per-GPU headroom; high concurrency requires horizontal scaling, and per-GPU economics are mediocre |
| Prefill time | First-pass processing of long documents grows with context; sparse attention and cache reuse are the main optimizations |
| Startup time | Loading 1.56TB of weights from local fast storage (~3–7GB/s) means a cold start of ~4–8 minutes |
| Fault recovery | On a single-node failure, the workload must be rescheduled and the weights reloaded; larger nodes cost more to recover |
5.4 Download, Verification, and Storage
A 1.56TB weight package is itself an engineering exercise:
- Download channels: Besides the official repository, third-party mirror channels are typically available;
- Integrity verification: After downloading, verify each shard’s hash against the shard manifest in the index file; a missing or corrupted shard will cause the load to fail directly;
- Storage: Object storage can act as a relay between the model repository and local, with resumable downloads and multi-connection downloads; using a same-region object storage mount in the cloud reduces cross-region traffic and load time;
- Version management: Fix the weights, inference engine, and kernel libraries as a single version set; any single upgrade should go through regression testing.
5.5 Fine-Tuning: An Order of Magnitude Harder Than Inference
The vendor releases only MXFP4 weights, with no BF16 or FP8 versions, which significantly affects secondary development:
- Inference side: Mainstream inference engines vary in how well they support this format; confirm whether the kernels, operators, and parallelism strategies are complete;
- Fine-tuning side: Full-parameter fine-tuning needs high-precision weights, gradients, and optimizer states. Taking AdamW-style optimizers as an example, VRAM demand is about 16–18× the parameter count; 2.8T parameters means ~45–50TB of VRAM, far exceeding a single 8-GPU server. Even low-rank adaptation requires dequantizing the weights to higher precision, with VRAM demand several times that of inference;
- Output side: Fine-tuned weights are usually of higher precision, with size and deployment cost larger than the original model.
The realistic use of this kind of model is therefore mainly “inference + prompt engineering + external tools,” with a high bar for deep customization.
6. Selection Advice: Which Scenarios Suit Local Deployment
- Suits self-hosting: teams whose data cannot leave their premises, whose workloads are primarily coding and agent scenarios, with strong long-context needs, and that already have or can obtain 8-GPU high-VRAM nodes (or cloud heterogeneous-compute instances of equivalent power);
- Prefer the cloud API first: occasional use, quick validation of results, and teams lacking inference-optimization experience;
- Prefer smaller open models: VRAM budgets within a mainstream single 8-GPU configuration (e.g., 80GB-class) with high concurrency;
- Legal upfront: commercial products that reach the revenue or user-scale thresholds should complete their license assessment and authorization discussions before launch;
- Wait and see: scenarios that need full training pipelines and strict license standardization can wait for later releases with more permissive licenses and multi-precision versions.
7. Conclusion
The real significance of this weight release is not that “the strongest model is now downloadable,” but that it provides a complete sample through which to see the true cost structure of open-weight models:
- Model side: total parameters determine the capability ceiling, while activated parameters determine decode efficiency;
- Engineering side: VRAM capacity is the hard threshold, the KV cache is the hidden adversary of long context, and sparse attention solves compute but not capacity;
- Compliance side: both license terms and technical thresholds need to be evaluated in advance.
For technical teams, “can it run” and “is it worth running” are two different questions, and both should be answered clearly before downloading those 1.56TB.
