Open Weights for a Trillion-Parameter MoE Model: How High Is the Bar for Local Deployment?
A Chinese AI lab has released the full weights of a 2.8-trillion-parameter Mixture-of-Experts (MoE) model—the largest open-weight model to date at roughly 1.56TB. This article goes beyond launch-day benchmarks to answer the questions that matter once the weights are actually in your hands: how the architecture is composed, what the custom license allows and forbids, how much VRAM local deployment really needs (8 high-memory accelerators to start), and whether the model is worth adopting. It walks through the sparse-activation math behind the memory bill, how to read vendor-reported benchmark numbers, why "open weights" is not the same as "open source," a step-by-step VRAM and download estimate, and a pre-launch self-testing checklist for teams evaluating it.
Introduction: The largest open-weight model has arrived, but “open” doesn’t mean “zero barrier”
On November 4, a Chinese AI lab made good on an earlier promise and published the full weights of its flagship model. The architecture is a trillion-parameter-class Mixture-of-Experts (MoE) design; the download totals roughly 1.56TB across 2.8T parameters, which the team calls “the strongest open model you can currently download.”
Two footnotes are needed, though. First, the release uses a custom license with added conditions. Second, local deployment requires at least 8 accelerators to start.
This article doesn’t repeat the launch-day benchmark scores. Instead, it answers four questions that actually affect users once the weights land:
- What’s inside: how the 2.8T parameters are composed and which engineering techniques were used;
- How the license works: whether commercial use and redistribution are allowed;
- Whether ordinary people can run it: how much VRAM is needed and how to estimate the cost;
- Whether it’s worth using: how to read the benchmarks and what to test before going live.
1. Release timeline: why open weights deserve a closer look
The model first launched on October 24 as an API and web product, at which point the team promised to release the full weights before November 4. On November 4, the weights appeared as promised on a major open-model hosting community—the repository was public and downloadable with no application required. That repository had only been created on November 2 and was still closed on the morning of the 4th before officially opening that evening. Mirror copies on community platforms can also help reduce download times.
The weight bundle contains 112 files, of which 96 are safetensors shards totaling about 1.56TB. That makes it the largest open-weight model to date, substantially bigger than any model released publicly before it.
For engineering teams, open weights mean three things you can’t get from API calls:
- Self-hosting: the model can run in your own data center or private cloud, so data never has to leave your boundary. This matters a great deal for data-sensitive industries such as finance and healthcare;
- Further development: you can build on it through continued pre-training, domain fine-tuning, or distillation to produce your own derivative models;
- Cost control: under stable, high-concurrency conditions, the per-unit cost of inference on your own compute may fall below token-based API pricing. But this only holds if compute utilization is high enough—otherwise, a large cluster sitting idle is itself the biggest waste of all.
Conversely, open weights also transfer responsibility. Model quality, inference performance, service stability, and compliance risk all become the deployer’s problem. The following sections cover each of these in turn.
2. Model breakdown: how the 2.8T parameters are composed
According to the model card, this is a natively multimodal agent model, trained further from the previous generation. Its architecture parameters are as follows:
- Total parameters: 2.8T (roughly 2.8 trillion). Each inference activates only about 104B per token—an activation rate of about 3.7%;
- MoE structure: 896 experts in total. Each token is routed through 16 experts, plus 2 shared experts that participate constantly for every token;
- Layers: 93;
- Context: 1 million tokens;
- Vision encoder: a proprietary vision encoding module with image-understanding capability.
2.1 Sparse activation: compute tracks activated params, memory tracks total params
The key intuition for an MoE model is this: compute cost depends on activated parameters, while memory cost depends on total parameters. Per token, 104B activated parameters at 4-bit come to roughly 52GB—about the VRAM of a single high-end accelerator, and that’s just the weights. The full 2.8T parameters at 4-bit come to about 1.4TB, and every expert has to stay resident in VRAM. While any given token doesn’t touch most experts, a large enough batch will “wake up” nearly all of them, so you can’t load only the activated subset.
“896 choose 16” means the routing granularity is very fine: each expert is small in capacity, but the combination space is huge, so in principle every token gets a highly customized compute path. The shared experts carry the general capabilities every token needs—basic grammar, common-sense knowledge, and so on. This shared-plus-specialized approach helps reduce redundant learning of common patterns across different experts.
2.2 Changes versus the previous generation: only one structural change
Compared with the previous generation, only one thing changed in the architecture: 24 of the layers replaced full attention with sparse attention (DSA), to save compute on ultra-long context.
A common misreading is worth correcting here: sparse attention wasn’t designed solely for long context. Attention compute scales with the square of sequence length, so its share is small for short text and grows more prominent as context lengthens. There’s another often-overlooked detail: only 24 of the 93 layers were converted; the remaining 69 still use full attention. So the more accurate description is “saving compute for long context,” not “long context at zero cost”—and what actually gets tight in VRAM is often a different component.
2.3 Quantization: only one version was released
The weights themselves use the MXFP4 format, with MXFP8 activations, and are a product of quantization-aware training (QAT). QAT simulates quantization error during training so the model adapts to low precision ahead of time, which typically loses less accuracy than post-training quantization (PTQ). This is also the team’s basis for daring to release only one version. Two points deserve special attention, though:
- No BF16 or FP8 version: if you want higher-precision weights, you’d have to dequantize them yourself, and the compressed information can’t be recovered;
- Requirements on the inference engine: the engine must support MX-format dequantization and MXFP8 activations. Many common 4-bit schemes are built on INT4 or group-wise FP4, which are not compatible with MXFP4—so don’t assume that “supports 4-bit” means “will run.”
2.4 A byproduct of training on thousands of GPUs
896 experts, 93 layers, a 1-million-token context, native multimodality—a configuration like this essentially only appears on training clusters of a thousand GPUs or more. The move from full attention to sparse attention, and from BF16 to MXFP4, is fundamentally the same logic: pursuing efficiency under compute constraints.
This explains why only a single MXFP4 version was released: the original weights were produced in exactly this kind of low-precision, high-parallelism training environment. The flip side is that any inference environment capable of running such a model must have large-scale multi-GPU interconnect—so “local deployment” was never meant for a single person with a single card.
3. Reading the benchmarks: how to interpret vendor-reported numbers
All of the vendor’s comparison data are self-reported, in which Model A and Model B are two frontier closed-source models:
| Evaluation category | This model | Model A | Model B |
|---|---|---|---|
| Terminal coding benchmark 4.0 | 65.7% | 66.4% | 58.2% |
| Long-horizon software engineering benchmark | 51.7% | 46.3% | — |
| Multimodal science benchmark | 38.1% | 63.3% | — |
| Extreme-difficulty subject knowledge benchmark | 39.4% | 60.9% | — |
The official framing is that this model beats Model A on 16 of 37 benchmarks. A third-party independent evaluation platform also gives it a composite intelligence index of 55, placing it near the global top.
Lay the table out and the picture is clear: the results are mixed—strong on coding and agent tasks, weaker on science and some reasoning tasks.
| Capability dimension | Result | Notes |
|---|---|---|
| Terminal coding | Roughly on par with A, about 7.5 points ahead of B | First-tier level |
| Long-horizon software engineering | About 5.4 points ahead of A | Better stability across multi-step task chains |
| Multimodal science | About 25 points behind A | Clear gap in scientific image reasoning |
| Extreme-difficulty subject knowledge | About 21.5 points behind A | Complex reasoning remains a weak spot |
How to look at these numbers rationally:
- The vendor has an incentive to cherry-pick: the publisher naturally picks benchmarks that favor it, and a 16/37 ratio itself shows the overall picture is mixed—don’t just look at the highlights;
- Watch for evaluation-environment differences: self-reported scores usually involve prompt tuning, best-of-multiple sampling, and preset toolchains, which aren’t equivalent to a single call in a real business scenario;
- Look at discrimination, not absolute values: Model A scores only 66.4% on Terminal coding 4.0, which tells you this benchmark version is hard. When questions are broadly difficult, a one- or two-point gap may correspond to just one or two items. What really matters is the magnitude of the difference—for example, the roughly 25-point gap on science tasks;
- Read third-party rankings for trends: an independent platform’s composite index only reflects a rough position, and the specific ranking shifts as models are added or removed, so it’s better for spotting trends than for reading exact placements.
The conclusion isn’t “does this model work or not,” but rather: if your workload is mostly code generation, terminal operation, and multi-step tool calling, it deserves a spot on your shortlist; if it’s scientific computing, professional image interpretation, or complex reasoning, be prepared for gaps and validate with your own data.
4. License analysis: open weights is not the same as open source
This is the point most worth calling out separately in this release. The repository carries a custom license, and the hosting community’s metadata marks it as license:other—neither MIT nor a standard open-source license like Apache-2.0. That license is word-for-word identical to the previous generation’s, with only the name changed.
The main clauses are genuinely MIT-style: free to use, copy, modify, distribute, fine-tune, and use commercially. That’s far more permissive than many model licenses that allow research only and forbid commercial use, and it’s the basis on which the model can be called “open.” But the meaning of a custom license lies precisely in the additional stipulations beyond the MIT-style text. Common add-on clauses in the industry include:
- Attribution and notice obligations when redistributing or offering the model as a service;
- Exclusions or restrictions on certain use cases;
- Additional requirements for large-scale commercial use or offering hosted services;
- Naming and labeling requirements for derivative models.
So the question “can I use it commercially?” can’t be settled by the single line “commercial use allowed”—you have to check the original text against your specific way of using it.
One common confusion also needs clearing up:
- Open source emphasizes open source code and the freedom to research, modify, and redistribute. Under the Open Source Definition published by the Open Source Initiative (OSI), a compliant open-source license must not impose field-of-use restrictions;
- Open weights only means the model weights can be downloaded. Whether training code, data mixtures, and the full training pipeline are included depends on the publisher.
So as long as the license contains use-case restrictions, the model can’t be considered open source in the strict sense—“open weights” is the more accurate term. A complete set of tens of weight files doesn’t mean you can train it from scratch.
Practical advice for enterprise users:
- Submit it for legal review: have your legal team check the original text clause by clause against your use case, especially the scenario of offering it as a service to others;
- Keep a copy of the license: archive the license text, version number, and timestamp from the moment of download, to guard against unilateral changes upstream;
- Document your derivative work: record the naming and labeling of derivative models in your release notes and model card;
- Meet regulatory requirements: if you offer generative AI services to the public, you may also need to satisfy jurisdiction-specific requirements, such as security assessments and algorithm registration;
- Accept upstream risk: custom licenses are maintained at the publisher’s discretion, and there is precedent for this in the open-model space—reports note that some popular models originally released under custom licenses later switched to standard open-source licenses. From an engineering standpoint, keep the ability to migrate to another model.
5. The deployment ledger: how much VRAM does 1.56TB actually require?
5.1 Do the VRAM math
Deployment VRAM roughly breaks down into four parts:
VRAM need ≈ weights + KV cache + runtime overhead + headroom
At 2.8T parameters and 4-bit, the weights come to about 1.4TB, and the download comes to about 1.56TB once shard metadata is included. To hold roughly 1.5TB of weights across 8 cards, each card must hold at least about 190GB. This means “8 cards to start” is possible only with whole machines of a few new-generation high-memory accelerators—it’s not just a matter of counting cards. Plug your own hardware into the table below for the exact card count:
| VRAM per GPU (est.) | GPUs needed for weights only | Practical starting point |
|---|---|---|
| 80GB-class | 19 | 20+ |
| 96GB-class | 16 | 16–20 |
| 141GB-class | 11 | 12–16 |
| 192GB-class | 8 | 8+ |
The practical starting point is usually a few cards more than the theoretical value, because of runtime overhead:
- KV cache: the KV cache for a 1-million-token context can reach tens or even hundreds of GB. The exact size depends jointly on the number of layers, KV heads, head dimension, precision, and parallelism strategy, and can be estimated with the formula below:
KV VRAM ≈ 2 (K and V) × full-attention layers × KV heads × head dimension × bytes per element × sequence length
With 69 full-attention layers, 8 KV heads, a head dimension of 128, and FP8 storage, each 10,000 tokens need about 1.4GB, and 1 million tokens need about 140GB—divided across 8 cards, roughly 17.5GB per card.
- Runtime overhead: the framework’s own footprint, communication buffers, and activations, temporary tensors, and so on;
- Long-context bottleneck: the bottleneck in long-context inference is often not the weights but the KV cache. The longer the sequence, the less room is left for concurrency.
These figures are order-of-magnitude estimates—rely on measurements after actual deployment.
5.2 Downloading is an engineering task in itself
A 1.56TB weight bundle can’t be downloaded straight from a browser; engineering-wise you need:
- Resumable downloads: 96 safetensors shards plus small files like config.json and tokenizer settings—112 files in total—where any single failure must be retryable on its own;
- Shard verification: check the hash of each shard after download; corrupted shards often don’t surface until you load the weights;
- Nearby mirrors: download from a nearby mirror to avoid speed losses from long-haul network traffic;
- Storage placement: weights are best kept in object storage for centralized management, organized into per-version directories, then mounted to inference nodes.
A rough download estimate: on a 100 Mbps line at full speed, it takes about 35 hours; on a gigabit connection, about 3.5 hours. Even downloading only the 96 shards (about 1.5TB) takes around 30 hours. For teams that switch models frequently, weight storage and distribution is itself infrastructure that needs planning.
5.3 One-line conclusion: ordinary people can’t run it locally
Consumer GPUs typically have 24GB of VRAM or less per card, and even high-end workstation cards are usually in the tens of GB—yet this “entry ticket” demands 8 cards per machine with nearly 200GB each. In terms of hardware scale, that’s a small data center. For individual developers, only three realistic paths exist:
- Rent cloud GPU instances on demand;
- Use the official API directly;
- Use a managed inference service.
If you truly must deploy in your own environment while keeping the purchase small, you can consider using a distilled or smaller model to handle most requests, reserving the trillion-parameter model for the most complex tasks. This is also the common hybrid architecture for putting such models into production today.
6. Selection approach: who should run it, and how
Back to the original question—can ordinary people run it? Let’s answer it separately.
Individual users who want to run it locally: the answer for now is no, and there’s no need to. Using it via API offers better value.
Enterprises with GPU resources: the key isn’t “can we run it” but “once it’s running, can we use it well.” On a trillion-parameter model like this, VRAM is just the entry barrier—what really drives cost is utilization. If your average daily request volume can’t keep 8 high-end cards at high load, the per-unit cost of self-hosting will likely exceed token-based API pricing.
Teams preparing to put it on the shortlist should evaluate in this order:
- Business fit: is your workload mostly code, terminal operation, and multi-step tool calling? If so, test it first; if it’s mostly scientific reasoning, lower your expectations;
- License compliance: can you meet the custom license’s added conditions, and are you willing to accept the risk of unilateral upstream changes?
- Engineering fit: does your inference engine support MXFP4 weights and MXFP8 activations, and can your team handle the ops work of multi-GPU parallelism, inference optimization, and monitoring/alerting?
- Total cost of ownership: factor in hardware depreciation or cloud rent, power, ops headcount, and storage and bandwidth, then compare against API costs.
Expressed in code, this judgment can be written plainly:
# Pseudocode: feasibility check before adopting a trillion-parameter-class open-weight modeldef should_self_host(model, use_case, license_ok, engine_supports_mx, util):# 1. The license is the first hard gate: allowing commercial use doesn't mean it allows your use caseif not license_ok:return False, "License terms don't match; decide after legal review"# 2. Engine support is the second hard gate: common INT4 schemes aren't necessarily MXFP4-compatibleif not engine_supports_mx:return False, "Inference engine must support MX-format dequantization and MXFP8 activations"# 3. Business fit: quantify it with your own eval set, not vendor benchmarksif use_case.task_score < BASELINE_MODEL.task_score:return False, "Doesn't beat the current baseline model in this business scenario"# 4. Economics: convert self-hosted per-token cost based on actual utilizationif util < UTIL_THRESHOLD:return False, "Utilization too low; idle-cluster cost of self-hosting is too high"return True, "Ready to enter a pilot"# The meaning is illustrative only: the gates aren't limited to these four, and VRAM must be budgeted separately using the earlier formula
Pre-launch self-testing checklist
Vendor benchmarks can only help you filter out unsuitable candidates; the final decision relies on your own evaluation:
- Task-level testing: use no fewer than a few hundred real business samples, with human-labeled reference answers or implementations, focusing on function-call success rate, instruction-following across multi-turn dialogue, and retrieval and reasoning over long documents;
- Context-length testing: use “needle-in-a-haystack” methods to check recall of key information at different lengths—100K, 500K, 1M tokens, and so on. A stated context length isn’t the same as an effective context length;
- Multimodal testing: prepare real business images—screenshots, forms, receipts—to test multimodal capability;
- Throughput and latency testing: measure time-to-first-token (TTFT), time-per-output-token (TPOT), and output tokens per second, separately under single requests and low concurrency. MoE models vary widely in performance across different batches;
- Baseline benchmarking: compare against your current production model on the same test set, then convert to a per-task cost using unit price to get comparable numbers.
Sample inference parameters
The YAML below only expresses the key configuration dimensions; field names vary by inference framework:
model_path: /mnt/models/xxx-2p8t-mxfp4 # Path to the weightstensor_parallel_size: 8 # Tensor parallelism degree; must match the machine's card countmoe_backend: mxfp4_w_mxfp8_a # Expert compute backend: MXFP4 weights, MXFP8 activationsenable_expert_parallel: true # Add expert parallelism for large-scale deploymentmax_model_len: 131072 # Set per actual workload; don't max it out to 1Mkv_cache_dtype: fp8 # Low-precision KV cache storage to further shrink VRAMenable_prefix_caching: true # Reuse KV cache for shared prefixes like system prompts
Two practical reminders:
- Don’t max out the context: max_model_len directly affects how much KV cache is reserved; setting it as needed is what improves concurrency;
- Reuse shared prefixes: fixed prefixes like system prompts and tool definitions don’t need to be recomputed every turn once prefix caching is enabled, which noticeably reduces time-to-first-token.
Conclusion: open weights are a starting point, not the finish line
The arrival of this weight bundle turns “the strongest open model” from a number on a launch slide into an engineering object you can download, deploy, and fine-tune. Behind each figure—2.8T parameters, 1.56TB, 8 cards to start—is a clear engineering meaning:
- 2.8T and 104B: compute and memory are two separate ledgers;
- 1.56TB: a concrete demand on your network, storage, and hardware;
- 8 cards to start: defines the real barrier to “local deployment”;
- Custom license: a reminder that open weights still fall short of open source.
For most developers, the rational stance is to treat it as a candidate tool, not a silver bullet. On coding and agent tasks, it’s already first-tier; on science and complex reasoning, the gap is still clear. What ultimately decides whether it lands is always three things: whether the license fits, whether the scenario fits, and whether the cost adds up.
