Running a 2.8T Open-Weight Model: 1.56TB Weights and the Real Math Behind the 8-GPU Floor
The day a model's full weights go public, the questions developers ask stop being "how good is it" and start being "can we run it, is it worth running, and can we ship it commercially." This article examines what actually lands after a 2.8T-parameter model releases its complete weights — a 1.56TB download split across 96 shards, currently the largest openly downloadable open-weight model. We break down the MoE architecture (2.8T total, 104B activated per token), the custom license and the revenue and user thresholds hidden inside it, the VRAM reality behind the "8-GPU" floor, the decoding and memory-bandwidth bottleneck, MX-format quantization compatibility, and the path from download to production. The conclusion: open weights hand the deployment decision back to you, but that decision carries a clear price tag in capacity, licensing, and performance.
Introduction: The Test Starts After the Weights Land
On November 4, 2025, a newly released model (referred to here as “the model”) published its complete weights as previously promised, downloadable with no application required. The shards add up to roughly 1.56TB, making it the largest open-weight model currently available for public download. Once the news broke, the question developers cared about most was no longer “how strong is the model” but three far more practical ones:
- Can we run it? What’s the hardware floor?
- Is it worth running? Do cost and quality line up?
- Can we use it commercially? What does the license actually permit?
This article deals only with what happens after the weights land: what’s inside them, how the license is calculated, whether an ordinary team can run the model, and which costs to work out before you start. The conclusion: open weights hand the deployment choice back to the user, but that choice comes with an explicit price tag.
Release Timeline: From “Promised” to “Shipped”
Understanding how the weights were rolled out helps judge whether a model fits into a long-term technology choice.
- Capability first. In late October, the model became available through an API and a web interface. Many teams had already gotten used to its coding and task-execution ability and built prototype applications on top of the API.
- Promise second. At launch, the full weights were explicitly promised within one week. This “service first, open weights later” approach lets teams try before they commit and collect feedback early.
- Delivery on schedule. The weights opened on the very day promised, split into 112 files, of which 96 are weight shards, with no application required. In addition to major international model-hosting platforms, the files are also mirrored on other communities.
For evaluators, the more telling point is the “stay closed before delivery, open in one batch at the deadline” process. It shows the weight release wasn’t a marketing afterthought but part of a pre-scheduled plan. For teams preparing to put the model into production, that predictability is itself valuable information.
Inside the Weights: How the 2.8T Parameters Break Down
MoE Sparse Structure: Total vs. Activated Parameters
The model is a natively multimodal agent model, continued-trained from the previous generation. Its overall scale:
- Total parameters: 2.8T (trillion);
- Activated parameters per forward pass: 104B (billion);
- Network depth: 93 layers;
- Context length: up to 1 million tokens;
- Vision: a proprietary vision encoder with image input support.
It uses a mixture-of-experts (MoE) structure:
- Expert layers: each MoE layer holds 896 routed experts, and each token activates only 16 of them;
- Shared experts: another 2 shared experts are always active;
- Experts active per layer: 16×1 + 2 = 18.
Note that “896” typically refers to the number of experts per layer. The official disclosure doesn’t itemize everything, so the model card remains the source of truth for exact figures.
This design produces one key takeaway: parameter count is not the same as compute consumption.
- VRAM: must hold all 2.8T parameters;
- Compute: each token passes through only about 3.7% (104B ÷ 2.8T) of the parameters.
That’s the classic profile of “large parameter count, relatively small compute.”
Attention Optimization: Saving Compute for Long Contexts
Compared with the previous generation, the official architecture has a single change: 24 full-attention layers were replaced with sparse attention (DSA).
The cost problem with full attention: in standard attention, every query scans all historical tokens, and the compute grows with the square of sequence length. At a 1-million-token context, using full attention in every layer would make prefill costs unacceptably high.
The idea behind sparse attention: let each query attend only to the history most relevant to it. By combining a local window with key-token selection, it sharply reduces the number of attention pairs that actually need computing. As a result:
- Prefill compute drops noticeably, and time-to-first-token for long documents and long conversations improves;
- The trade-off is that it needs a dedicated compute kernel, so you must confirm your inference framework supports it;
- From public information, every layer and every attention head still keeps a full KV cache to draw from, so VRAM usage for long contexts isn’t actually reduced.
Quantization-Aware Training: Low Precision Is the Design Target
The model is a product of quantization-aware training (QAT):
- Weight format: 4-bit microscaled float (MX series);
- Activation precision: 8-bit microscaled float;
- Release version: this is the only precision offered; no higher-precision version has been published.
QAT differs fundamentally from common post-training quantization (PTQ). PTQ compresses a model after training is done and tends to lose precision. QAT folds quantization error into the training process, teaching the model to work in low precision, which is why it can hold high accuracy even at an aggressive 4-bit width.
But this also means low precision is the model’s native form, not a temporary compression. When hardware doesn’t support the format, converting it to a higher precision on your own is a workable compatibility measure, but it can’t recover information already lost during QAT.
Reading the License: Open Weights ≠ Open Source
The License Structure
The model repository uses a custom license, tagged “other” on the hosting platform rather than as a standard open-source license like MIT or Apache. Its structure has two layers:
- Base layer: essentially the same as the MIT license. It allows free use, copying, modification, distribution, and commercial use, covering common scenarios such as fine-tuning, redistribution, and embedding in commercial products;
- Restriction layer: it constrains only large-scale external business:
- Clause 1: if you offer the model as a service and 12-month cumulative revenue exceeds US$20 million, you need a separate agreement before commercial use;
- Clause 2: if your commercial product exceeds 100 million monthly active users, you also need a separate agreement (refer to the license text for exact figures).
Who Do These Clauses Affect
Put the two layers together and the impact is fairly clear:
- Individual developers, startups, and the vast majority of small and mid-sized companies: under normal usage they’ll barely feel any extra restriction and can use, fine-tune, and commercialize the model freely;
- Large-scale cloud services or AI platforms: if you’re running a scaled business that trades on the model’s capabilities for revenue, you’ll need to assess whether you cross the revenue threshold and prepare to sign an agreement;
- Products with hundreds of millions of users: even if you don’t sell the model directly, check this condition. The key tests are “user scale” and “whether you offer a service to others,” not “whether you sell the model directly.”
Why “Open Weights” ≠ “Open Source”
The difference between the two isn’t whether the weights are public, but whether the terms of use attach a threshold based on identity or scale.
| Dimension | Standard open-source license (e.g., MIT, Apache-2.0) | The model’s open-weight license |
|---|---|---|
| Commercial use | Allowed | Allowed; very-large-scale cases require a separate agreement |
| User restrictions | None | Tied to revenue and MAU scale |
| Modification and redistribution | Allowed | Allowed |
| Redistribution obligations | Retain copyright and license notice | Same requirement to retain notices |
Strictly speaking, this kind of license is “conditional openness.” It keeps the most central freedoms of the open-source spirit — use, modify, distribute, commercialize — but sets up a separate negotiation channel for a handful of very large players. Evaluating both licenses side by side is how you avoid mistaking “free to download” for “free to commercialize unconditionally.”
Three Practical Tips for Teams
- Do a license review before it goes in the repo: add the license text to your internal compliance checklist and confirm whether your business might trigger the revenue or user-scale threshold;
- Pin the weight version: record the weight version and license terms so a future version update that changes the license doesn’t slip by unnoticed;
- Keep redistribution materials: when redistributing, retain the original license file, copyright notices, and any usage-restriction content from the model card.
# Illustrative code: a first-pass license check, not a substitute for legal advicedef can_use_commercially(license_terms, is_maas=False, is_self_hosted=False,annual_revenue_usd=0, product_mau=0,):if not license_terms.allow_commercial:return False, "Base license does not permit commercial use"# Whether the threshold clauses apply to "self-hosting" depends on the license textif is_self_hosted and is_maas:if annual_revenue_usd > license_terms.maas_revenue_cap:return False, "External-service revenue exceeds threshold; separate agreement required"if product_mau > license_terms.maas_mau_cap:return False, "Product MAU exceeds threshold; separate agreement required"return True, "Commercial use allowed within the license terms"
Benchmarks: Strengths, Weaknesses, and How to Read Them
A few groups of numbers in the official comparison table are worth noting:
| Capability | The model | Leading closed model A | Another closed model B |
|---|---|---|---|
| Terminal coding tasks | 65.7% | 66.4% | 58.2% |
| Long-horizon software engineering | 51.7% | 46.3% | Not listed |
| Scientific terminal tasks | 38.1% | 63.3% | Not listed |
| Graduate-level problems | 39.4% | 60.9% | Not listed |
Note: all of the above are vendor-reported figures; test names and scores are taken from the original report.
The model comes close to or exceeds top closed models on coding execution and long-horizon agent tasks, but lags clearly on scientific reasoning that demands deep domain knowledge. The vendor claims that of 37 benchmarks, 16 beat the competition; looking at the table, it’s genuinely a mixed record.
A third-party evaluation body also gave the model an intelligence index of 55, ranking it fourth globally. But different organizations use very different indicator systems and weightings, so the ranking is only a reference point and can’t replace validation against your own workloads.
Three things to watch when reading results like these:
- Self-reported framing: the benchmark names, versions, question selection, and runtime environment are all chosen by the publisher;
- Capability differences: agent tasks test multi-step planning, tool use, and error recovery; a strong general model doesn’t necessarily lead on single-domain Q&A;
- Scores aren’t experience: the real path from input to output also involves engineering factors such as system prompts, tool interfaces, and retrieval quality.
VRAM Math: What “8 GPUs” Actually Means
Where the Weight Size Comes From
Basic arithmetic confirms the weight-file size:
2.8 × 10^12 parameters × 0.5 bytes/parameter ≈ 1.4 × 10^12 bytes = 1.4 TB
This is close to the roughly 1.56TB the vendor reports. The difference comes mainly from overhead in file metadata, the shared and routed networks, the vision encoder, and similar parts.
VRAM Needs Go Beyond Weights
Runtime VRAM contains at least four parts:
- Model weights: must stay resident in VRAM in full;
- KV cache: grows with context length and concurrent requests;
- Compute and communication overhead: activation tensors, framework reservations, parallel-communication buffers;
- Vision input: temporary tensors from image encoding.
The KV cache can be estimated with the formula below:
KV bytes = sequence length × layers × 2 (K and V) × KV heads × head dim × bytes per element
Because the weights are already around 1.4TB, the KV cache is comparatively small. But it still eats into the headroom available for concurrency; refer to the model card for exact head counts and dimensions.
Why “8 GPUs Is the Starting Point”
Take a high-end GPU with 80GB of VRAM per card as an example:
- Card count: 8 cards total only 640GB, less than half the weight size;
- Gap: even if all VRAM went to weights, you’d still be short about 760GB;
- Real threshold: if you insist on a single 8-GPU node, estimating for “weights + KV + buffers,” each card needs about 200GB of VRAM, or you’d pair it with host-memory offload (CPU offload).
So “8 GPUs” is more like a lower bound. Total VRAM, per-card VRAM, and inter-card interconnect bandwidth are the real hard constraints. When planning deployment, start with a capacity estimate using a script like the one below:
def estimate_vram(total_params, bits_per_param=4, layers=93,seq_len=32768, kv_bytes_per_token=100, # illustrative; verify against the model cardoverhead_ratio=0.12,):weights = total_params * bits_per_param / 8kv_cache = seq_len * kv_bytes_per_tokentotal = (weights + kv_cache) * (1 + overhead_ratio)return {"weights_gb": round(weights / 1024, 1),"kv_cache_gb": round(kv_cache / 1024, 2),"total_gb": round(total / 1024, 1),}# 1B = 1 billion parameters; results are in GiBif __name__ == "__main__":print(estimate_vram(total_params=2.8e12)) # ~1533.7 GiBfor gpus in (8, 16, 32):per_gpu = estimate_vram(total_params=2.8e12)["total_gb"] / gpusprint(f"At {gpus} GPUs, each card needs about {per_gpu:.0f} GiB")
The results show that at 8 GPUs each card needs about 192GiB, and at 16 GPUs about 96GiB. So “doubling the cards” doesn’t necessarily mean “doubling the cost”: with fewer cards, the VRAM requirement per card gets more demanding.
Choosing a Parallelism Strategy
Once the card count is set, you still need to decide how to split the model across cards:
- Tensor parallelism (TP): slices each layer’s weights across multiple cards. It requires per-step inter-card sync, so it fits single-node setups with high-speed interconnect;
- Expert parallelism (EP): spreads the 896 experts across cards. Each card loads only a subset of experts, which fits the MoE structure naturally and uses the least per-card weight memory;
- Pipeline parallelism (PP): splits layers across cards or nodes, passing only activations between stages, at the cost of pipeline bubbles.
Large-scale MoE inference usually mixes these. In practice, first decide how many experts stay resident per card, then work backward to the card count and network bandwidth you need.
The Decoding Bottleneck: Enough Compute ≠ Fast Enough
Token-by-Token Decoding Is Memory-Bandwidth-Bound
During autoregressive generation, every token requires the GPU to read the activated experts’ weights from VRAM into the compute units. With 104B activated parameters at 4-bit weights:
- Read per token: 104B × 0.5 bytes = 52GB.
Assuming further that 8 cards deliver about 20TB/s of aggregate VRAM bandwidth (an assumption only; use actual hardware specs), the theoretical ceiling is:
20 TB/s ÷ 52 GB/token ≈ 384 token/s
This is an upper bound that ignores every other cost. Real single-request speed is usually only a fraction of it, because of:
- attention computation;
- routing and expert dispatch;
- parallel communication;
- memory-access efficiency.
Trading Batching for Throughput
MoE inference has one key advantage: within a batch, requests route to highly scattered experts that collectively touch almost all the weights, so VRAM-bandwidth utilization is very high. That means throughput can approach the ceiling computed above as concurrency rises.
The cost is higher queue latency. Production systems have to trade off “single-request response speed” against “overall throughput.” Common techniques include:
- continuous batching;
- dynamic batching with request-priority scheduling;
- separate scheduling for prefill and decode;
- prefix caching to reuse repeated system prompts and history across turns.
The Quantization Compatibility Trap
MX-series weights require the compute units or the inference framework to support the corresponding decoding at the hardware level. If the hardware doesn’t support it, the framework usually has to dequantize to a higher precision (such as BF16 or FP8) before computing:
- VRAM overhead: may multiply;
- Speed: not necessarily faster;
- Hardware floor: effectively raised.
There’s also a more overlooked problem: the quantization scale factors needed by the 4-bit QAT model are carried in the weight file, and other quantization schemes can’t simply “borrow” this information.
There are two possible alternatives:
- use weight quantization to compress the weights further;
- use KV-cache quantization to cut long-context overhead.
Both need recalibration and accuracy validation, and neither is cheap.
On top of that, sparse attention depends on a dedicated compute kernel. If your chosen framework doesn’t support it, it falls back to a generic implementation, which can be noticeably slower for long contexts.
Follow one principle when selecting: confirm the weight format’s compatibility with your framework and hardware before deciding which cards to buy and use.
Engineering: From Download to Live
Acquiring and Verifying the Weights
- Download integrity: if any of the 96 shards is missing or corrupt, the model won’t load; compare each one against the official checksums;
- Download acceleration: a single file can run to tens of GB, and even on a saturated gigabit connection it can take over two hours. Teams with unstable access to international platforms should prefer a mirror closer to them;
- Storage planning: budget at least about 2× the space for the weights — one copy for the raw shards and one for the files actually loaded.
Deploying the Inference Service
A reference deployment looks like this:
┌──────────────┐ request routing ┌─────────────────────────────────┐│ Gateway │ ────────────────────► │ Inference instances (multi-GPU) ││ auth / rate │ ◄──────────────────── │ weight shards · KV cache · batching ││ limit/audit │ streaming response └─────────────────────────────────┘└──────────────┘ ▲│ │▼ │┌────────────────────────────────────────────────────────────┐│ Logging / monitoring & alerts: TTFT · output speed · VRAM usage · error rate │└────────────────────────────────────────────────────────────┘
Key points:
- Parallelism config: on a single node, prefer high-bandwidth inter-card interconnect; for cross-node deployment you need low-latency, high-speed networking, and expert parallelism is worth considering first;
- Capacity headroom: leave a safety margin in VRAM and concurrency so a burst of long-context requests doesn’t stall or reject an entire batch;
- Progressive rollout: start by gray-releasing a small amount of traffic to validate quality, then scale up gradually.
Storage and Distribution
At the TB scale, distributing the weights is itself an engineering problem:
- Object storage: hold weight replicas and distribute them nearby;
- Content delivery: serve cross-region users from a nearby location;
- Version management: record weight versions and license terms;
- Caching: cache weights locally on inference nodes to avoid re-pulling them on every startup.
Fine-Tuning Options
If you need to optimize further on business data, low-rank adaptation methods like LoRA train only a small number of new parameters:
- VRAM-friendly: the base weights stay resident and the extra VRAM cost is small;
- Scope: suitable for style alignment and task adaptation, not for injecting large amounts of new knowledge;
- Merge limitations: if you plan to merge the adapted weights back into the base and redistribute, merging introduces new numerical error that also needs evaluation.
Observability
A self-hosted model service can’t be judged only by “does it reply.” At minimum, monitor:
- Latency: time-to-first-token, inter-token output interval, end-to-end time;
- Throughput: token throughput, concurrency, queue time;
- Resources: VRAM usage, VRAM-bandwidth utilization, inter-card communication overhead;
- Quality: task success rate, human-takeover rate, harmful-output interception rate.
Deployment Strategy: Hosted First, Self-Hosted Later
For most teams, “self-hosting” shouldn’t be the starting point but the end of an evolution:
- Prototype phase: use a hosted cloud service to validate business value. Focus on general factors like stability, rate-limiting policy, and data region;
- Optimization phase: once usage and the cost model are clear, evaluate self-hosting, comparing “cost per token × monthly call volume” against hosted fees;
- Self-hosted phase: embed the model service in your own infrastructure. General capabilities from cloud platforms such as Baidu AI Cloud — GPU compute, container services, object storage, log monitoring — can carry weight storage, inference serving, and operational observability.
One last reminder: a model’s “openness” doesn’t mean every weight under the same or a similar name can be used freely. Rely on the weights and license text published through official channels — that’s the first line of defense against risk.
Summary: In the Open-Weight Era, Engineering Is What Counts
Five sentences to close:
- Structure sets the cost: the 2.8T-total, 104B-activated MoE design makes compute far smaller than the parameter count, but VRAM must hold all the weights;
- 8 GPUs is just the start: at 4-bit weights, each card needs about 200GB of VRAM, or host-memory offload;
- Bandwidth is the bottleneck: single-request decoding is bounded by VRAM bandwidth, and batching is the key to throughput;
- The license must be read carefully: open weights allow commercial use, but very-large-scale external service may trigger a separate agreement;
- Capability has limits: coding and agent execution stand out; scientific reasoning still lags.
Public weights are only the start. For developers, the real threshold has shifted from “can we get the model” to “can we evaluate it, do the math, and manage it.” Only by getting these three accounts straight — capacity, licensing, and performance — can you decide whose machine this trillion-parameter model should run on.
