Deploying a 2.8T Open-Weight Model Locally: A Complete Walkthrough
A leading model team has released the weights of a flagship 2.8-trillion-parameter model, making it the largest open-weight model to date. This guide breaks down what that means in practice: how its sparse Mixture-of-Experts (MoE) architecture and sparse attention actually work, why MXFP4 quantization shapes your deployment options, how to read the vendor's benchmark claims critically, what the custom license does and does not allow, and the real hardware bill behind a 1.56 TB weight download. You'll also get a practical checklist covering license review, download verification, load testing, and production service governance, so you can decide whether this model fits your workload before committing to the infrastructure.
1. What Happened: A Heavyweight Release You Can Actually Download
In late October, a leading model team launched a flagship model first through an API and a web interface, promising to publish the full weights by November 4. On the evening of November 4, it delivered: the model repository went public on a major open-source model hosting platform, downloadable without any application. The weights became the largest open-weight release to date, roughly an order of magnitude larger than comparable models that came before.
The repository contains 112 files, including 96 weight shards, totaling about 1.56 TB. The architecture looks like this:
- About 2.8T total parameters, with only about 104B activated per token;
- 896 routed experts, 16 selected per token, plus 2 shared experts;
- 93 layers, with a context length of 1 million tokens;
- A built-in vision encoder for native multimodal input.
It is a continued-training model built on the previous generation, with only one structural change: full attention in 24 of the layers was replaced with sparse attention (DSA) to save compute under ultra-long contexts.
At launch, the vendor self-reported that the model beat a leading closed-source flagship on 16 of 37 benchmarks. A third-party evaluation body gave it a composite intelligence score of 55.
But “downloadable” does not mean “freely usable,” and it certainly does not mean “everyone can run it locally.” The license attaches revenue and user-scale thresholds, and 1.56 TB of weights places heavy demands on VRAM, memory, network bandwidth, and the software stack. The sections below take these one at a time.
2. Model Breakdown: Where the 2.8T Parameters Come From and Where the Compute Goes
2.1 Sparse MoE: More Parameters Does Not Mean More Compute
Models like this stack their parameter count out of many “expert” sub-networks. When a token passes through the routing network, it is dispatched to only a few experts, so each computation step touches only a small slice of the whole network. For this model:
896 routed experts, 16 selected per token; plus 2 shared experts active for every tokenTotal parameters: ~2.8T Activated per token: ~104B
Working backward from the total, each routed expert averages about 3.1B parameters (2.8T ÷ 896 ≈ 3.1B). The 16 routed experts for a single token add up to roughly 50B, less than half of the activated total; the rest comes from the attention layers, the shared experts, and the token embeddings, which are always part of the computation.
Two engineering takeaways follow:
- Compute scales with activated parameters; VRAM scales with total parameters. The 104B activated parameters mean the compute is comparable to a hundred-billion-class dense model, but the full set of weights still has to stay resident in VRAM. This is the root of the gap between “the model is huge” and “it runs.”
- Shared experts carry general capability. The two shared experts are used by every token, typically to absorb general language and commonsense knowledge, letting the routed experts specialize in narrower domains and easing the “expert homogenization” problem.
2.2 Sparse Attention: It Saves Compute, Not Storage
In standard full attention, every token attends to all preceding tokens, so compute grows quadratically with sequence length. Replacing 24 of the 93 layers with sparse attention noticeably cuts attention compute for long sequences.
One common misconception needs clearing up: sparse attention does not reduce KV cache storage. The cache still has to keep key-value pairs for all historical tokens; the sparse mechanism only changes which parts are read and computed at each step. As a result, the VRAM pressure from long contexts does not disappear with sparse attention. This is why the KV cache is listed separately as a cost later on.
2.3 Quantization and Native Multimodality: Precision and Deployment Are Decided Up Front
Weights use MXFP4 and activations use MXFP8. The MX formats are Microscaling formats defined by the OCP (Open Compute Project), and the basic rules are:
- Every 32 consecutive elements share one 8-bit exponent scale factor in E8M0 format;
- Each MXFP4 element uses E2M1 encoding (1 sign bit, 2 exponent bits, 1 mantissa bit) and can represent only about 6 distinct values;
- Each MXFP8 element is an 8-bit float, commonly in E4M3 or E5M2 encoding.
Elements within a block share a dynamic range, so each element needs only a few bits. This is the key to low-bit quantization balancing both precision and size.
The vendor did not release a BF16 or FP8 version. Two things must be confirmed before deployment:
- The inference engine must natively support the MX formats. Otherwise you can only dequantize offline first, which usually means roughly double the disk space plus uncertainty in precision and throughput.
- The quantized artifacts come from quantization-aware training (QAT), so quantization error has already been partly compensated during training. Do not stack additional dynamic quantization on top, as it will break that balance.
The model also includes a built-in vision encoder as part of a native multimodal design. When deploying, budget preprocessing and encoding overhead for visual inputs such as images; do not tune the inference service as if it were a text-only model.
3. Capability Assessment: How to Read Benchmark Scores Rationally
Start with the vendor’s self-reported comparison data (all vendor-tested):
| Capability dimension | This model | Leading closed-source model | Another closed-source model |
|---|---|---|---|
| Terminal coding execution (a coding benchmark v4.0) | 65.7% | 66.4% | 58.2% |
| Long-horizon software engineering tasks | 51.7% | 46.3% | — |
| Scientific reasoning (a science QA benchmark) | 38.1% | 63.3% | — |
| Hard knowledge reasoning (a hard knowledge test) | 39.4% | 60.9% | — |
The model’s strengths are concentrated in coding execution and long-horizon software engineering tasks, where it trades wins with the leading closed-source model. On scientific reasoning and hard knowledge questions, the gap is clear.
Keep these caveats in mind when interpreting such data:
- Self-reported data is selective. Vendors are motivated to show benchmarks that favor them, so the results must be cross-checked against third-party evaluations.
- Test protocols move scores significantly. The number of few-shot examples, whether tool calls are allowed, whether the score is best-of-N or single-pass, and how strict the grading is can all shift results by a few to a dozen-plus percentage points.
- Science benchmarks are extremely sensitive to knowledge updates. Once a test set becomes public, there is a risk that training data has indirectly “seen” it, so knowledge-intensive tasks warrant extra caution.
For teams building their own service, a more reliable approach is to build a private evaluation set:
from collections import Counterdef evaluate(model, cases, n_samples=5, temperature=0.7):"""Evaluate the model on a private task set and return metrics per category.cases: [{"id": ..., "input": ..., "answer": ...}, ...]"""records = []for case in cases:samples = [model.generate(case["input"],temperature=temperature,max_tokens=case.get("max_tokens", 2048),)for _ in range(n_samples)]# Business-specific judge: run unit tests for code, rule-based or human review for QApassed = sum(1 for s in samples if judge(case, s))records.append({"id": case["id"],"pass": passed > 0, # pass@n criterion: passed at least once"accuracy": passed / n_samples, # stability: fraction of runs that passed})total = len(records)return {"pass_at_n": sum(r["pass"] for r in records) / total,"mean_accuracy": sum(r["accuracy"] for r in records) / total,"flaky_cases": [r["id"] for r in records if 0 < r["accuracy"] < 1],}
pass_at_nmeasures whether the model “can do it”;mean_accuracymeasures whether the model is “stable”; combining the two reflects real production behavior;flaky_casesrecords samples that pass inconsistently, making it easier to tune prompts or add checks for those specific cases.
4. License Reading: Open Weights Are Not Open Source
The repository carries a custom model license, and the hosting platform tags it as license:other, not a standard open-source license such as MIT or Apache. According to comparison, it is word-for-word identical to the previous generation’s license, with only the name changed. This indicates these terms have become a fixed practice for the team’s flagship models, not a one-off arrangement.
The license body is MIT-style, permitting free use, copying, modification, distribution, fine-tuning, and commercial use. On top of that, two restrictions apply, and they target only large-scale external business:
- Enterprises offering the model as a service (MaaS) with cumulative revenue over $20 million over 12 months must sign a separate agreement with the model provider before commercial use;
- Enterprises whose commercial product has more than 100 million monthly active users must likewise sign a separate agreement.
For the vast majority of enterprises, internal use, product embedding, and small-scale commercial use fall outside these restrictions. The ones that need to be careful are growth-stage companies near the thresholds, such as a startup whose model-service revenue is approaching $20 million. In practice, a few points need to be pinned down:
- Revenue basis: Whether it refers to a single contracting entity or the consolidated group, and whether it counts service revenue or total revenue, should be confirmed in writing with the licensor.
- Time window: Whether it is a calendar year or any rolling 12 months determines when the obligation triggers and which metrics to monitor.
- Liability propagation: Whether derivative models fine-tuned from this model, and downstream customers’ redistribution, are equally bound.
- Version differences: License terms may differ across model generations. Open-source governance processes should retain the original license text and approval records per version rather than carrying over conclusions from the previous generation.
5. The Hardware Bill: Why “8 GPUs to Start” Is Only the Beginning
5.1 VRAM: Weights Are Just the First Line Item
| Component | Estimation method | Approximate footprint |
|---|---|---|
| Model weights | ~0.53 bytes/param × 2.8T | ~1.49 TB |
| Inference framework metadata, vocabulary, vision encoder | Measured in practice | ~70 GB |
| Weights total | Official release size | ~1.56 TB |
| KV cache | Grows linearly with context length | A few GB to a few hundred GB |
| Activations, communication buffers, framework overhead | Reserve 10%–15% of weights | ~150–230 GB |
- The weight size is verifiable. An MXFP4 element is 4 bits, plus one shared scale factor byte per 32 elements, averaging about 0.53 bytes/param. Multiplied by 2.8T, that is about 1.49 TB, roughly matching the ~1.56 TB actual release size.
- A single 8-GPU node is not roomy. At 141 GB per card, 8 cards total about 1.13 TB, which cannot hold all the weights and requires CPU offloading to make up the difference. 12 cards at about 1.69 TB leave room for the KV cache and runtime overhead. At 192 GB per card, 8 cards total 1.54 TB, close to the total weight size, but it stays tight under long contexts, so a 16-GPU multi-node setup is the safer choice.
- The KV cache cannot be ignored. A rough estimate uses the formula below:
KV bytes ≈ 2 (K and V) × num_layers × num_kv_heads × head_dim × bytes_per_value × context_length
With 93 layers, 8 KV heads, 128 dims per head, and BF16 storage, that is about 370 KB per token. 10K tokens is about 3.6 GB, 128K tokens about 46 GB, and 1M tokens a hefty ~355 GB. If the engine pre-allocates for the maximum context by default, VRAM can be exhausted at startup.
5.2 Host Memory and Disk
- Weights are usually buffered in memory during loading; with CPU offloading, the offloaded experts stay resident in memory long-term. Plan for memory of 1–1.5× the weights, i.e., over 2 TB.
- Disk space should be at least 2× the weights: one copy of the original files, one of the converted or dequantized artifacts. At 100 Mbps, downloading 1.56 TB takes about 35 hours even at full speed, so prefer a download method with resumable support.
5.3 Parallelism Strategy and Network
- Expert parallelism: MoE models commonly use expert parallelism, with different cards holding different experts and each token routed via all-to-all communication to the card holding its expert. When inter-card bandwidth is insufficient, communication becomes the throughput bottleneck.
- Single node vs. multi node: Within a node, high-speed interconnects are typically relied upon; across nodes, high-speed dedicated networking is needed, and tensor-parallel communication overhead grows sharply across nodes.
- Scheduling optimization: Routing distribution is naturally uneven, so hot experts need dynamic replication or rebalancing; otherwise a few cards become the bottleneck.
- Low-bit overhead: Unpacking and dequantizing low-bit weights must be fused into the compute kernels, or actual speed may fall well below the BF16 version.
5.4 Software Stack
- The inference engine must support MoE, MX formats, sparse attention, long context, and multimodal input all at once, which sets a fairly high version requirement; prefer an engine that tracks updates quickly.
- Pin and unify driver, compute library, and engine versions, and avoid version mismatches across nodes.
5.5 Self-Build or Cloud
| Dimension | Self-built cluster | Cloud on-demand / managed |
|---|---|---|
| Best fit | High, steady call volume; data that cannot leave its domain | Validation and evaluation; traffic with peaks and troughs |
| Cost structure | High fixed investment, including facilities, power, and operations | Pay per usage; low idle cost |
| Scaling speed | Long procurement cycle | Minutes |
| Ops burden | Requires a dedicated inference ops team | Relatively light |
A pragmatic path is to start with a small-scale validation on on-demand GPU instances in the cloud, run the accuracy assessment and capacity planning, and then decide whether to commit to a long-term self-build. Whichever route you choose, continuously monitor GPU VRAM utilization, throughput, latency, and cost per request, and configure the corresponding alerts.
6. Deployment Practice Checklist
1. License Review
Before using any model with a custom license, complete the review:
- Have legal confirm whether the revenue and user-scale thresholds are triggered;
- Clarify the group basis and how the rolling 12 months are calculated;
- Confirm that fine-tuning, distribution, and commercial use fall within the authorized scope.
2. Download and Verification
- Compare each of the 96 shards against the official hash values one by one;
- Prefer a mirror source with resumable-download support;
- Keep a hash manifest of the source files for converted or dequantized artifacts, to make provenance traceable.
3. Startup and Load Testing
# Illustrative: actual parameter names depend on the inference engine you usepython -m <inference_engine_module> <model_path> \--tensor-parallel-size 8 \ # Tensor parallel degree; must divide the number of attention heads--expert-parallel-size 8 \ # Expert parallel degree; must divide the number of routed experts--max-model-len 131072 \ # Set per business need; do not push straight to 1M--gpu-memory-utilization 0.90 # Reserve buffer to avoid mid-run OOM crashes
Load testing should cover at least these five categories:
- VRAM: Peak usage at idle and at load, and whether fragmentation or leaks appear;
- Throughput: Output tokens per second and input processing speed;
- Latency: Time to first token and time per token;
- Long context: Increase input length in stages and watch the degradation curve for VRAM and speed;
- Accuracy: The score drop of the quantized model on your private set.
4. Service Governance
- Route by scenario at the gateway layer: send coding tasks to this model first, and route scientific reasoning tasks to other models better suited to them;
- Apply length limits and graded quotas to long-context requests;
- Log the token consumption, latency, and error code of each request, to make cost allocation and issue triage easier.
7. Summary
This weight release sends a clear signal: the parameter scale of open-weight models has crossed into the trillion era. For developers and enterprises, the real questions are not “can you download it,” but three more practical ones:
- License boundary: Whether the custom license matches your business scale;
- Compute cost: Whether the VRAM, memory, and bandwidth investment that 1.56 TB of weights demands is worth it;
- Capability match: The model excels at coding execution and long-horizon software engineering tasks, but is clearly weaker than leading closed-source models on scientific reasoning and hard knowledge questions.
In engineering practice, a workable order is: do the license review first, then the compute estimate, and finally scenario-based load testing. Open weights give enterprises control over their own deployment choices, but turning that control into real productivity starts with accounting for every cost.
