Deploying a 2.8T-Parameter Open-Weight Model on 8 GPUs: VRAM, Licensing, and Benchmarks
A leading AI lab has released the full weights of its flagship model: 2.8 trillion parameters, roughly 1.56TB of weight files, making it the largest openly available model by parameter count at the time. This article focuses on three engineering questions that matter more than the download itself: how to load and serve 1.56TB of weights locally, whether the model's custom license permits commercial use, and how to read vendor-reported benchmarks for a real procurement decision. It covers the model's MoE architecture (2.8T total parameters, 104B activated per token), the VRAM math behind the "8-GPU" starting point, the KV-cache demands of long-context inference, an example deployment configuration, license constraints, benchmark interpretation, and recommendations for teams of different sizes.
Introduction: What an “Open Weights” Release Really Means
On November 4, 2025, a major model developer released the complete weights of its latest flagship model on a public model hosting platform, as previously promised: 2.8 trillion parameters (2.8T) in total, with all weight files adding up to roughly 1.56TB—the largest publicly available model by parameter count at the time. Yet the real impact of this release goes well beyond “one more downloadable model.” It raises three engineering questions at once:
- How do you load 1.56TB of weights locally and run inference?
- Can an “open weights” model under a custom license actually be used in commercial products?
- How should the officially reported benchmarks be interpreted to support a real procurement decision?
For teams planning to build their own inference service or run a private deployment, answering these three questions matters far more than counting how many files were downloaded.
1. Recap: From API Launch to Weight Availability
The new model first launched via API and web on October 24, when the developer promised the full weights would be released before November 4. On November 4, the weights were published on schedule and made available to anyone without an application.
The hosting repository was created on November 2 and remained closed during the day of November 4, officially opening that evening. The repository contained 112 files, of which 96 were sharded weight files totaling roughly 1.56TB. Sharding is a standard practice for ultra-large models, but its engineering implications are often underestimated:
- Shards and parallelism mapping: the number of weight shards is usually tied to deployment partitioning methods such as tensor parallelism and expert parallelism; the shard count and distribution often hint at the inference parallel topology the developer has validated.
- Download and verification: downloading over a hundred files is prone to partial failures, so you must verify each shard against the checksums provided in the repository and confirm that downloads can resume after an interruption. In practice, a more robust approach is to fully sync the weights to object storage first, then mount them to inference instances. General-purpose object storage from a provider such as Baidu AI Cloud can serve this role, and combined with image pre-warming, it can shorten the time to scale out new instances.
This is the largest open-weight model to date, an order of magnitude larger than comparable earlier models. The parameter scale brings not just the expectation of being “stronger,” but also a whole new set of deployment constraints.
| Dimension | Representative meaning | Engineering focus |
|---|---|---|
| Total 2.8T parameters | weight files ~1.56TB | requires multi-GPU tensor-parallel loading |
| 104B activated per token | MoE sparse activation | decode compute pressure close to a dense ~100B model |
| 1M-token context | ultra-long context | KV-cache capacity and long-sequence attention cost |
| Quantized format (4-bit weights) | only one version offered | high VRAM floor, no lower-precision option |
2. Model Architecture: 2.8T Total Parameters, 104B Activated
According to the model card, the model is a natively multimodal agent model, continued training from the previous generation. Its main parameters are as follows:
- Overall architecture: MoE (mixture of experts) with 896 experts in total; each token is routed to 16 experts for computation, plus 2 always-active shared experts.
- Depth and context: 93 layers, with a maximum context length of 1 million tokens.
- Vision: equipped with a proprietary vision encoder, supporting image understanding.
- Structural change: compared with the previous generation, only one change was made—replacing 24 layers of full attention with sparse attention, primarily to reduce the compute cost of long contexts.
- Quantization: weights are in a 4-bit microscaling floating-point format, and activations in an 8-bit microscaling floating-point format, both products of quantization-aware training (QAT). The developer released only this one version, with no higher-precision weights.
2.1 Total Parameters ≠ Compute Cost
The core idea of MoE is “many parameters, little compute.” The model stores the parameters of all 896 experts, but processes only 16 of them (plus the shared experts) for any given token. Roughly by activation ratio:
16 ÷ 896 ≈ 1.79%2.8T × 1.79% ≈ 50B (routed experts)50B + shared experts, attention, embeddings, etc. ≈ 104B (consistent with the official disclosure)
This means:
- The VRAM threshold is set by total parameters: all expert weights must stay resident in VRAM, waiting for the router to hit them.
- The compute threshold is closer to activated parameters: the matrix multiplications per token during decode are roughly equivalent to those of a tens-of-billions-scale dense model.
- Uneven resource use: different tokens route to different experts, so the load across GPUs is not equal at any given moment.
2.2 The Role of Shared Experts
Shared experts apply to all tokens and handle cross-domain general-purpose computation, letting the routed experts focus on differentiated knowledge. This is a common design in mainstream MoE architectures today. It introduces a fixed overhead that cannot be avoided through sparsification; the benefit is reduced knowledge duplication across experts and improved parameter efficiency.
2.3 Why There Is No Higher-Precision Version
Both weights and activations use the microscaling floating-point format: values are represented as 4-bit or 8-bit floats, with an accompanying set of shared scale factors. Multiple values share one scale factor, keeping low-precision computation numerically stable while requiring specialized operator support.
Releasing only a quantized version directly raises the hardware bar. Microscaling operators lack efficient implementations on consumer GPUs and most older-generation accelerator cards, and the cards that can run them tend to have VRAM large enough anyway—further compressing the room for “running locally for free.”
3. Running It Locally: The VRAM Math Behind the 8-GPU Starting Point
3.1 Breaking Down VRAM Requirements
Inference VRAM is made up of three main parts:
VRAM ≈ model weights + KV cache + runtime overhead
Splitting 1.56TB of weights across 8 GPUs means each card must carry about 180 GiB of weights. Adding KV cache and runtime overhead, a single card needs on the order of 200GB of VRAM. With cards that have less VRAM (such as the 192GB class), you need around 10 cards; with 141GB-class accelerators, you need 12 or more.
So “8 cards” doesn’t mean just any 8 cards—it is built on whole-machine servers with very large VRAM and high-speed inter-GPU interconnect. Inter-card bandwidth directly determines the efficiency of layer-by-layer synchronization during tensor parallelism.
| Single-card VRAM | Cards needed for weights only (est.) | Notes |
|---|---|---|
| 141GB class | 12+ | older-generation large-VRAM accelerators |
| 180GB class | 8–9 | mainstream large-VRAM inference cards |
| 192GB class | 9+ | close to the whole-machine limit |
| 288GB class | ~6 | new-generation large-VRAM accelerators |
3.2 The Often-Overlooked KV Cache
Weights are only part of the threshold; long context is the real VRAM black hole. KV-cache size can be estimated with:
KV cache bytes ≈ 2 (K and V) × layers × KV heads × dim per head × sequence length × bytes per element
Using common approximate values (actual numbers should follow the model configuration):
- 93 layers, with K and V together about 360,000 elements per token;
- at a sequence length of 1 million, a single request’s KV cache is about 672GB at FP16;
- even with an FP8 cache, still about 336GB;
- even a request of just 128,000 tokens is about 43GB at FP8.
This KV cache shares the same VRAM as the weights. So “supporting a 1M context” refers to the model’s architectural capability, not a single instance’s concurrent throughput. One more point worth noting: sparse attention reduces attention computation, but does not reduce the KV cache that must be stored.
3.3 Example 8-GPU Deployment Configuration
The following illustrates a configuration using the parameter naming of an open-source inference framework; actual fields depend on the version you use:
# 8-GPU single-machine deployment (illustrative)model_path: /models/local-large-moe-2.8ttensor_parallel_size: 8 # tensor parallelism: split each layer's weights across 8 cardsenable_expert_parallel: true # expert parallelism: distribute 896 experts across cardsmax_model_len: 131072 # context limit: reduced from 1M to control KV cachegpu_memory_utilization: 0.92 # VRAM utilization capkv_cache_dtype: fp8 # FP8 KV cache, essential for long contextenable_chunked_prefill: true # chunked prefill, easing VRAM and latency spikes for long promptslimit_mm_per_prompt: {image: 1} # limit images per request
Key points:
- Tensor parallelism: splits each layer’s matrix computation horizontally across cards, each holding part of the weights, with per-layer results synchronized via inter-card communication.
- Expert parallelism: assigns different experts to different cards. Tokens routed to a target expert’s card introduce all-to-all communication, so high-speed interconnect and topology-aware scheduling are especially important.
- Long-context strategy: in production,
max_model_lenis usually kept to the length the business actually needs; when ultra-long context is truly required, mechanisms such as chunked prefill or paged attention can reduce peak VRAM and fragmentation.
3.4 Supporting Operations
- Capacity planning: first estimate peak concurrent requests and average context length, then back out the KV-cache requirements to determine the instance and card count.
- Canary and evaluation: before launch, compare outputs of the API and the local deployment on real business samples, watching for quality differences introduced by quantization.
- Monitoring and alerting: set up monitoring for GPU VRAM, utilization, time-to-first-token latency, and per-token output latency, with threshold-based alerts.
- Logging and tracing: include request ID, token count, routed expert distribution, and similar data in logs for easier diagnosis.
4. Licensing: Open Weights ≠ Open Source
This is the most easily misread part of the release. The model repository carries a custom license, tagged as “other” on the hosting platform rather than a standard open-source license like MIT or Apache. Its terms are essentially the same as the previous generation’s license, with only a change of name.
The main body adopts an MIT-style grant: free use, copying, modification, distribution, fine-tuning, and commercial use are all permitted. On top of this, two restrictions apply only to large-scale businesses:
- A company offering the model as a service (MaaS) with cumulative revenue exceeding $20 million over 12 months must sign a separate agreement with the model’s rights holder before commercial use.
- Commercial products exceeding an agreed number of monthly active users likewise require a separate license.
What this means for developers:
- Low bar for ordinary commercial use: the vast majority of small and mid-sized teams and individual developers can use, fine-tune, and integrate the model within the license terms without extra negotiation.
- Large-scale companies should confirm early: companies near or above the revenue or MAU thresholds should start commercial discussions early rather than leaving licensing issues until after launch.
- Be cautious about redistributing weights: a custom license differs from standard open-source licenses; before secondary distribution (e.g., packaging into a product image), review the redistribution terms line by line.
- Clarify ownership of fine-tuned outputs: models fine-tuned from these weights are also subject to the license when offered externally, so it is advisable to note the base model and license type in project documentation.
5. Reading the Benchmarks: How to Interpret Vendor-Reported Scores
The vendor’s comparison table at release was entirely self-reported, with mixed results (the comparison is against a certain international closed-source flagship model):
| Evaluation area | This model | Closed-source flagship | Conclusion |
|---|---|---|---|
| Terminal-task coding | 65.7% | 66.4% | roughly on par |
| Long-horizon software engineering | 51.7% | 46.3% | ahead |
| Scientific terminal tasks | 38.1% | 63.3% | clearly behind |
| High-difficulty composite reasoning | 39.4% | 60.9% | clearly behind |
The vendor claims that of 37 benchmarks, 16 outperform the comparison model—so the picture is mixed: coding and agent tasks stand out, while science and some reasoning tasks are weaker. In addition, a third-party evaluation body’s intelligence index gives it 55, placing it near the top of that leaderboard.
A few tips for reading charts:
- Distinguish “capability” from “preference”: benchmark scores are heavily influenced by prompt templates, sampling parameters, and evaluation frameworks; self-reported results are useful as reference, not as conclusions.
- Align to your business scenario: if the business is mostly code generation and tool calling, refer to coding and agent benchmarks; for scientific reasoning, evaluate carefully.
- Build your own evaluation set: draw 100–300 representative tasks from real business, define quantifiable scoring rules (pass rate, human scores, etc.), and compare models under the same conditions.
Example scoring rule:
# Business evaluation sketch: pass rate and human score by case typedef score_case(case, output) -> dict:if case["type"] == "code":passed = run_unit_tests(output, case["tests"]) # judged by unit test passreturn {"passed": passed, "human_score": None}return {"passed": None, "human_score": rate_by_rubric(output, case["rubric"])}
6. How Different-Scale Users Should Choose
| User type | Recommended approach | Core reason |
|---|---|---|
| Individual / small team | prefer the API; rent large-VRAM GPU instances on demand for occasional local needs | self-hosting an 8-GPU cluster has high fixed cost and hard-to-guarantee utilization |
| Mid-sized enterprise (stable concurrency) | on-demand or annual cloud GPU instances, containerized, with autoscaling and monitoring/alerting | balances cost control with keeping data in-house |
| Large enterprise (strict data compliance) | self-hosted or dedicated resource pool, distributing weights from object storage, with an image registry and canary release pipeline | data sovereignty and supply-chain control |
| Scientific computing / research | not recommended as a primary option for now | clear gap vs. the closed-source flagship on science and hard reasoning |
When estimating cost, compare “cards per instance × unit price × duration” against “API token-based pricing” in the same table, and factor in GPU utilization. When utilization is persistently low, self-hosting is usually less cost-effective than the API.
7. Summary
The signal from this weight release is clear: the scale of open-weight models has entered the trillion-parameter era, and the gap between “can be downloaded” and “can be run” is widening. For technical teams, three things must be done at once:
- Compute: build a VRAM budget based on “weights + KV cache + runtime overhead,” not just the parameter count.
- Licensing: treat the custom license as part of the engineering constraints and complete a compliance review at the project-planning stage.
- Selection: validate model capability with your own evaluation set rather than relying on vendor-reported scores.
On the cloud-platform side, Baidu AI Cloud offers general-purpose capabilities such as GPU cloud hosts, object storage, container services, and monitoring/alerting to support elastic inference deployment and operations for this kind of ultra-large model. The challenges of ultra-large models ultimately come down to engineering details—get the VRAM and licensing math right first, then talk about performance, and you are less likely to stumble.
